ability to enable cni portmap plugin

[ekozan]

Hi,

Can you add the possibility of enable portmap plugin on all host on kubernetes

This will solve the hostport problems

Regards :)

[bboreham]

Something like this

[ekozan]

Yeah but it's dont work :/

[bboreham]

Any more details?

[ekozan]

Yes sorry :)
I have compiled it and add the binary on the /opt/cni/bin
/etc/cni/10-mynet.conflist

{
    "cniVersion": "0.5.1",
    "name": "mynet",
    "plugins": [
        {
            "type": "portmap",
            "capabilities": {"portMappings": true},
            "snat": true
        }
    ]
}

And restart kubelet and cni pod
After that the cni pod restart with no problems but ... all others pod stay stuck on container creating :/

[bboreham]

You can't run with just a portmap plugin; you need a network too.

By analogy to the example I pointed to, for Weave Net it would look something like:

{
    "cniVersion": "0.5.1",
    "name": "mynet",
    "plugins": [
        {
            "type": "weave",
            "name": "weave",
            "type": "weave-net",
            "hairpinMode": true
        },
        {
            "type": "portmap",
            "capabilities": {"portMappings": true},
            "snat": true
        }
    ]
}

caution: I haven't tried this

[ekozan]

this conflist ovewrite other conf file ? it's not realy clear for me :)

I'll give a try this night

[bboreham]

Yes, Kubernetes will only read one CNI config file, the first one it comes to (probably in alphanumeric order)

[nicolai86]

@bboreham I tried the following, but this sadly did not work:

1. place https://github.com/projectcalico/cni-plugin/releases/download/v1.9.0/portmap in /opt/cni/bin/portmap
2. configure /etc/cni/net.d/10-weave.conflist like this (no other file in the directory):

{
    "cniVersion": "0.5.1",
    "name": "weave",
    "plugins": [
        {
            "type": "weave",
            "name": "weave",
            "type": "weave-net",
            "hairpinMode": true
        },
        {
            "type": "portmap",
            "capabilities": {
                "portMappings": true
            },
            "snat": true
        }
    ]
}

3. restart the VM

I'm using https://gist.github.com/nicolai86/abfda3b82ec2ba6b4eac788a68c5ff86 as an example to test if hostPort is working. but it doesn't.

[ekozan]

I have made somme mistake on my post "cniVersion" is "0.3.1" ( protocol version )
And i correct the double "type" the correct is "weave-net"
But Nop :)

it's not work

{
    "cniVersion": "0.3.1",
    "name": "mynet",
    "plugins": [
	{
    	    "name": "weave",
            "type": "weave-net",
    	    "hairpinMode": true
	},
        {
            "type": "portmap",
            "capabilities": {"portMappings": true},
            "snat": true
        }
    ]
}

[ekozan]

\o/ working :)
"cniVersion": "0.3.0",

{
    "cniVersion": "0.3.0",
    "name": "mynet",
    "plugins": [
        {
            "name": "weave",
            "type": "weave-net",
            "hairpinMode": true
        },
        {
            "type": "portmap",
            "capabilities": {"portMappings": true},
            "snat": true
        }
    ]
}

[nicolai86]

@ekozan can you -vvv your steps a little? download portmap and save your content to /etc/cni/net.d/10-mynet.conflist?

[luxas]

@bboreham Will writing this new CNI spec happen automatically in the future?

[bboreham]

@luxas that would be best, yes. Need to think about how to detect if the user has the portmap plugin installed (or provide a way for Weave Net to install it).

PRs welcome 🙂

[luxas]

Maybe just do something like hit /version of the API Server?

[ekozan]

full step

$ git clone https://github.com/containernetworking/plugins 
$ cd plugins
$ ./build.sh
$ sudo cp bin/* /opt/cni/bin/
$ sudo sh -c 'cat >/etc/cni/net.d/10-mynet.conflist <<-EOF
{
    "cniVersion": "0.3.0",
    "name": "mynet",
      "plugins": [
        {
            "name": "weave",
            "type": "weave-net",
            "hairpinMode": true
        },
        {
            "type": "portmap",
            "capabilities": {"portMappings": true},
            "snat": true
        }
    ]
}
EOF'
$ rm /etc/cni/net.d/10-weave.conf
$ kubectl delete po --all --namespace=kube-system  

Enjoy

[nicolai86]

@ekozan thank you for the steps. how did you verify the hostPort is actually working?

[bboreham]

Maybe just do something like hit /version of the API Server?

@luxas as I see it two things are key:

1. does the machine have the portmap CNI plugin installed on disk?
2. does the local kubelet have the code to pass details of hostPort mappings?

The API server version doesn't indicate either of these things. I don't think there is a published API to get the kubelet version.

I think at present we would need a manual switch for the operator to turn on if they are sure the prerequisites are met.

[lukemarsden]

FTR, I got this working without requiring git or go (but adding a dependency on wget 😏) on the target system with a small tweak to the instructions in #3016 (comment) to use the released cni-plugins v0.6.0-rc1 package. On each of your nodes, run:

$ cd /tmp && mkdir cni-plugins && cd cni-plugins && \
    wget https://github.com/containernetworking/plugins/releases/download/v0.6.0-rc1/cni-plugins-amd64-v0.6.0-rc1.tgz && \
    tar zxfv cni-plugins-amd64-v0.6.0-rc1.tgz
$ sudo cp /tmp/cni-plugins/portmap /opt/cni/bin/
$ sudo sh -c 'cat >/etc/cni/net.d/10-mynet.conflist <<-EOF
{
    "cniVersion": "0.3.0",
    "name": "mynet",
      "plugins": [
        {
            "name": "weave",
            "type": "weave-net",
            "hairpinMode": true
        },
        {
            "type": "portmap",
            "capabilities": {"portMappings": true},
            "snat": true
        }
    ]
}
EOF'
$ rm /etc/cni/net.d/10-weave.conf

On a node with working kubectl (e.g. the master), run:

$ kubectl delete po --all --namespace=kube-system  

Note: if the /etc/cni/net.d/ file doesn't end .conflist, this won't work, and will fail with mysterious no plugin name provided errors.

[Bregor]

@lukemarsden why you need to restart all pods in kube-system?
Maybe just kubectl delete po -n kube-system -l name=weave-net will do the trick?

[lukemarsden]

@Bregor I was just copying #3016 (comment) 😄 I'm not sure what you need to delete exactly to "restart CNI", would be good if you could run some tests to find out?

[bboreham]

Restarting pods has no effect on CNI.

I might have expected you need to restart kubelet, but if not, that's cool.

[Bregor]

@lukemarsden, @ekozan which kubernetes version did you use?

I use kubernetes-1.6.8 with no success at all.

I tried following CNI + Plugins combinations:

• cni-0.5.1 + plugins-0.6.0-rc1
• cni-0.5.1 + plugins-0.6.0
• cni-0.5.2 + plugins-0.6.0-rc1
• cni-0.5.2 + plugins-0.6.0
• cni-0.6.0 + plugins-0.6.0-rc1
• cni-0.6.0 + plugins-0.6.0

Also tried all combinations above with weave-1.9.8 and 2.0.1.

No success at all.

Events:

nginx     2017-08-21 15:10:15 +0300 MSK   2017-08-21 15:00:48 +0300 MSK   45        nginx-3341093162-f3kkq   Pod                 Warning   FailedSync   kubelet, 10.130.20.89   Error syncing pod, skipping failed to "KillPodSandbox" for "3b3bc4a9-8664-11e7-a06b-264916ab5742" with KillPodSandboxError: "rpc error: code = 2 desc = NetworkPlugin cni failed to teardown pod \"nginx-3341093162-f3kkq_nginx\" network: no plugin name provided"

Kubelet logs:

Aug 21 15:11:47 kube01 kubelet[24531]: W0821 15:11:47.377124   24531 docker_sandbox.go:263] NetworkPlugin cni failed on the status hook for pod "nginx-3341093162-f3kkq_nginx": Cannot find the network namespace, skipping pod network status for container {"docker" "1369eaf944618152e832a70ae221e808f888700082f1f712d05bef871b2e0933"}
Aug 21 15:11:47 kube01 kubelet[24531]: E0821 15:11:47.379565   24531 cni.go:278] Error deleting network: no plugin name provided
Aug 21 15:11:47 kube01 kubelet[24531]: E0821 15:11:47.382972   24531 remote_runtime.go:109] StopPodSandbox "1369eaf944618152e832a70ae221e808f888700082f1f712d05bef871b2e0933" from runtime service failed: rpc error: code = 2 desc = NetworkPlugin cni failed to teardown pod "nginx-3341093162-f3kkq_nginx" network: no plugin name provided
Aug 21 15:11:47 kube01 kubelet[24531]: E0821 15:11:47.383066   24531 kuberuntime_manager.go:784] Failed to stop sandbox {"docker" "1369eaf944618152e832a70ae221e808f888700082f1f712d05bef871b2e0933"}
Aug 21 15:11:47 kube01 kubelet[24531]: E0821 15:11:47.383285   24531 kuberuntime_manager.go:573] killPodWithSyncResult failed: failed to "KillPodSandbox" for "3b3bc4a9-8664-11e7-a06b-264916ab5742" with KillPodSandboxError: "rpc error: code = 2 desc = NetworkPlugin cni failed to teardown pod \"nginx-3341093162-f3kkq_nginx\" network: no plugin name provided"
Aug 21 15:11:47 kube01 kubelet[24531]: E0821 15:11:47.383340   24531 pod_workers.go:182] Error syncing pod 3b3bc4a9-8664-11e7-a06b-264916ab5742 ("nginx-3341093162-f3kkq_nginx(3b3bc4a9-8664-11e7-a06b-264916ab5742)"), skipping: failed to "KillPodSandbox" for "3b3bc4a9-8664-11e7-a06b-264916ab5742" with KillPodSandboxError: "rpc error: code = 2 desc = NetworkPlugin cni failed to teardown pod \"nginx-3341093162-f3kkq_nginx\" network: no plugin name provided"

Contents of /etc/cni/net.d/10-weave.conf:

{
    "cniVersion": "0.3.0",
    "name": "weave",
    "plugins": [
        {
            "name": "weave",
            "type": "weave-net",
            "hairpinMode": true
        },
        {
            "type": "portmap",
            "capabilities": {
                "portMappings": true
            },
            "snat": true
        }
    ]
}

What am I doing wrong?..

[Bregor]

Ah, nevermind.
Just realized, that ability to use NetworkConfigList became available only from 1.7.0-alpha3 :(

[deitch]

@bboreham when I run the kube add-on, it automatically generates /etc/cni/net.d/10-weave.conf. If I have 10-weave.conflist, CNI will pick up the first and miss my generated conflist.

Short of giving it a lower number (higher priority), is there any way to get it to not generate 10-weave.conf?

Conversely, if there were a way to tell the add-on to add the portmap - generate the conflistwith portmap instead of the single conf - perhaps via env var, that would be valuable.

You mentioned earlier that it has an issue detecting; this would give at least a partial resolution.

[dtshepherd]

@bboreham I see you tagged "help wanted" on here. Do you have any thoughts how you would solve this? I'm using kops to spin up our cluster, but we need portmap. It seems the way Calico solved this with kops was to package the portmap binary with their docker image and then copy it onto the host through the /opt/cni/bin mount.

I'm wondering if this needs to be a combined fix between kops and weave. We could have kops install the portmap binary (it already has the full CNI tarball available since it loads other CNI binaries), then update weave to install the proper /etc/cni/net.d/10-weave.conflist. The current conf doesn't include portmapper obviously.

[murali-reddy]

I'm wondering if this needs to be a combined fix between kops and weave. We could have kops install the portmap binary (it already has the full CNI tarball available since it loads other CNI binaries)

With latest releases of kops portmap plugin is part of the bundle of CNI plugins downloaded on to the node. So nothing extra needed to download the plugin. However it is not by default placed in /opt/cni/bin.

My guess is we might have to change this check/add portmap to the list
https://github.com/kubernetes/kops/blob/master/nodeup/pkg/model/network.go#L46
to place portmap in the CNI bin directory.

@dtshepherd let me know if you would like to give it a try, otherwise I am happy make the change in Kops.

[bboreham]

Do you have any thoughts how you would solve this?

Not really. Probably the whole thing needs taking apart and doing again.

The current implementation is a bit minamalist and simplest-thing-that-works. If I remember correctly, at the time nobody was installing add-ons purely by running a daemonset; the state of the art was to modify the Kubernetes in-tree saltstack.

The inside of a DameonSet is a terrible place to make decisions about what is installed and where. To that extent, places like kops and kubeadm are much better.

[deitch]

The inside of a DameonSet is a terrible place to make decisions about what is installed and where. To that extent, places like kops and kubeadm are much better.

The problem with places outside is that they are, well, outside. There is something great about it all just being in a single yml you can kubectl apply, and keeping it clean.

FWIW, the last installation I did was a mix of the cloud-init downloading the standard CNI distro and installing it, and a modified yml for the DaemonSet with an init container that installs an overriding config file:

          initContainers:
            # set up our correct conflist in /etc/cni/net.d
            - name: init-conflist
              image: busybox
              command: ['sh','-c','mkdir -p /host/etc/cni/net.d/ && cp /conf/00-weave.conflist /host/etc/cni/net.d/']
              volumeMounts:
              - name: cni-conf
                mountPath: /host/etc
              - name: config-volume
                mountPath: /conf

Where the 00-weave.conflist looks like:

        {
            "cniVersion": "0.3.0",
            "name": "mynet",
              "plugins": [
                {
                    "name": "weave",
                    "type": "weave-net",
                    "hairpinMode": true
                },
                {
                    "type": "portmap",
                    "capabilities": {"portMappings": true},
                    "snat": true
                }
            ]
        }

I rely on the ordering of numbers 00 to get it pulled in. It is somewhat brittle, but at least everything "weave-ish" is in that one place, and everything "standard" is in my cloud-init.

[dtshepherd]

My guess is we might have to change this check/add portmap to the list
https://github.com/kubernetes/kops/blob/master/nodeup/pkg/model/network.go#L46
to place portmap in the CNI bin directory.

@murali-reddy Yes, it is a one-liner change in that file but we also need to update the weave install template for kops (https://github.com/kubernetes/kops/blob/master/upup/models/cloudup/resources/addons/networking.weave/k8s-1.6.yaml.template). However, it would be nice for weave to support generating the conflist rather than the conf in /etc/cni/net.d/. I guess for now, we could use the hack suggested by @deitch.

@bboreham As an initial rework to weave, could we do something similar to flannel and use a configmap to generate the /etc/cni/net.d/ file? Currently, the CNI conf is hardcoded in the weave script.

[bboreham]

configmap fine as long as it's optional.

It was so long ago when I did it, optional configmaps didn't exist in Kubernetes...

[dtshepherd]

Yeah the optional configmaps are nice. So maybe implement the logic "if cni configmap exists, use it... otherwise, default to the hardcoded 10-weave.conf.

[murali-reddy]

So portmap CNI will be installed by default in case of kops with upcoming 1.10 release.

kubernetes/kops#5474
kubernetes/kops#5494

this will only cover kops side of things. Still weave need to create .conflist

[fjudith]

Hello,

Do you know this version will be publish to DockerHub.

Lastest (git-ad4e4238a5d5) still provides non portmap 10-weave.conf

Can't wait to test it !

[murali-reddy]

Do you know this version will be publish to DockerHub.

@fjudith support for portmap will be part of 2.5 release

I just need to add a integration test to finish #3356

Let me know if you would like to test out the changes in #3356, i will share the docker image.

[fjudith]

I have a lab for this puposal running Containerd (1.2.0-Beta.2) and Docker (17.03.2) along with Kubernetes 1.11.2.
I'm open for further testings.

I'm not using the stable containerd v1.1.3 as it does not detects cni configs pushed by k8s like weave-kube. It works find with 1.2.0

[murali-reddy]

@fjudith Great! Please use the image muralireddy/weave-kube:portmap. Note that once you use this image its going to remove /etc/cni/net.d/10-weave.conf on all nodes and replace with /etc/cni/net.d/10-weave.conflist. So if you have revert back please delete .conflist file.

[annismckenzie]

That is exactly why I'd go with an at least in part backwards compatible version if you're going to remove the 10-weave.conf file. Use 11-weave.conflist and then on downgrade the old Weave image will still work because it'll create the 10-weave.conf again and as it takes precedence, it'll not crash. Upgrading to your beta image will drop it again and use the conflist again. Might be worth the effort.

[murali-reddy]

Use 11-weave.conflist and then on downgrade the old Weave image will still work

Ok. that make sense. That might work. As you say its worth the effort to try out. Let me test out and see if can make both forward and backward compatible.

[murali-reddy]

I tested out my PR #3356 again from the perspective of backward compatibility. There is nothing in particular need to be done. When upgrade is done to new release 10-weave.conf is deleted and 10-weave.conflist will be used further. If Weave need to be downgraded to old release, 10-weave.conf will be created. In case there are multiple conf file first one in lexicographical order is chosen. So 10-weave.conf will be used.