New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

ability to enable cni portmap plugin #3016

Closed
ekozan opened this Issue Jun 13, 2017 · 60 comments

Comments

Projects
None yet
@ekozan

ekozan commented Jun 13, 2017

Hi,

Can you add the possibility of enable portmap plugin on all host on kubernetes

This will solve the hostport problems

Regards :)

@bboreham bboreham changed the title from ability to enable chi portmap plugin to ability to enable cni portmap plugin Jun 15, 2017

@bboreham

This comment has been minimized.

Member

bboreham commented Jun 16, 2017

Something like this

@ekozan

This comment has been minimized.

ekozan commented Jul 4, 2017

Yeah but it's dont work :/

@bboreham

This comment has been minimized.

Member

bboreham commented Jul 4, 2017

Any more details?

@ekozan

This comment has been minimized.

ekozan commented Jul 5, 2017

Yes sorry :)
I have compiled it and add the binary on the /opt/cni/bin
/etc/cni/10-mynet.conflist

{
    "cniVersion": "0.5.1",
    "name": "mynet",
    "plugins": [
        {
            "type": "portmap",
            "capabilities": {"portMappings": true},
            "snat": true
        }
    ]
}

And restart kubelet and cni pod
After that the cni pod restart with no problems but ... all others pod stay stuck on container creating :/

@bboreham

This comment has been minimized.

Member

bboreham commented Jul 5, 2017

You can't run with just a portmap plugin; you need a network too.

By analogy to the example I pointed to, for Weave Net it would look something like:

{
    "cniVersion": "0.5.1",
    "name": "mynet",
    "plugins": [
        {
            "type": "weave",
            "name": "weave",
            "type": "weave-net",
            "hairpinMode": true
        },
        {
            "type": "portmap",
            "capabilities": {"portMappings": true},
            "snat": true
        }
    ]
}

caution: I haven't tried this

@ekozan

This comment has been minimized.

ekozan commented Jul 5, 2017

this conflist ovewrite other conf file ? it's not realy clear for me :)

I'll give a try this night

@bboreham

This comment has been minimized.

Member

bboreham commented Jul 5, 2017

Yes, Kubernetes will only read one CNI config file, the first one it comes to (probably in alphanumeric order)

@nicolai86

This comment has been minimized.

nicolai86 commented Jul 5, 2017

@bboreham I tried the following, but this sadly did not work:

  1. place https://github.com/projectcalico/cni-plugin/releases/download/v1.9.0/portmap in /opt/cni/bin/portmap
  2. configure /etc/cni/net.d/10-weave.conflist like this (no other file in the directory):
{
    "cniVersion": "0.5.1",
    "name": "weave",
    "plugins": [
        {
            "type": "weave",
            "name": "weave",
            "type": "weave-net",
            "hairpinMode": true
        },
        {
            "type": "portmap",
            "capabilities": {
                "portMappings": true
            },
            "snat": true
        }
    ]
}
  1. restart the VM

I'm using https://gist.github.com/nicolai86/abfda3b82ec2ba6b4eac788a68c5ff86 as an example to test if hostPort is working. but it doesn't.

@ekozan

This comment has been minimized.

ekozan commented Jul 5, 2017

I have made somme mistake on my post "cniVersion" is "0.3.1" ( protocol version )
And i correct the double "type" the correct is "weave-net"
But Nop :)

it's not work

{
    "cniVersion": "0.3.1",
    "name": "mynet",
    "plugins": [
	{
    	    "name": "weave",
            "type": "weave-net",
    	    "hairpinMode": true
	},
        {
            "type": "portmap",
            "capabilities": {"portMappings": true},
            "snat": true
        }
    ]
}
@ekozan

This comment has been minimized.

ekozan commented Jul 5, 2017

\o/ working :)
"cniVersion": "0.3.0",

{
    "cniVersion": "0.3.0",
    "name": "mynet",
    "plugins": [
        {
            "name": "weave",
            "type": "weave-net",
            "hairpinMode": true
        },
        {
            "type": "portmap",
            "capabilities": {"portMappings": true},
            "snat": true
        }
    ]
}
@nicolai86

This comment has been minimized.

nicolai86 commented Jul 5, 2017

@ekozan can you -vvv your steps a little? download portmap and save your content to /etc/cni/net.d/10-mynet.conflist?

@luxas

This comment has been minimized.

Contributor

luxas commented Jul 5, 2017

@bboreham Will writing this new CNI spec happen automatically in the future?

@bboreham

This comment has been minimized.

Member

bboreham commented Jul 5, 2017

@luxas that would be best, yes. Need to think about how to detect if the user has the portmap plugin installed (or provide a way for Weave Net to install it).

PRs welcome 🙂

@luxas

This comment has been minimized.

Contributor

luxas commented Jul 5, 2017

Maybe just do something like hit /version of the API Server?

@ekozan

This comment has been minimized.

ekozan commented Jul 5, 2017

full step

$ git clone https://github.com/containernetworking/plugins 
$ cd plugins
$ ./build.sh
$ sudo cp bin/* /opt/cni/bin/
$ sudo sh -c 'cat >/etc/cni/net.d/10-mynet.conflist <<-EOF
{
    "cniVersion": "0.3.0",
    "name": "mynet",
      "plugins": [
        {
            "name": "weave",
            "type": "weave-net",
            "hairpinMode": true
        },
        {
            "type": "portmap",
            "capabilities": {"portMappings": true},
            "snat": true
        }
    ]
}
EOF'
$ rm /etc/cni/net.d/10-weave.conf
$ kubectl delete po --all --namespace=kube-system  

Enjoy

@nicolai86

This comment has been minimized.

nicolai86 commented Jul 5, 2017

@ekozan thank you for the steps. how did you verify the hostPort is actually working?

@bboreham

This comment has been minimized.

Member

bboreham commented Jul 7, 2017

Maybe just do something like hit /version of the API Server?

@luxas as I see it two things are key:

  1. does the machine have the portmap CNI plugin installed on disk?
  2. does the local kubelet have the code to pass details of hostPort mappings?

The API server version doesn't indicate either of these things. I don't think there is a published API to get the kubelet version.

I think at present we would need a manual switch for the operator to turn on if they are sure the prerequisites are met.

@lukemarsden

This comment has been minimized.

lukemarsden commented Aug 9, 2017

FTR, I got this working without requiring git or go (but adding a dependency on wget 😏) on the target system with a small tweak to the instructions in #3016 (comment) to use the released cni-plugins v0.6.0-rc1 package. On each of your nodes, run:

$ cd /tmp && mkdir cni-plugins && cd cni-plugins && \
    wget https://github.com/containernetworking/plugins/releases/download/v0.6.0-rc1/cni-plugins-amd64-v0.6.0-rc1.tgz && \
    tar zxfv cni-plugins-amd64-v0.6.0-rc1.tgz
$ sudo cp /tmp/cni-plugins/portmap /opt/cni/bin/
$ sudo sh -c 'cat >/etc/cni/net.d/10-mynet.conflist <<-EOF
{
    "cniVersion": "0.3.0",
    "name": "mynet",
      "plugins": [
        {
            "name": "weave",
            "type": "weave-net",
            "hairpinMode": true
        },
        {
            "type": "portmap",
            "capabilities": {"portMappings": true},
            "snat": true
        }
    ]
}
EOF'
$ rm /etc/cni/net.d/10-weave.conf

On a node with working kubectl (e.g. the master), run:

$ kubectl delete po --all --namespace=kube-system  

Note: if the /etc/cni/net.d/ file doesn't end .conflist, this won't work, and will fail with mysterious no plugin name provided errors.

@Bregor

This comment has been minimized.

Contributor

Bregor commented Aug 11, 2017

@lukemarsden why you need to restart all pods in kube-system?
Maybe just kubectl delete po -n kube-system -l name=weave-net will do the trick?

@lukemarsden

This comment has been minimized.

lukemarsden commented Aug 11, 2017

@Bregor I was just copying #3016 (comment) 😄 I'm not sure what you need to delete exactly to "restart CNI", would be good if you could run some tests to find out?

@bboreham

This comment has been minimized.

Member

bboreham commented Aug 11, 2017

Restarting pods has no effect on CNI.

I might have expected you need to restart kubelet, but if not, that's cool.

@Bregor

This comment has been minimized.

Contributor

Bregor commented Aug 21, 2017

@lukemarsden, @ekozan which kubernetes version did you use?

I use kubernetes-1.6.8 with no success at all.

I tried following CNI + Plugins combinations:

  • cni-0.5.1 + plugins-0.6.0-rc1
  • cni-0.5.1 + plugins-0.6.0
  • cni-0.5.2 + plugins-0.6.0-rc1
  • cni-0.5.2 + plugins-0.6.0
  • cni-0.6.0 + plugins-0.6.0-rc1
  • cni-0.6.0 + plugins-0.6.0

Also tried all combinations above with weave-1.9.8 and 2.0.1.

No success at all.

Events:

nginx     2017-08-21 15:10:15 +0300 MSK   2017-08-21 15:00:48 +0300 MSK   45        nginx-3341093162-f3kkq   Pod                 Warning   FailedSync   kubelet, 10.130.20.89   Error syncing pod, skipping failed to "KillPodSandbox" for "3b3bc4a9-8664-11e7-a06b-264916ab5742" with KillPodSandboxError: "rpc error: code = 2 desc = NetworkPlugin cni failed to teardown pod \"nginx-3341093162-f3kkq_nginx\" network: no plugin name provided"

Kubelet logs:

Aug 21 15:11:47 kube01 kubelet[24531]: W0821 15:11:47.377124   24531 docker_sandbox.go:263] NetworkPlugin cni failed on the status hook for pod "nginx-3341093162-f3kkq_nginx": Cannot find the network namespace, skipping pod network status for container {"docker" "1369eaf944618152e832a70ae221e808f888700082f1f712d05bef871b2e0933"}
Aug 21 15:11:47 kube01 kubelet[24531]: E0821 15:11:47.379565   24531 cni.go:278] Error deleting network: no plugin name provided
Aug 21 15:11:47 kube01 kubelet[24531]: E0821 15:11:47.382972   24531 remote_runtime.go:109] StopPodSandbox "1369eaf944618152e832a70ae221e808f888700082f1f712d05bef871b2e0933" from runtime service failed: rpc error: code = 2 desc = NetworkPlugin cni failed to teardown pod "nginx-3341093162-f3kkq_nginx" network: no plugin name provided
Aug 21 15:11:47 kube01 kubelet[24531]: E0821 15:11:47.383066   24531 kuberuntime_manager.go:784] Failed to stop sandbox {"docker" "1369eaf944618152e832a70ae221e808f888700082f1f712d05bef871b2e0933"}
Aug 21 15:11:47 kube01 kubelet[24531]: E0821 15:11:47.383285   24531 kuberuntime_manager.go:573] killPodWithSyncResult failed: failed to "KillPodSandbox" for "3b3bc4a9-8664-11e7-a06b-264916ab5742" with KillPodSandboxError: "rpc error: code = 2 desc = NetworkPlugin cni failed to teardown pod \"nginx-3341093162-f3kkq_nginx\" network: no plugin name provided"
Aug 21 15:11:47 kube01 kubelet[24531]: E0821 15:11:47.383340   24531 pod_workers.go:182] Error syncing pod 3b3bc4a9-8664-11e7-a06b-264916ab5742 ("nginx-3341093162-f3kkq_nginx(3b3bc4a9-8664-11e7-a06b-264916ab5742)"), skipping: failed to "KillPodSandbox" for "3b3bc4a9-8664-11e7-a06b-264916ab5742" with KillPodSandboxError: "rpc error: code = 2 desc = NetworkPlugin cni failed to teardown pod \"nginx-3341093162-f3kkq_nginx\" network: no plugin name provided"

Contents of /etc/cni/net.d/10-weave.conf:

{
    "cniVersion": "0.3.0",
    "name": "weave",
    "plugins": [
        {
            "name": "weave",
            "type": "weave-net",
            "hairpinMode": true
        },
        {
            "type": "portmap",
            "capabilities": {
                "portMappings": true
            },
            "snat": true
        }
    ]
}

What am I doing wrong?..

@Bregor

This comment has been minimized.

Contributor

Bregor commented Aug 21, 2017

Ah, nevermind.
Just realized, that ability to use NetworkConfigList became available only from 1.7.0-alpha3 :(

@deitch

This comment has been minimized.

Contributor

deitch commented Oct 6, 2017

@bboreham when I run the kube add-on, it automatically generates /etc/cni/net.d/10-weave.conf. If I have 10-weave.conflist, CNI will pick up the first and miss my generated conflist.

Short of giving it a lower number (higher priority), is there any way to get it to not generate 10-weave.conf?

Conversely, if there were a way to tell the add-on to add the portmap - generate the conflistwith portmap instead of the single conf - perhaps via env var, that would be valuable.

You mentioned earlier that it has an issue detecting; this would give at least a partial resolution.

@dtshepherd

This comment has been minimized.

Collaborator

dtshepherd commented Jul 13, 2018

@bboreham I see you tagged "help wanted" on here. Do you have any thoughts how you would solve this? I'm using kops to spin up our cluster, but we need portmap. It seems the way Calico solved this with kops was to package the portmap binary with their docker image and then copy it onto the host through the /opt/cni/bin mount.

I'm wondering if this needs to be a combined fix between kops and weave. We could have kops install the portmap binary (it already has the full CNI tarball available since it loads other CNI binaries), then update weave to install the proper /etc/cni/net.d/10-weave.conflist. The current conf doesn't include portmapper obviously.

@murali-reddy

This comment has been minimized.

Contributor

murali-reddy commented Jul 16, 2018

I'm wondering if this needs to be a combined fix between kops and weave. We could have kops install the portmap binary (it already has the full CNI tarball available since it loads other CNI binaries)

With latest releases of kops portmap plugin is part of the bundle of CNI plugins downloaded on to the node. So nothing extra needed to download the plugin. However it is not by default placed in /opt/cni/bin.

My guess is we might have to change this check/add portmap to the list
https://github.com/kubernetes/kops/blob/master/nodeup/pkg/model/network.go#L46
to place portmap in the CNI bin directory.

@dtshepherd let me know if you would like to give it a try, otherwise I am happy make the change in Kops.

@bboreham

This comment has been minimized.

Member

bboreham commented Jul 16, 2018

Do you have any thoughts how you would solve this?

Not really. Probably the whole thing needs taking apart and doing again.

The current implementation is a bit minamalist and simplest-thing-that-works. If I remember correctly, at the time nobody was installing add-ons purely by running a daemonset; the state of the art was to modify the Kubernetes in-tree saltstack.

The inside of a DameonSet is a terrible place to make decisions about what is installed and where. To that extent, places like kops and kubeadm are much better.

@deitch

This comment has been minimized.

Contributor

deitch commented Jul 16, 2018

The inside of a DameonSet is a terrible place to make decisions about what is installed and where. To that extent, places like kops and kubeadm are much better.

The problem with places outside is that they are, well, outside. There is something great about it all just being in a single yml you can kubectl apply, and keeping it clean.

FWIW, the last installation I did was a mix of the cloud-init downloading the standard CNI distro and installing it, and a modified yml for the DaemonSet with an init container that installs an overriding config file:

          initContainers:
            # set up our correct conflist in /etc/cni/net.d
            - name: init-conflist
              image: busybox
              command: ['sh','-c','mkdir -p /host/etc/cni/net.d/ && cp /conf/00-weave.conflist /host/etc/cni/net.d/']
              volumeMounts:
              - name: cni-conf
                mountPath: /host/etc
              - name: config-volume
                mountPath: /conf

Where the 00-weave.conflist looks like:

        {
            "cniVersion": "0.3.0",
            "name": "mynet",
              "plugins": [
                {
                    "name": "weave",
                    "type": "weave-net",
                    "hairpinMode": true
                },
                {
                    "type": "portmap",
                    "capabilities": {"portMappings": true},
                    "snat": true
                }
            ]
        }

I rely on the ordering of numbers 00 to get it pulled in. It is somewhat brittle, but at least everything "weave-ish" is in that one place, and everything "standard" is in my cloud-init.

@dtshepherd

This comment has been minimized.

Collaborator

dtshepherd commented Jul 16, 2018

My guess is we might have to change this check/add portmap to the list
https://github.com/kubernetes/kops/blob/master/nodeup/pkg/model/network.go#L46
to place portmap in the CNI bin directory.

@murali-reddy Yes, it is a one-liner change in that file but we also need to update the weave install template for kops (https://github.com/kubernetes/kops/blob/master/upup/models/cloudup/resources/addons/networking.weave/k8s-1.6.yaml.template). However, it would be nice for weave to support generating the conflist rather than the conf in /etc/cni/net.d/. I guess for now, we could use the hack suggested by @deitch.

@bboreham As an initial rework to weave, could we do something similar to flannel and use a configmap to generate the /etc/cni/net.d/ file? Currently, the CNI conf is hardcoded in the weave script.

@bboreham

This comment has been minimized.

Member

bboreham commented Jul 16, 2018

configmap fine as long as it's optional.

It was so long ago when I did it, optional configmaps didn't exist in Kubernetes...

@dtshepherd

This comment has been minimized.

Collaborator

dtshepherd commented Jul 16, 2018

Yeah the optional configmaps are nice. So maybe implement the logic "if cni configmap exists, use it... otherwise, default to the hardcoded 10-weave.conf.

@murali-reddy

This comment has been minimized.

Contributor

murali-reddy commented Jul 23, 2018

So portmap CNI will be installed by default in case of kops with upcoming 1.10 release.

kubernetes/kops#5474
kubernetes/kops#5494

this will only cover kops side of things. Still weave need to create .conflist

@murali-reddy murali-reddy self-assigned this Jul 25, 2018

@murali-reddy murali-reddy added this to the 2.5 milestone Jul 25, 2018

murali-reddy added a commit that referenced this issue Jul 25, 2018

Fix to use 10-weave.conflist instead of 10-weave.conf with support fo…
…r portmap

as additional plugin along with weave-net

Fixes #3016

murali-reddy added a commit that referenced this issue Jul 27, 2018

Fix to use 10-weave.conflist instead of 10-weave.conf with support fo…
…r portmap

as additional plugin along with weave-net

Fixes #3016

murali-reddy added a commit that referenced this issue Jul 30, 2018

Fix to use 10-weave.conflist instead of 10-weave.conf as CNI config f…
…ile.

Also adds support for portmap as additional plugin along with weave-net

Fixes #3016

murali-reddy added a commit that referenced this issue Jul 30, 2018

Fix to use 10-weave.conflist instead of 10-weave.conf as CNI config f…
…ile.

Also adds support for portmap as additional plugin along with weave-net

Fixes #3016

murali-reddy added a commit that referenced this issue Jul 30, 2018

Fix to use 10-weave.conflist instead of 10-weave.conf as CNI config f…
…ile.

Also adds support for portmap as additional plugin along with weave-net

Fixes #3016
@fjudith

This comment has been minimized.

fjudith commented Aug 30, 2018

Hello,

Do you know this version will be publish to DockerHub.

Lastest (git-ad4e4238a5d5) still provides non portmap 10-weave.conf

Can't wait to test it !

@murali-reddy

This comment has been minimized.

Contributor

murali-reddy commented Aug 31, 2018

Do you know this version will be publish to DockerHub.

@fjudith support for portmap will be part of 2.5 release

I just need to add a integration test to finish #3356

Let me know if you would like to test out the changes in #3356, i will share the docker image.

@fjudith

This comment has been minimized.

fjudith commented Aug 31, 2018

I have a lab for this puposal running Containerd (1.2.0-Beta.2) and Docker (17.03.2) along with Kubernetes 1.11.2.
I'm open for further testings.

I'm not using the stable containerd v1.1.3 as it does not detects cni configs pushed by k8s like weave-kube. It works find with 1.2.0

@murali-reddy

This comment has been minimized.

Contributor

murali-reddy commented Aug 31, 2018

@fjudith Great! Please use the image muralireddy/weave-kube:portmap. Note that once you use this image its going to remove /etc/cni/net.d/10-weave.conf on all nodes and replace with /etc/cni/net.d/10-weave.conflist. So if you have revert back please delete .conflist file.

@annismckenzie

This comment has been minimized.

annismckenzie commented Aug 31, 2018

That is exactly why I'd go with an at least in part backwards compatible version if you're going to remove the 10-weave.conf file. Use 11-weave.conflist and then on downgrade the old Weave image will still work because it'll create the 10-weave.conf again and as it takes precedence, it'll not crash. Upgrading to your beta image will drop it again and use the conflist again. Might be worth the effort.

murali-reddy added a commit that referenced this issue Sep 3, 2018

Fix to use 10-weave.conflist instead of 10-weave.conf as CNI config f…
…ile.

Also adds support for portmap as additional plugin along with weave-net

Fixes #3016
@murali-reddy

This comment has been minimized.

Contributor

murali-reddy commented Sep 3, 2018

Use 11-weave.conflist and then on downgrade the old Weave image will still work

Ok. that make sense. That might work. As you say its worth the effort to try out. Let me test out and see if can make both forward and backward compatible.

@murali-reddy

This comment has been minimized.

Contributor

murali-reddy commented Sep 5, 2018

I tested out my PR #3356 again from the perspective of backward compatibility. There is nothing in particular need to be done. When upgrade is done to new release 10-weave.conf is deleted and 10-weave.conflist will be used further. If Weave need to be downgraded to old release, 10-weave.conf will be created. In case there are multiple conf file first one in lexicographical order is chosen. So 10-weave.conf will be used.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment