Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

"portmap" plugin lost #890

Open
delfer opened this issue Nov 28, 2017 · 15 comments
Open

"portmap" plugin lost #890

delfer opened this issue Nov 28, 2017 · 15 comments

Comments

@delfer
Copy link

@delfer delfer commented Nov 28, 2017

Flannel v0.9.1 image does not contains "portmap" plugin enabled in 014b2d5#diff-7891b552b026259e99d479b5e30d31ca

Expected Behavior

working cluster

Current Behavior

kube-dns pod in ContainerCreating state with Failed create pod sandbox.
journalctl shows:

cni.go:319] Error deleting network: failed to find plugin "portmap" in path [/opt/flannel/bin /opt/cni/bin]
remote_runtime.go:115] StopPodSandbox "8046a5441a0f18637c643665d0d7bbf77ced11a0e987f9ee1f633e8e95afe952" from runtime service failed: rpc error: code = Unknown desc = NetworkPlugin cni failed to teardown pod "kube-dns-545bc4bfd4-sbhpp_kube-system" network: failed to find plugin "portmap" in path [/opt/flannel/bin /opt/cni/bin]
kuberuntime_gc.go:152] Failed to stop sandbox "8046a5441a0f18637c643665d0d7bbf77ced11a0e987f9ee1f633e8e95afe952" before removing: rpc error: code = Unknown desc = NetworkPlugin cni failed to teardown pod "kube-dns-545bc4bfd4-sbhpp_kube-system" network: failed to find plugin "portmap" in path [/opt/flannel/bin /opt/cni/bin]

Possible Solution

Do not enable portmap http://raw.githubusercontent.com/coreos/flannel/v0.9.1/Documentation/kube-flannel.yml

Steps to Reproduce (for bugs)

  1. kubeadm init --pod-network-cidr=10.244.0.0/16
  2. kubectl apply -f http://raw.githubusercontent.com/coreos/flannel/master/Documentation/kube-flannel.yml
  3. kubeadm init join ...

Your Environment

  • Flannel version: v0.9.1
  • Backend used (e.g. vxlan or udp): vxlan
  • Kubernetes version (if used): v1.8.4
  • Operating System and version: Debian 9
@magic7s
Copy link

@magic7s magic7s commented Dec 1, 2017

I have this issue as well.
OS: Ubuntu 16.04
root@ip-10-0-0-10:~# dpkg-query -L kubernetes-cni /. /opt /opt/cni /opt/cni/bin /opt/cni/bin/dhcp /opt/cni/bin/host-local /opt/cni/bin/bridge /opt/cni/bin/tuning /opt/cni/bin/macvlan /opt/cni/bin/flannel /opt/cni/bin/cnitool /opt/cni/bin/ptp /opt/cni/bin/loopback /opt/cni/bin/ipvlan /opt/cni/bin/noop

Fixed by downloading portmap to /opt/cni/bin
https://github.com/projectcalico/cni-plugin/releases/download/v1.9.1/portmap

magic7s added a commit to magic7s/ansible-kubeadm that referenced this issue Dec 3, 2017
…template from

https://raw.githubusercontent.com/coreos/flannel/master/Documentation/kube-flannel.yml requires portmap.
Because flannel requires but kubernetes-cni does not install, it needs to be manually copied to /opt/cni/bin/portmap
If portmap is not in place the symptom is the kube-dns will be stuck in CreatingContainer.
This additional task should be temp until either flannel no longer requires portmap or kubernetes-cni installs by default.
@osoriano
Copy link
Contributor

@osoriano osoriano commented Dec 5, 2017

Sounds like there are two workarounds.

  1. Use the 0.9.1 kube-flannel.yml
  2. Update the CNI plugin installation on the host

AFAIK we don't package CNI plugins into the flannel image. Maybe we should update the docs? Sorry for the breakage, the portmap plugin is used for hostPort support

@tomdee
Copy link
Contributor

@tomdee tomdee commented Dec 7, 2017

@osoriano Thanks for the summary.

IIUC this is only a problem for people using the kube-flannel.yml from master. It would be great to find a way to stop people from doing that!

And it would also be great if flannel had a better way of ensuring that the CNI plugins it needs are installed on the host, maybe https://github.com/coreos/flannel-cni could be updated to install the portmap plugin

@klausenbusk
Copy link

@klausenbusk klausenbusk commented Dec 8, 2017

maybe https://github.com/coreos/flannel-cni could be updated to install the portmap plugin

https://github.com/coreos/flannel-cni already install the portmap plugin and is used by bootkube where the portmap plugin is enabled.

@cmoscardi
Copy link

@cmoscardi cmoscardi commented Dec 8, 2017

+1, just ran into this

@ghost
Copy link

@ghost ghost commented Dec 9, 2017

I have used flannel-cni:v0.3.0 as initContainer to copy portmap (and flannel!) from the container to the host and ... portmapping is still not working, hostPort has no effect
I was checking with the netstat -lptn command and it was not showing me open ports, I think because they are in the different network namespace. Accessing the host on the hostPort from the outside works as expected.

@klausenbusk
Copy link

@klausenbusk klausenbusk commented Dec 9, 2017

I was checking with the netstat -lptn command and it was not showing me open ports, I think because they are in the different network namespace. Accessing the host on the hostPort from the outside works as expected.

portmap works by creating a iptables rule.. Check with iptables-save | grep <port> or something like that.

@ghost
Copy link

@ghost ghost commented Dec 10, 2017

@klausenbusk after I discovered that hostPort is accessible, I have also used iptables to check the rules. Just didn't mention this in the previous comment.

@fengyd2018
Copy link

@fengyd2018 fengyd2018 commented Feb 12, 2018

The 0.9.1 kube-flannel.yml is used, but hostPort still cannot work.
Any extra work is needed?

Enviroment:
Linux master 3.10.0-693.el7.x86_64 #1 SMP Tue Aug 22 21:09:27 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux

CentOS Linux release 7.4.1708 (Core)

[root@master ~]# kubectl version
Client Version: version.Info{Major:"1", Minor:"9", GitVersion:"v1.9.2", GitCommit:"5fa2db2bd46ac79e5e00a4e6ed24191080aa463b", GitTreeState:"clean", BuildDate:"2018-01-18T10:09:24Z", GoVersion:"go1.9.2", Compiler:"gc", Platform:"linux/amd64"}
Server Version: version.Info{Major:"1", Minor:"9", GitVersion:"v1.9.3", GitCommit:"d2835416544f298c919e2ead3be3d0864b52323b", GitTreeState:"clean", BuildDate:"2018-02-07T11:55:20Z", GoVersion:"go1.9.2", Compiler:"gc", Platform:"linux/amd64"}

@ghost
Copy link

@ghost ghost commented Feb 12, 2018

The 0.9.1 kube-flannel.yml is used

You mean this file? As you can clearly see, the portmapping is not enabled in the ConfigMap. Cf. https://github.com/coreos/flannel/blob/ce1f224dbd4f00f4abf6009bf412c31618d73921/Documentation/kube-flannel.yml#L55-L73

@fengyd2018
Copy link

@fengyd2018 fengyd2018 commented Feb 12, 2018

Thanks for your answer.

I tried with portmappign enabled, but hostPort cannot work.
https://raw.githubusercontent.com/coreos/flannel/master/Documentation/kube-flannel.yml

I just reset the kubernetes cluster and install it again, and hostPort can work now.

@fengyd2018
Copy link

@fengyd2018 fengyd2018 commented Feb 12, 2018

I think the kubernetes docs should be updated
https://kubernetes.io/docs/setup/independent/create-cluster-kubeadm/

In (3/4) Installing a pod network, the flannel version is still v0.9.1.
kubectl apply -f https://raw.githubusercontent.com/coreos/flannel/v0.9.1/Documentation/kube-flannel.yml

@m11y
Copy link

@m11y m11y commented Aug 10, 2018

I encounter the same problem, any updates?

@m11y
Copy link

@m11y m11y commented Aug 10, 2018

Download cni plugins binaries and put it in /opt/cni/bin solve my problem.

@aronica
Copy link

@aronica aronica commented Oct 16, 2018

Download cni plugins binaries and put it in /opt/cni/bin solve my problem.

This works for me with kubelet version v1.8.1

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Linked pull requests

Successfully merging a pull request may close this issue.

None yet
9 participants