New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

"portmap" plugin lost #890

Open
delfer opened this Issue Nov 28, 2017 · 15 comments

Comments

Projects
None yet
9 participants
@delfer
Copy link

delfer commented Nov 28, 2017

Flannel v0.9.1 image does not contains "portmap" plugin enabled in 014b2d5#diff-7891b552b026259e99d479b5e30d31ca

Expected Behavior

working cluster

Current Behavior

kube-dns pod in ContainerCreating state with Failed create pod sandbox.
journalctl shows:

cni.go:319] Error deleting network: failed to find plugin "portmap" in path [/opt/flannel/bin /opt/cni/bin]
remote_runtime.go:115] StopPodSandbox "8046a5441a0f18637c643665d0d7bbf77ced11a0e987f9ee1f633e8e95afe952" from runtime service failed: rpc error: code = Unknown desc = NetworkPlugin cni failed to teardown pod "kube-dns-545bc4bfd4-sbhpp_kube-system" network: failed to find plugin "portmap" in path [/opt/flannel/bin /opt/cni/bin]
kuberuntime_gc.go:152] Failed to stop sandbox "8046a5441a0f18637c643665d0d7bbf77ced11a0e987f9ee1f633e8e95afe952" before removing: rpc error: code = Unknown desc = NetworkPlugin cni failed to teardown pod "kube-dns-545bc4bfd4-sbhpp_kube-system" network: failed to find plugin "portmap" in path [/opt/flannel/bin /opt/cni/bin]

Possible Solution

Do not enable portmap http://raw.githubusercontent.com/coreos/flannel/v0.9.1/Documentation/kube-flannel.yml

Steps to Reproduce (for bugs)

  1. kubeadm init --pod-network-cidr=10.244.0.0/16
  2. kubectl apply -f http://raw.githubusercontent.com/coreos/flannel/master/Documentation/kube-flannel.yml
  3. kubeadm init join ...

Your Environment

  • Flannel version: v0.9.1
  • Backend used (e.g. vxlan or udp): vxlan
  • Kubernetes version (if used): v1.8.4
  • Operating System and version: Debian 9
@magic7s

This comment has been minimized.

Copy link

magic7s commented Dec 1, 2017

I have this issue as well.
OS: Ubuntu 16.04
root@ip-10-0-0-10:~# dpkg-query -L kubernetes-cni /. /opt /opt/cni /opt/cni/bin /opt/cni/bin/dhcp /opt/cni/bin/host-local /opt/cni/bin/bridge /opt/cni/bin/tuning /opt/cni/bin/macvlan /opt/cni/bin/flannel /opt/cni/bin/cnitool /opt/cni/bin/ptp /opt/cni/bin/loopback /opt/cni/bin/ipvlan /opt/cni/bin/noop

Fixed by downloading portmap to /opt/cni/bin
https://github.com/projectcalico/cni-plugin/releases/download/v1.9.1/portmap

magic7s added a commit to magic7s/ansible-kubeadm that referenced this issue Dec 3, 2017

Due to bug in coreos/flannel (coreos/flannel#890) the downloaded temp…
…late from

https://raw.githubusercontent.com/coreos/flannel/master/Documentation/kube-flannel.yml requires portmap.
Because flannel requires but kubernetes-cni does not install, it needs to be manually copied to /opt/cni/bin/portmap
If portmap is not in place the symptom is the kube-dns will be stuck in CreatingContainer.
This additional task should be temp until either flannel no longer requires portmap or kubernetes-cni installs by default.
@osoriano

This comment has been minimized.

Copy link
Contributor

osoriano commented Dec 5, 2017

Sounds like there are two workarounds.

  1. Use the 0.9.1 kube-flannel.yml
  2. Update the CNI plugin installation on the host

AFAIK we don't package CNI plugins into the flannel image. Maybe we should update the docs? Sorry for the breakage, the portmap plugin is used for hostPort support

@tomdee

This comment has been minimized.

Copy link
Member

tomdee commented Dec 7, 2017

@osoriano Thanks for the summary.

IIUC this is only a problem for people using the kube-flannel.yml from master. It would be great to find a way to stop people from doing that!

And it would also be great if flannel had a better way of ensuring that the CNI plugins it needs are installed on the host, maybe https://github.com/coreos/flannel-cni could be updated to install the portmap plugin

@klausenbusk

This comment has been minimized.

Copy link

klausenbusk commented Dec 8, 2017

maybe https://github.com/coreos/flannel-cni could be updated to install the portmap plugin

https://github.com/coreos/flannel-cni already install the portmap plugin and is used by bootkube where the portmap plugin is enabled.

@cmoscardi

This comment has been minimized.

Copy link

cmoscardi commented Dec 8, 2017

+1, just ran into this

@ghost

This comment has been minimized.

Copy link

ghost commented Dec 9, 2017

I have used flannel-cni:v0.3.0 as initContainer to copy portmap (and flannel!) from the container to the host and ... portmapping is still not working, hostPort has no effect
I was checking with the netstat -lptn command and it was not showing me open ports, I think because they are in the different network namespace. Accessing the host on the hostPort from the outside works as expected.

@klausenbusk

This comment has been minimized.

Copy link

klausenbusk commented Dec 9, 2017

I was checking with the netstat -lptn command and it was not showing me open ports, I think because they are in the different network namespace. Accessing the host on the hostPort from the outside works as expected.

portmap works by creating a iptables rule.. Check with iptables-save | grep <port> or something like that.

@ghost

This comment has been minimized.

Copy link

ghost commented Dec 10, 2017

@klausenbusk after I discovered that hostPort is accessible, I have also used iptables to check the rules. Just didn't mention this in the previous comment.

@fengyd2018

This comment has been minimized.

Copy link

fengyd2018 commented Feb 12, 2018

The 0.9.1 kube-flannel.yml is used, but hostPort still cannot work.
Any extra work is needed?

Enviroment:
Linux master 3.10.0-693.el7.x86_64 #1 SMP Tue Aug 22 21:09:27 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux

CentOS Linux release 7.4.1708 (Core)

[root@master ~]# kubectl version
Client Version: version.Info{Major:"1", Minor:"9", GitVersion:"v1.9.2", GitCommit:"5fa2db2bd46ac79e5e00a4e6ed24191080aa463b", GitTreeState:"clean", BuildDate:"2018-01-18T10:09:24Z", GoVersion:"go1.9.2", Compiler:"gc", Platform:"linux/amd64"}
Server Version: version.Info{Major:"1", Minor:"9", GitVersion:"v1.9.3", GitCommit:"d2835416544f298c919e2ead3be3d0864b52323b", GitTreeState:"clean", BuildDate:"2018-02-07T11:55:20Z", GoVersion:"go1.9.2", Compiler:"gc", Platform:"linux/amd64"}

@ghost

This comment has been minimized.

Copy link

ghost commented Feb 12, 2018

The 0.9.1 kube-flannel.yml is used

You mean this file? As you can clearly see, the portmapping is not enabled in the ConfigMap. Cf.

cni-conf.json: |
{
"name": "cbr0",
"plugins": [
{
"type": "flannel",
"delegate": {
"hairpinMode": true,
"isDefaultGateway": true
}
},
{
"type": "portmap",
"capabilities": {
"portMappings": true
}
}
]
}

@fengyd2018

This comment has been minimized.

Copy link

fengyd2018 commented Feb 12, 2018

Thanks for your answer.

I tried with portmappign enabled, but hostPort cannot work.
https://raw.githubusercontent.com/coreos/flannel/master/Documentation/kube-flannel.yml

I just reset the kubernetes cluster and install it again, and hostPort can work now.

@fengyd2018

This comment has been minimized.

Copy link

fengyd2018 commented Feb 12, 2018

I think the kubernetes docs should be updated
https://kubernetes.io/docs/setup/independent/create-cluster-kubeadm/

In (3/4) Installing a pod network, the flannel version is still v0.9.1.
kubectl apply -f https://raw.githubusercontent.com/coreos/flannel/v0.9.1/Documentation/kube-flannel.yml

@wangjunwei87

This comment has been minimized.

Copy link

wangjunwei87 commented Aug 10, 2018

I encounter the same problem, any updates?

@wangjunwei87

This comment has been minimized.

Copy link

wangjunwei87 commented Aug 10, 2018

Download cni plugins binaries and put it in /opt/cni/bin solve my problem.

@aronica

This comment has been minimized.

Copy link

aronica commented Oct 16, 2018

Download cni plugins binaries and put it in /opt/cni/bin solve my problem.

This works for me with kubelet version v1.8.1

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment