Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

kube-proxy:1.17 for s390x image is broken? #87197

Open
cheeye opened this issue Jan 14, 2020 · 28 comments
Open

kube-proxy:1.17 for s390x image is broken? #87197

cheeye opened this issue Jan 14, 2020 · 28 comments

Comments

@cheeye
Copy link

@cheeye cheeye commented Jan 14, 2020

What happened:
When I tried to use kubernetes 1.17 for s390x with kubeadm, I got below messages in the kube-proxy pod. It ended up failure of installing cni like flannel. But if I switch back to kubernetes 1.15, everything works fine without these messages. Checked it with sig-network in the k8s slack, there could be something broken in the image kube-proxy:1.17 for s390x.

E0108 13:16:53.070052       1 proxier.go:787] Failed to ensure that filter chain KUBE-EXTERNAL-SERVICES exists: error creating chain "KUBE-EXTERNAL-SERVICES": exit status 2: update-alternatives: error: alternative /usr/sbin/iptables-legacy for iptables not registered; not setting
I0108 13:16:53.070129       1 proxier.go:779] Sync failed; retrying in 30s
W0108 13:17:22.902894       1 iptables.go:508] Could not set up iptables canary mangle/KUBE-PROXY-CANARY: error creating chain "KUBE-PROXY-CANARY": exit status 2: update-alternatives: error: alternative /usr/sbin/iptables-legacy for iptables not registered; not setting

What you expected to happen:

How to reproduce it (as minimally and precisely as possible):

Anything else we need to know?:

Environment:

  • Kubernetes version (use kubectl version): 1.17
  • Cloud provider or hardware configuration: s390x
  • OS (e.g: cat /etc/os-release): rhel7.7
  • Kernel (e.g. uname -a): 3.10.0-1062.9.1.el7.s390x #1 SMP Mon Dec 2 08:33:00 EST 2019 s390x s390x s390x GNU/Linux
  • Install tools: kubeadm
  • Network plugin and version (if this is a network-related bug): flannel
  • Others: image name k8s.gcr.io/kube-proxy:v1.17.0
@cheeye

This comment has been minimized.

Copy link
Author

@cheeye cheeye commented Jan 14, 2020

/sig network

@k8s-ci-robot k8s-ci-robot added sig/network and removed needs-sig labels Jan 14, 2020
@cheeye

This comment has been minimized.

Copy link
Author

@cheeye cheeye commented Jan 14, 2020

Please let me know if I put the improper sig group name. Thanks.

@athenabot

This comment has been minimized.

Copy link

@athenabot athenabot commented Jan 14, 2020

/triage unresolved

Comment /remove-triage unresolved when the issue is assessed and confirmed.

🤖 I am a bot run by vllry. 👩‍🔬

@neolit123

This comment has been minimized.

Copy link
Member

@neolit123 neolit123 commented Jan 15, 2020

to enable higher verbosity try adding e.g. --v=5 to the kube-proxy Pod template in the DaemonSet.

/usr/sbin/iptables-legacy

do you have the binary?

@cheeye

This comment has been minimized.

Copy link
Author

@cheeye cheeye commented Jan 15, 2020

@neolit123 I don't have that binary in the host.

@neolit123

This comment has been minimized.

Copy link
Member

@neolit123 neolit123 commented Jan 15, 2020

@cheeye

This comment has been minimized.

Copy link
Author

@cheeye cheeye commented Jan 15, 2020

Are you referring to update-alternatives --set iptables /usr/sbin/iptables-legacy in the doc link you sent? Is it supposed to be done in the kube-proxy pod/container or in the host? There is no iptables-legacy binary in the host or available package for RHEL. So I guess it won't work with command update-alternatives. Please let me know if I understand it incorrectly. Thanks!

@neolit123

This comment has been minimized.

Copy link
Member

@neolit123 neolit123 commented Jan 15, 2020

should be done on the host.

@neolit123

This comment has been minimized.

Copy link
Member

@neolit123 neolit123 commented Jan 15, 2020

or you can try kube-proxy in IPVS mode.

$ kubeadm init --config config.yaml

# config.yaml
apiVersion: kubeadm.k8s.io/v1beta2
kind: ClusterConfiguration
kubernetesVersion: v1.17.0
---
apiVersion: kubeproxy.config.k8s.io/v1alpha1
kind: KubeProxyConfiguration
mode: ipvs
@cheeye

This comment has been minimized.

Copy link
Author

@cheeye cheeye commented Jan 15, 2020

Maybe I didn't do it correctly. The ipvs mode is not working.

W0115 13:32:00.595540       1 iptables.go:165] Error checking iptables version, assuming version at least 1.4.11: exit status 2
W0115 13:32:00.595932       1 proxier.go:608] Failed to load kernel module ip_vs with modprobe. You can ignore this message when kube-proxy is running inside container without mounting /lib/modules
W0115 13:32:00.595961       1 proxier.go:608] Failed to load kernel module ip_vs_rr with modprobe. You can ignore this message when kube-proxy is running inside container without mounting /lib/modules
W0115 13:32:00.595990       1 proxier.go:608] Failed to load kernel module ip_vs_wrr with modprobe. You can ignore this message when kube-proxy is running inside container without mounting /lib/modules
W0115 13:32:00.596014       1 proxier.go:608] Failed to load kernel module ip_vs_sh with modprobe. You can ignore this message when kube-proxy is running inside container without mounting /lib/modules
W0115 13:32:00.596044       1 proxier.go:608] Failed to load kernel module nf_conntrack_ipv4 with modprobe. You can ignore this message when kube-proxy is running inside container without mounting /lib/modules
W0115 13:32:00.599672       1 proxier.go:608] Failed to load kernel module ip_vs with modprobe. You can ignore this message when kube-proxy is running inside container without mounting /lib/modules
W0115 13:32:00.599705       1 proxier.go:608] Failed to load kernel module ip_vs_rr with modprobe. You can ignore this message when kube-proxy is running inside container without mounting /lib/modules
W0115 13:32:00.599739       1 proxier.go:608] Failed to load kernel module ip_vs_wrr with modprobe. You can ignore this message when kube-proxy is running inside container without mounting /lib/modules
W0115 13:32:00.599772       1 proxier.go:608] Failed to load kernel module ip_vs_sh with modprobe. You can ignore this message when kube-proxy is running inside container without mounting /lib/modules
W0115 13:32:00.599817       1 proxier.go:608] Failed to load kernel module nf_conntrack_ipv4 with modprobe. You can ignore this message when kube-proxy is running inside container without mounting /lib/modules
E0115 13:32:00.599883       1 server_others.go:333] can't determine whether to use ipvs proxy, error: error getting ipset version, error: executable file not found in $PATH
I0115 13:32:00.609123       1 node.go:135] Successfully retrieved node IP: x.x.x.x
I0115 13:32:00.609198       1 server_others.go:145] Using iptables Proxier.
W0115 13:32:00.609297       1 proxier.go:286] clusterCIDR not specified, unable to distinguish between internal and external traffic
I0115 13:32:00.610203       1 server.go:571] Version: v1.17.0
I0115 13:32:00.610653       1 conntrack.go:52] Setting nf_conntrack_max to 131072
I0115 13:32:00.610932       1 config.go:313] Starting service config controller
I0115 13:32:00.610955       1 shared_informer.go:197] Waiting for caches to sync for service config
I0115 13:32:00.611002       1 config.go:131] Starting endpoints config controller
I0115 13:32:00.611013       1 shared_informer.go:197] Waiting for caches to sync for endpoints config
W0115 13:32:00.620381       1 iptables.go:508] Could not set up iptables canary mangle/KUBE-PROXY-CANARY: error creating chain "KUBE-PROXY-CANARY": exit status 2: update-alternatives: error: alternative /usr/sbin/iptables-legacy for iptables not registered; not setting
I0115 13:32:00.711063       1 shared_informer.go:204] Caches are synced for endpoints config
I0115 13:32:00.711159       1 shared_informer.go:204] Caches are synced for service config
E0115 13:32:00.715844       1 proxier.go:787] Failed to ensure that filter chain KUBE-EXTERNAL-SERVICES exists: error creating chain "KUBE-EXTERNAL-SERVICES": exit status 2: update-alternatives: error: alternative /usr/sbin/iptables-legacy for iptables not registered; not setting
I0115 13:32:00.715857       1 proxier.go:779] Sync failed; retrying in 30s

I have the ipset package installed.

# lsmod | grep -e ip_vs -e nf_conntrack_ipv4
nf_conntrack_ipv4      20028  2
nf_defrag_ipv4         12826  1 nf_conntrack_ipv4
ip_vs_sh               12843  0
ip_vs_wrr              12832  0
ip_vs_rr               12755  0
ip_vs                 196247  6 ip_vs_rr,ip_vs_sh,ip_vs_wrr
nf_conntrack          188126  6 ip_vs,nf_nat,nf_nat_ipv4,xt_conntrack,nf_nat_masquerade_ipv4,nf_conntrack_ipv4
libcrc32c              12771  4 xfs,ip_vs,nf_nat,nf_conntrack
@cheeye

This comment has been minimized.

Copy link
Author

@cheeye cheeye commented Jan 15, 2020

And I checked the release info of the rhel7. It says the nftables is supported officially starting from rhel7.6. But it's not enabled by default.. I have to install nftables related package explicitly, which means the default iptables in rhel7 is still the "legacy" one.

@neolit123

This comment has been minimized.

Copy link
Member

@neolit123 neolit123 commented Jan 15, 2020

looking at the logs above, for IPVS you seem to be missing critical kernel modules and the ipset binary. so you either have to find them for the platform or rebuild from source.

And I checked the release info of the rhel7. It says the nftables is supported officially starting from rhel7.6. But it's not enabled by default.. I have to install nftables related package explicitly, which means the default iptables in rhel7 is still the "legacy" one.

if that is the case, the kube-proxy maintainers should confirm, but my guess is that this new version of kube-proxy assumes that it needs the /usr/sbin/iptables-legacy binary on newer distros and RHEL 7.7 as a relatively new distro still has legacy mode as the default.

so maybe this can be considered as a bug in kube-proxy?
you can try asking in the #sig-network channel on k8s slack.

alternative /usr/sbin/iptables-legacy for iptables not registered; not setting

you can try workarounding the problem by creating a symbolic link from /usr/sbin/iptables-legacy to your existing iptables binary.

@cheeye

This comment has been minimized.

Copy link
Author

@cheeye cheeye commented Jan 15, 2020

Thanks, @neolit123 . Actually, the #sig-network channel asked me to open an issue here as they think something is broken in the kube-proxy 1.17.. So what am I supposed to do to report this issue/bug?

@neolit123

This comment has been minimized.

Copy link
Member

@neolit123 neolit123 commented Jan 15, 2020

someone from SIG Network needs to find this issue and investigate if it's an actual bug.

you can try workarounding the problem by creating a symbolic link from /usr/sbin/iptables-legacy to your existing iptables binary.

try this still.

@cheeye

This comment has been minimized.

Copy link
Author

@cheeye cheeye commented Jan 15, 2020

OK. I will contact the sig-network.

As for creating a symbolic link from /usr/sbin/iptables-legacy, it still doesn't work.

W0115 14:25:56.224700       1 iptables.go:165] Error checking iptables version, assuming version at least 1.4.11: exit status 2
W0115 14:25:56.225089       1 proxier.go:608] Failed to load kernel module ip_vs with modprobe. You can ignore this message when kube-proxy is running inside container without mounting /lib/modules
W0115 14:25:56.225117       1 proxier.go:608] Failed to load kernel module ip_vs_rr with modprobe. You can ignore this message when kube-proxy is running inside container without mounting /lib/modules
W0115 14:25:56.225142       1 proxier.go:608] Failed to load kernel module ip_vs_wrr with modprobe. You can ignore this message when kube-proxy is running inside container without mounting /lib/modules
W0115 14:25:56.225167       1 proxier.go:608] Failed to load kernel module ip_vs_sh with modprobe. You can ignore this message when kube-proxy is running inside container without mounting /lib/modules
W0115 14:25:56.225193       1 proxier.go:608] Failed to load kernel module nf_conntrack_ipv4 with modprobe. You can ignore this message when kube-proxy is running inside container without mounting /lib/modules
W0115 14:25:56.227914       1 server_others.go:323] Unknown proxy mode "", assuming iptables proxy
I0115 14:25:56.237178       1 node.go:135] Successfully retrieved node IP: x.x.x.x
I0115 14:25:56.237199       1 server_others.go:145] Using iptables Proxier.
I0115 14:25:56.237442       1 server.go:571] Version: v1.17.1
I0115 14:25:56.237872       1 conntrack.go:52] Setting nf_conntrack_max to 131072
I0115 14:25:56.239292       1 config.go:131] Starting endpoints config controller
I0115 14:25:56.239323       1 shared_informer.go:197] Waiting for caches to sync for endpoints config
I0115 14:25:56.239368       1 config.go:313] Starting service config controller
I0115 14:25:56.239384       1 shared_informer.go:197] Waiting for caches to sync for service config
W0115 14:25:56.248201       1 iptables.go:508] Could not set up iptables canary mangle/KUBE-PROXY-CANARY: error creating chain "KUBE-PROXY-CANARY": exit status 2: update-alternatives: error: alternative /usr/sbin/iptables-legacy for iptables not registered; not setting
I0115 14:25:56.339468       1 shared_informer.go:204] Caches are synced for service config
I0115 14:25:56.339592       1 shared_informer.go:204] Caches are synced for endpoints config
E0115 14:25:56.344110       1 proxier.go:787] Failed to ensure that filter chain KUBE-EXTERNAL-SERVICES exists: error creating chain "KUBE-EXTERNAL-SERVICES": exit status 2: update-alternatives: error: alternative /usr/sbin/iptables-legacy for iptables not registered; not setting
I0115 14:25:56.344164       1 proxier.go:779] Sync failed; retrying in 30s
@franznemeth

This comment has been minimized.

Copy link

@franznemeth franznemeth commented Jan 15, 2020

I have the exact same problem on ubuntu 18.04.3 s390x (4.15.0-74-generic #84-Ubuntu SMP Thu Dec 19 08:05:42 UTC 2019 s390x s390x s390x GNU/Linux).
Also using v1.17 via kubeadm.

@aojea

This comment has been minimized.

Copy link
Member

@aojea aojea commented Jan 15, 2020

The image used in kube-proxy is created from https://github.com/kubernetes/kubernetes/tree/master/build/debian-iptables
based in a debian-buster image with all the binaries included.

Is is possible for any of you with the s390x to try to build the image locally and find out if the problem is that the base images don't include the /usr/sbin/iptables-legacy binary?

@neolit123

This comment has been minimized.

Copy link
Member

@neolit123 neolit123 commented Jan 15, 2020

As for creating a symbolic link from /usr/sbin/iptables-legacy, it still doesn't work.

the symlink workaround is for iptables, not IPVS. so don't pass mode: ipvs in that case.

@franznemeth

This comment has been minimized.

Copy link

@franznemeth franznemeth commented Jan 15, 2020

I've built from master with make sub-build-s390x:

docker run --rm staging-k8s.gcr.io/debian-iptables-s390x:v12.0.1 /usr/sbin/iptables-legacy
iptables v1.8.3 (legacy): no command specified
Try `iptables -h' or 'iptables --help' for more information.

The binary is definitly in the base image

@aojea

This comment has been minimized.

Copy link
Member

@aojea aojea commented Jan 15, 2020

can you run the wrapper script /usr/sbin/iptables-wrapper?

@franznemeth

This comment has been minimized.

Copy link

@franznemeth franznemeth commented Jan 15, 2020

It's running with no output

@aojea

This comment has been minimized.

Copy link
Member

@aojea aojea commented Jan 15, 2020

Can you obtain a shell int the container and try to run iptables -L directly?

this is my output in x64

docker run -it --privileged k8s.gcr.io/kube-proxy:v1.17.0 sh
Unable to find image 'k8s.gcr.io/kube-proxy:v1.17.0' locally
v1.17.0: Pulling from kube-proxy
597de8ba0c30: Already exists 
3f0663684f29: Already exists 
e1f7f878905c: Already exists 
3029977cf65d: Already exists 
cc627398eeaa: Already exists 
d3609306ce38: Already exists 
d7c1c982f192: Pull complete 
Digest: sha256:b2ba9441af30261465e5c41be63e462d0050b09ad280001ae731f399b2b00b75
Status: Downloaded newer image for k8s.gcr.io/kube-proxy:v1.17.0
# iptables -L
Chain INPUT (policy ACCEPT)
target     prot opt source               destination         

Chain FORWARD (policy ACCEPT)
target     prot opt source               destination         

Chain OUTPUT (policy ACCEPT)
target     prot opt source               destination         

@cheeye

This comment has been minimized.

Copy link
Author

@cheeye cheeye commented Jan 16, 2020

Can you obtain a shell int the container and try to run iptables -L directly?

this is my output in x64

docker run -it --privileged k8s.gcr.io/kube-proxy:v1.17.0 sh
Unable to find image 'k8s.gcr.io/kube-proxy:v1.17.0' locally
v1.17.0: Pulling from kube-proxy
597de8ba0c30: Already exists 
3f0663684f29: Already exists 
e1f7f878905c: Already exists 
3029977cf65d: Already exists 
cc627398eeaa: Already exists 
d3609306ce38: Already exists 
d7c1c982f192: Pull complete 
Digest: sha256:b2ba9441af30261465e5c41be63e462d0050b09ad280001ae731f399b2b00b75
Status: Downloaded newer image for k8s.gcr.io/kube-proxy:v1.17.0
# iptables -L
Chain INPUT (policy ACCEPT)
target     prot opt source               destination         

Chain FORWARD (policy ACCEPT)
target     prot opt source               destination         

Chain OUTPUT (policy ACCEPT)
target     prot opt source               destination         

From the official image, I got below:

# docker run -it --privileged k8s.gcr.io/kube-proxy:v1.17.0 sh
# iptables -L
update-alternatives: error: alternative /usr/sbin/iptables-legacy for iptables not registered; not setting
@cheeye

This comment has been minimized.

Copy link
Author

@cheeye cheeye commented Jan 16, 2020

I tried to build my local image in s390x and run a container. Below is the result.

# docker run -it --privileged staging-k8s.gcr.io/debian-iptables-s390x:v12.0.1 sh
# iptables -L
Chain INPUT (policy ACCEPT)
target     prot opt source               destination

Chain FORWARD (policy ACCEPT)
target     prot opt source               destination

Chain OUTPUT (policy ACCEPT)
target     prot opt source               destination
@aojea

This comment has been minimized.

Copy link
Member

@aojea aojea commented Jan 16, 2020

so it seems that the cross-compiling toolchain of the image/s is not working, right?

@franznemeth

This comment has been minimized.

Copy link

@franznemeth franznemeth commented Jan 16, 2020

Looks like it, sadly I can't get the full dockerized build to work. The build container is x86 for some reason

@cheeye

This comment has been minimized.

Copy link
Author

@cheeye cheeye commented Jan 17, 2020

so it seems that the cross-compiling toolchain of the image/s is not working, right?

Look like.. Which group can we contact and report?

@neolit123

This comment has been minimized.

Copy link
Member

@neolit123 neolit123 commented Jan 17, 2020

/sig release

the owners of the build are sig-release, but ideally sig-network should help with sending PRs for this.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
6 participants
You can’t perform that action at this time.