Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Networking issues with Raspbian Lite / RPi4 #703

Open
alexellis opened this issue Aug 3, 2019 · 17 comments
Open

Networking issues with Raspbian Lite / RPi4 #703

alexellis opened this issue Aug 3, 2019 · 17 comments
Assignees

Comments

@alexellis
Copy link

@alexellis alexellis commented Aug 3, 2019

Describe the bug
A clear and concise description of what the bug is.

I flashed Raspbian Lite for my Raspberry Pi 4 (4GB) and updated / upgraded packages with apt:

Linux k4s-3 4.19.58-v7l+ #1245    SMP Fri Jul 12 17:31:45 BST 2019 armv7l GNU/Linux

I did this on 3 separate RPis and ran the curl / sh script to install k3s as a server on each. I then tried to access Traefik on port 80 with the Node's IP (using my laptop). It did not respond.

If I use curl from on the RPi it does respond with 404 not found.

I then deployed OpenFaaS with the following:

git clone https://github.com/openfaas/faas-netes && cd faas-netes && sudo kubectl apply -f namespaces.yml,yaml_armhf

The pods which rely on networking failed to start, namely the ones which talk to Prometheus on start-up, or the ones which talk to NATS on start-up.

sudo kubectl get pods -A

Screenshot 2019-08-03 at 22 51 47

Screenshot 2019-08-03 at 22 53 41

Additional context
Add any other context about the problem here.

This worked well for me every time I tried it previously. The only thing which I think is different is having run sudo apt upgrade. I did not run the firmware update command.

@ibuildthecloud

This comment has been minimized.

Copy link
Member

@ibuildthecloud ibuildthecloud commented Aug 5, 2019

Is DNS inside the pods working?

@alexellis

This comment has been minimized.

Copy link
Author

@alexellis alexellis commented Aug 5, 2019

I can power the device up and run a quick command if you have something handy for me to test with?

@alexellis

This comment has been minimized.

Copy link
Author

@alexellis alexellis commented Aug 6, 2019

@ibuildthecloud DNS isn't working and IP connectivity isn't working either.

CoreDNS appears to be started.

 lsmod
Module                  Size  Used by
ipt_REJECT             16384  0
nf_reject_ipv4         16384  1 ipt_REJECT
xt_multiport           16384  1
veth                   24576  0
xt_nat                 16384  11
xt_tcpudp              16384  37
vxlan                  49152  0
ip6_udp_tunnel         16384  1 vxlan
udp_tunnel             16384  1 vxlan
nft_chain_nat_ipv6     16384  4
nf_nat_ipv6            20480  1 nft_chain_nat_ipv6
xt_comment             16384  45
iptable_filter         16384  1
xt_mark                16384  7
iptable_nat            16384  2
ip_vs_sh               16384  0
ip_vs_wrr              16384  0
ip_vs_rr               16384  0
ip_vs                 143360  6 ip_vs_wrr,ip_vs_rr,ip_vs_sh
ipt_MASQUERADE         16384  7
nf_conntrack_netlink    40960  0
nft_counter            16384  15
nft_chain_nat_ipv4     16384  4
nf_nat_ipv4            16384  3 nft_chain_nat_ipv4,ipt_MASQUERADE,iptable_nat
xt_addrtype            16384  5
nft_compat             20480  4
nf_tables             122880  46 nft_chain_nat_ipv6,nft_chain_nat_ipv4,nft_compat,nft_counter
nfnetlink              16384  4 nft_compat,nf_conntrack_netlink,nf_tables
xt_conntrack           16384  7
nf_nat                 36864  3 xt_nat,nf_nat_ipv6,nf_nat_ipv4
nf_conntrack          135168  8 ip_vs,xt_nat,ipt_MASQUERADE,nf_conntrack_netlink,nf_nat_ipv6,xt_conntrack,nf_nat_ipv4,nf_nat
nf_defrag_ipv6         20480  1 nf_conntrack
nf_defrag_ipv4         16384  1 nf_conntrack
br_netfilter           24576  0
bridge                135168  1 br_netfilter
overlay               106496  11
bnep                   20480  2
hci_uart               40960  1
btbcm                  16384  1 hci_uart
serdev                 20480  1 hci_uart
bluetooth             389120  24 hci_uart,bnep,btbcm
ecdh_generic           28672  1 bluetooth
8021q                  32768  0
garp                   16384  1 8021q
stp                    16384  2 garp,bridge
llc                    16384  3 garp,bridge,stp
brcmfmac              311296  0
brcmutil               16384  1 brcmfmac
sha256_generic         20480  0
vc4                   172032  0
cfg80211              614400  1 brcmfmac
drm_kms_helper        184320  1 vc4
bcm2835_codec          36864  0
rfkill                 28672  6 bluetooth,cfg80211
raspberrypi_hwmon      16384  0
bcm2835_v4l2           45056  0
hwmon                  16384  1 raspberrypi_hwmon
v4l2_mem2mem           24576  1 bcm2835_codec
bcm2835_mmal_vchiq     32768  2 bcm2835_codec,bcm2835_v4l2
v4l2_common            16384  1 bcm2835_v4l2
videobuf2_dma_contig    20480  1 bcm2835_codec
videobuf2_vmalloc      16384  1 bcm2835_v4l2
videobuf2_memops       16384  2 videobuf2_dma_contig,videobuf2_vmalloc
v3d                    61440  0
videobuf2_v4l2         24576  3 bcm2835_codec,bcm2835_v4l2,v4l2_mem2mem
snd_soc_core          192512  1 vc4
videobuf2_common       45056  4 bcm2835_codec,bcm2835_v4l2,v4l2_mem2mem,videobuf2_v4l2
gpu_sched              28672  1 v3d
snd_compress           20480  1 snd_soc_core
videodev              200704  6 bcm2835_codec,v4l2_common,videobuf2_common,bcm2835_v4l2,v4l2_mem2mem,videobuf2_v4l2
snd_pcm_dmaengine      16384  1 snd_soc_core
media                  36864  2 videodev,v4l2_mem2mem
syscopyarea            16384  1 drm_kms_helper
vc_sm_cma              36864  1 bcm2835_mmal_vchiq
sysfillrect            16384  1 drm_kms_helper
drm                   438272  5 v3d,vc4,gpu_sched,drm_kms_helper
sysimgblt              16384  1 drm_kms_helper
fb_sys_fops            16384  1 drm_kms_helper
drm_panel_orientation_quirks    16384  1 drm
snd_bcm2835            24576  1
snd_pcm               102400  4 vc4,snd_pcm_dmaengine,snd_bcm2835,snd_soc_core
snd_timer              32768  1 snd_pcm
snd                    73728  7 snd_compress,snd_timer,snd_bcm2835,snd_soc_core,snd_pcm
argon_mem              16384  0
uio_pdrv_genirq        16384  0
fixed                  16384  0
uio                    20480  1 uio_pdrv_genirq
ip_tables              24576  2 iptable_filter,iptable_nat
x_tables               32768  12 xt_comment,xt_multiport,ipt_REJECT,xt_nat,ip_tables,nft_compat,iptable_filter,xt_mark,xt_tcpudp,ipt_MASQUERADE,xt_addrtype,xt_conntrack
ipv6                  450560  249 nf_nat_ipv6,bridge

Alex

@erikwilson

This comment has been minimized.

Copy link
Member

@erikwilson erikwilson commented Aug 6, 2019

Thanks for filing this issue @alexellis!

I am trying to replicate on a Raspberry Pi 2 (armv7) but am having a hard time doing so. I deployed OpenFaas using the commands you provided:

root@k3s-base:~/faas-netes# kubectl get all -A
NAMESPACE     NAME                                READY   STATUS      RESTARTS   AGE
kube-system   pod/coredns-b7464766c-cckf5         1/1     Running     0          10m
kube-system   pod/helm-install-traefik-w2d7g      0/1     Completed   0          10m
kube-system   pod/svclb-traefik-xm5c4             2/2     Running     0          8m26s
kube-system   pod/traefik-56688c4464-mhv7x        1/1     Running     0          8m25s
openfaas      pod/alertmanager-757cc474bc-6rqfw   1/1     Running     0          7m11s
openfaas      pod/faas-idler-59dfd85f6c-vgnhm     1/1     Running     2          7m12s
openfaas      pod/gateway-597f6578bc-7v9ch        2/2     Running     0          7m12s
openfaas      pod/nats-d4c9d8d95-bwfwf            1/1     Running     0          7m11s
openfaas      pod/prometheus-68d68d7466-xwclk     1/1     Running     0          7m9s
openfaas      pod/queue-worker-df9d5749c-zgfrh    1/1     Running     0          7m9s

NAMESPACE     NAME                       TYPE           CLUSTER-IP      EXTERNAL-IP   PORT(S)                      AGE
default       service/kubernetes         ClusterIP      10.43.0.1       <none>        443/TCP                      10m
kube-system   service/kube-dns           ClusterIP      10.43.0.10      <none>        53/UDP,53/TCP,9153/TCP       10m
kube-system   service/traefik            LoadBalancer   10.43.48.154    10.20.3.70    80:31121/TCP,443:31253/TCP   8m26s
openfaas      service/alertmanager       ClusterIP      10.43.96.238    <none>        9093/TCP                     7m12s
openfaas      service/gateway            ClusterIP      10.43.61.241    <none>        8080/TCP                     7m12s
openfaas      service/gateway-external   NodePort       10.43.106.66    <none>        8080:31112/TCP               7m12s
openfaas      service/nats               ClusterIP      10.43.107.135   <none>        4222/TCP                     7m11s
openfaas      service/prometheus         ClusterIP      10.43.121.9     <none>        9090/TCP                     7m10s

NAMESPACE     NAME                           DESIRED   CURRENT   READY   UP-TO-DATE   AVAILABLE   NODE SELECTOR   AGE
kube-system   daemonset.apps/svclb-traefik   1         1         1       1            1           <none>          8m26s

NAMESPACE     NAME                           READY   UP-TO-DATE   AVAILABLE   AGE
kube-system   deployment.apps/coredns        1/1     1            1           10m
kube-system   deployment.apps/traefik        1/1     1            1           8m26s
openfaas      deployment.apps/alertmanager   1/1     1            1           7m12s
openfaas      deployment.apps/faas-idler     1/1     1            1           7m12s
openfaas      deployment.apps/gateway        1/1     1            1           7m12s
openfaas      deployment.apps/nats           1/1     1            1           7m11s
openfaas      deployment.apps/prometheus     1/1     1            1           7m11s
openfaas      deployment.apps/queue-worker   1/1     1            1           7m10s

NAMESPACE     NAME                                      DESIRED   CURRENT   READY   AGE
kube-system   replicaset.apps/coredns-b7464766c         1         1         1       10m
kube-system   replicaset.apps/traefik-56688c4464        1         1         1       8m26s
openfaas      replicaset.apps/alertmanager-757cc474bc   1         1         1       7m12s
openfaas      replicaset.apps/faas-idler-59dfd85f6c     1         1         1       7m12s
openfaas      replicaset.apps/gateway-597f6578bc        1         1         1       7m12s
openfaas      replicaset.apps/nats-d4c9d8d95            1         1         1       7m11s
openfaas      replicaset.apps/prometheus-68d68d7466     1         1         1       7m10s
openfaas      replicaset.apps/queue-worker-df9d5749c    1         1         1       7m9s

NAMESPACE     NAME                             COMPLETIONS   DURATION   AGE
kube-system   job.batch/helm-install-traefik   1/1           101s       10m

The logs for CoreDNS:

root@k3s-base:~/faas-netes# kubectl logs -n kube-system deployment.apps/coredns
.:53
2019-08-06T18:39:47.494Z [INFO] CoreDNS-1.3.0
2019-08-06T18:39:47.495Z [INFO] linux/arm, go1.11.4, c8f0e94
CoreDNS-1.3.0
linux/arm, go1.11.4, c8f0e94
2019-08-06T18:39:47.495Z [INFO] plugin/reload: Running configuration MD5 = ef347efee19aa82f09972f89f92da1cf

Following the instructions for testing DNS at https://kubernetes.io/docs/tasks/administer-cluster/dns-debugging-resolution/

root@k3s-base:~/faas-netes# kubectl apply -f https://k8s.io/examples/admin/dns/busybox.yaml
pod/busybox created

And resolving the OpenFaas service:

root@k3s-base:~/faas-netes# kubectl exec -ti busybox -- nslookup nats.openfaas.svc.cluster.local
Server:    10.43.0.10
Address 1: 10.43.0.10 kube-dns.kube-system.svc.cluster.local

Name:      nats.openfaas.svc.cluster.local
Address 1: 10.43.107.135 nats.openfaas.svc.cluster.local

I am then able to access the portal through port 31112:
Screen Shot 2019-08-06 at 11 59 24 AM

@alexellis

This comment has been minimized.

Copy link
Author

@alexellis alexellis commented Aug 6, 2019

Thanks for spending some time on it. There are at least two things you are doing differently.

  1. I used RPi4
  2. I ran sudo apt upgrade -qy

I did mention the upgrade step in the original post. Upgrading your RPi3 packages may break, or this may only break with RPi4.

Alex

@alexellis

This comment has been minimized.

Copy link
Author

@alexellis alexellis commented Aug 6, 2019

Take a note of my uname -a output, can you try to compare it to yours before and after running sudo apt update && sudo apt upgrade --qy?

Linux k4s-3 4.19.58-v7l+ #1245 SMP Fri Jul 12 17:31:45 BST 2019 armv7l GNU/Linux
@erikwilson

This comment has been minimized.

Copy link
Member

@erikwilson erikwilson commented Aug 6, 2019

Ok, original uname -a:

Linux k3s-base 4.14.79-v7+ #1159 SMP Sun Nov 4 17:50:20 GMT 2018 armv7l GNU/Linux

After upgrading:

Linux k3s-base 4.19.57-v7+ #1244 SMP Thu Jul 4 18:45:25 BST 2019 armv7l GNU/Linux

Pods:

root@k3s-base:~# kubectl get pods -A
NAMESPACE     NAME                                READY   STATUS      RESTARTS   AGE
default       pod/busybox                         1/1     Running     3          157m
kube-system   pod/coredns-b7464766c-cckf5         1/1     Running     1          169m
kube-system   pod/helm-install-traefik-w2d7g      0/1     Completed   0          169m
kube-system   pod/svclb-traefik-xm5c4             2/2     Running     2          167m
kube-system   pod/traefik-56688c4464-mhv7x        1/1     Running     1          167m
openfaas      pod/alertmanager-757cc474bc-6rqfw   1/1     Running     1          166m
openfaas      pod/faas-idler-59dfd85f6c-vgnhm     1/1     Running     5          166m
openfaas      pod/gateway-597f6578bc-7v9ch        2/2     Running     3          166m
openfaas      pod/nats-d4c9d8d95-bwfwf            1/1     Running     1          166m
openfaas      pod/prometheus-68d68d7466-xwclk     1/1     Running     1          166m
openfaas      pod/queue-worker-df9d5749c-zgfrh    1/1     Running     2          166m

Retrying with a fresh install produced the same (working) result. Some of the pods entered CrashLoopBackOff state until everything stabilized, which took about 4m30s.

Sorry I don't have an RPi4 to test with, but if you could provide more info, like the CoreDNS logs and service resolve test that would be awesome.

It may be that the RPi just needs more time to bring everything up, and eventually all of your pods will stay in a running state.

@alexellis

This comment has been minimized.

Copy link
Author

@alexellis alexellis commented Aug 7, 2019

Please see these two examples.

Screenshot 2019-08-07 at 07 40 31

Screenshot 2019-08-07 at 07 40 27

Working RPi4:

  • only has k3s / git
  • didn't get sudo apt upgrade
  • Only pod that restarted was faas-idler twice

RPi4 with k3s which is not working:

  • has k3s and latest docker package
  • did get sudo apt upgrade
@alexellis

This comment has been minimized.

Copy link
Author

@alexellis alexellis commented Aug 7, 2019

Screenshot 2019-08-07 at 08 01 49

Still crashing after 22m, due to no DNS, no IP connectivity.

These are the modules loaded:

Module                  Size  Used by
ipt_REJECT             16384  3
nf_reject_ipv4         16384  1 ipt_REJECT
xt_multiport           16384  1
xt_nat                 16384  14
xt_tcpudp              16384  49
veth                   24576  0
nft_chain_nat_ipv6     16384  4
nf_nat_ipv6            20480  1 nft_chain_nat_ipv6
vxlan                  49152  0
ip6_udp_tunnel         16384  1 vxlan
udp_tunnel             16384  1 vxlan
xt_comment             16384  54
xt_mark                16384  7
iptable_nat            16384  2
iptable_filter         16384  1
ip_vs_sh               16384  0
ip_vs_wrr              16384  0
ip_vs_rr               16384  0
ip_vs                 143360  6 ip_vs_wrr,ip_vs_rr,ip_vs_sh
ipt_MASQUERADE         16384  7
nf_conntrack_netlink    40960  0
nft_counter            16384  15
nft_chain_nat_ipv4     16384  4
nf_nat_ipv4            16384  3 nft_chain_nat_ipv4,ipt_MASQUERADE,iptable_nat
xt_addrtype            16384  6
nft_compat             20480  4
nf_tables             122880  46 nft_chain_nat_ipv6,nft_chain_nat_ipv4,nft_compat,nft_counter
nfnetlink              16384  4 nft_compat,nf_conntrack_netlink,nf_tables
xt_conntrack           16384  7
nf_nat                 36864  3 xt_nat,nf_nat_ipv6,nf_nat_ipv4
nf_conntrack          135168  8 ip_vs,xt_nat,ipt_MASQUERADE,nf_conntrack_netlink,nf_nat_ipv6,xt_conntrack,nf_nat_ipv4,nf_nat
nf_defrag_ipv6         20480  1 nf_conntrack
nf_defrag_ipv4         16384  1 nf_conntrack
br_netfilter           24576  0
bridge                135168  1 br_netfilter
overlay               106496  21
bnep                   20480  2
hci_uart               40960  1
btbcm                  16384  1 hci_uart
serdev                 20480  1 hci_uart
bluetooth             389120  24 hci_uart,bnep,btbcm
ecdh_generic           28672  1 bluetooth
8021q                  32768  0
garp                   16384  1 8021q
stp                    16384  2 garp,bridge
llc                    16384  3 garp,bridge,stp
brcmfmac              311296  0
brcmutil               16384  1 brcmfmac
sha256_generic         20480  0
cfg80211              614400  1 brcmfmac
vc4                   172032  0
rfkill                 28672  6 bluetooth,cfg80211
v3d                    61440  0
drm_kms_helper        184320  1 vc4
gpu_sched              28672  1 v3d
bcm2835_codec          36864  0
drm                   438272  5 v3d,vc4,gpu_sched,drm_kms_helper
raspberrypi_hwmon      16384  0
drm_panel_orientation_quirks    16384  1 drm
bcm2835_v4l2           45056  0
v4l2_mem2mem           24576  1 bcm2835_codec
hwmon                  16384  1 raspberrypi_hwmon
bcm2835_mmal_vchiq     32768  2 bcm2835_codec,bcm2835_v4l2
videobuf2_dma_contig    20480  1 bcm2835_codec
v4l2_common            16384  1 bcm2835_v4l2
snd_soc_core          192512  1 vc4
videobuf2_vmalloc      16384  1 bcm2835_v4l2
videobuf2_memops       16384  2 videobuf2_dma_contig,videobuf2_vmalloc
videobuf2_v4l2         24576  3 bcm2835_codec,bcm2835_v4l2,v4l2_mem2mem
videobuf2_common       45056  4 bcm2835_codec,bcm2835_v4l2,v4l2_mem2mem,videobuf2_v4l2
snd_bcm2835            24576  1
snd_compress           20480  1 snd_soc_core
snd_pcm_dmaengine      16384  1 snd_soc_core
videodev              200704  6 bcm2835_codec,v4l2_common,videobuf2_common,bcm2835_v4l2,v4l2_mem2mem,videobuf2_v4l2
media                  36864  2 videodev,v4l2_mem2mem
vc_sm_cma              36864  1 bcm2835_mmal_vchiq
snd_pcm               102400  4 vc4,snd_pcm_dmaengine,snd_bcm2835,snd_soc_core
syscopyarea            16384  1 drm_kms_helper
sysfillrect            16384  1 drm_kms_helper
sysimgblt              16384  1 drm_kms_helper
snd_timer              32768  1 snd_pcm
fb_sys_fops            16384  1 drm_kms_helper
snd                    73728  7 snd_compress,snd_timer,snd_bcm2835,snd_soc_core,snd_pcm
argon_mem              16384  0
fixed                  16384  0
uio_pdrv_genirq        16384  0
uio                    20480  1 uio_pdrv_genirq
ip_tables              24576  2 iptable_filter,iptable_nat
x_tables               32768  12 xt_comment,xt_multiport,ipt_REJECT,xt_nat,ip_tables,nft_compat,iptable_filter,xt_mark,xt_tcpudp,ipt_MASQUERADE,xt_addrtype,xt_conntrack
ipv6                  450560  397 nf_nat_ipv6,bridge

vs. on the working unit:

Module                  Size  Used by
nft_chain_nat_ipv6     16384  4
nf_nat_ipv6            20480  1 nft_chain_nat_ipv6
nf_tables             122880  1 nft_chain_nat_ipv6
xt_multiport           16384  1
nf_conntrack_netlink    40960  0
nfnetlink              16384  2 nf_conntrack_netlink,nf_tables
veth                   24576  0
vxlan                  49152  0
ip6_udp_tunnel         16384  1 vxlan
udp_tunnel             16384  1 vxlan
xt_nat                 16384  16
xt_addrtype            16384  3
ipt_REJECT             16384  0
nf_reject_ipv4         16384  1 ipt_REJECT
xt_tcpudp              16384  52
ipt_MASQUERADE         16384  6
xt_conntrack           16384  6
xt_comment             16384  55
iptable_filter         16384  1
xt_mark                16384  7
iptable_nat            16384  2
nf_nat_ipv4            16384  2 ipt_MASQUERADE,iptable_nat
nf_nat                 36864  3 xt_nat,nf_nat_ipv6,nf_nat_ipv4
ip_vs_sh               16384  0
ip_vs_wrr              16384  0
ip_vs_rr               16384  0
ip_vs                 143360  6 ip_vs_wrr,ip_vs_rr,ip_vs_sh
nf_conntrack          135168  8 ip_vs,xt_nat,ipt_MASQUERADE,nf_conntrack_netlink,nf_nat_ipv6,xt_conntrack,nf_nat_ipv4,nf_nat
nf_defrag_ipv6         20480  1 nf_conntrack
nf_defrag_ipv4         16384  1 nf_conntrack
overlay               106496  22
br_netfilter           24576  0
bridge                135168  1 br_netfilter
bnep                   20480  2
hci_uart               40960  1
btbcm                  16384  1 hci_uart
serdev                 20480  1 hci_uart
bluetooth             389120  24 hci_uart,bnep,btbcm
ecdh_generic           28672  1 bluetooth
8021q                  32768  0
garp                   16384  1 8021q
stp                    16384  2 garp,bridge
llc                    16384  3 garp,bridge,stp
brcmfmac              311296  0
brcmutil               16384  1 brcmfmac
sha256_generic         20480  0
cfg80211              614400  1 brcmfmac
rfkill                 28672  6 bluetooth,cfg80211
vc4                   172032  0
v3d                    61440  0
drm_kms_helper        184320  1 vc4
gpu_sched              28672  1 v3d
raspberrypi_hwmon      16384  0
hwmon                  16384  1 raspberrypi_hwmon
snd_soc_core          192512  1 vc4
snd_compress           20480  1 snd_soc_core
drm                   438272  5 v3d,vc4,gpu_sched,drm_kms_helper
snd_pcm_dmaengine      16384  1 snd_soc_core
bcm2835_v4l2           45056  0
snd_bcm2835            24576  1
bcm2835_codec          36864  0
syscopyarea            16384  1 drm_kms_helper
snd_pcm               102400  4 vc4,snd_pcm_dmaengine,snd_bcm2835,snd_soc_core
sysfillrect            16384  1 drm_kms_helper
v4l2_mem2mem           24576  1 bcm2835_codec
snd_timer              32768  1 snd_pcm
sysimgblt              16384  1 drm_kms_helper
drm_panel_orientation_quirks    16384  1 drm
bcm2835_mmal_vchiq     32768  2 bcm2835_codec,bcm2835_v4l2
snd                    73728  7 snd_compress,snd_timer,snd_bcm2835,snd_soc_core,snd_pcm
fb_sys_fops            16384  1 drm_kms_helper
v4l2_common            16384  1 bcm2835_v4l2
videobuf2_dma_contig    20480  1 bcm2835_codec
videobuf2_vmalloc      16384  1 bcm2835_v4l2
videobuf2_memops       16384  2 videobuf2_dma_contig,videobuf2_vmalloc
videobuf2_v4l2         24576  3 bcm2835_codec,bcm2835_v4l2,v4l2_mem2mem
videobuf2_common       45056  4 bcm2835_codec,bcm2835_v4l2,v4l2_mem2mem,videobuf2_v4l2
videodev              200704  6 bcm2835_codec,v4l2_common,videobuf2_common,bcm2835_v4l2,v4l2_mem2mem,videobuf2_v4l2
media                  36864  2 videodev,v4l2_mem2mem
argon_mem              16384  0
uio_pdrv_genirq        16384  0
fixed                  16384  0
uio                    20480  1 uio_pdrv_genirq
ip_tables              24576  2 iptable_filter,iptable_nat
x_tables               32768  11 xt_comment,xt_multiport,ipt_REJECT,xt_nat,ip_tables,iptable_filter,xt_mark,xt_tcpudp,ipt_MASQUERADE,xt_addrtype,xt_conntrack
ipv6                  450560  482 nf_nat_ipv6,bridge
@alexellis

This comment has been minimized.

Copy link
Author

@alexellis alexellis commented Aug 7, 2019

Screenshot 2019-08-07 at 08 20 49

Guess what happened after I uninstalled docker-ce? ^

I suspect the containerd/runc version that ships with Docker may be breaking k3s.

@erikwilson

This comment has been minimized.

Copy link
Member

@erikwilson erikwilson commented Aug 7, 2019

Thanks for the info @alexellis!

If installed via curl script k3s will prepend its binary path (ie /var/lib/rancher/k3s/data/.../bin) and use the runc in that location. The k3s or containerd logs may contain more information. iptables may be breaking also, doing a diff of iptables-save before & after might help.

@hectoregm

This comment has been minimized.

Copy link

@hectoregm hectoregm commented Aug 18, 2019

I have the same issue with a Raspberry Pi 4 cluster dns and ip does not work after installing docker
Uninstalling docker restored network I suppose some iptables issue

@alexellis

This comment has been minimized.

Copy link
Author

@alexellis alexellis commented Aug 18, 2019

My work-around was to remove docker completely, then remove k3s and add it back in again.

It would be ideal if both of these tools could co-exist, so that we can do builds on cluster servers or agents.

@hectoregm

This comment has been minimized.

Copy link

@hectoregm hectoregm commented Aug 18, 2019

Got docker and k3s working together, issue appears that k3s uses iptables-legacy and docker is using iptables that in Buster is really iptables-nft, from issue kubernetes/kubernetes#71305 having both tables active is recipe for disaster

Running:

sudo iptables -F
sudo update-alternatives --set iptables /usr/sbin/iptables-legacy

and a reboot fixed and allows both to run in the cluster or worker

@erikwilson

This comment has been minimized.

Copy link
Member

@erikwilson erikwilson commented Aug 18, 2019

Thank you for debugging and finding the resolution @hectoregm !

I think we should maybe prefer the system iptables to our bundled version, similar to what we do with mount

mkdir -p bin/aux && rm bin/mount && ln -sf ../busybox bin/aux/mount

@erikwilson

This comment has been minimized.

Copy link
Member

@erikwilson erikwilson commented Oct 2, 2019

At the moment I think we should just add a warning to the install script if using an older version of iptables since this will be difficult to address for all projects.

The upstream iptables-nft binary probably needs to fallback to legacy binary or kernel modules if nft is not available.

@JakubOboza

This comment has been minimized.

Copy link

@JakubOboza JakubOboza commented Nov 18, 2019

I had exactly the same issue this weekend. New docker messes up iptables so using solo containerd is the solution. Thanks guys!

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
6 participants
You can’t perform that action at this time.