New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

ARM 0.8.0 image fails to load libc.so.6 in k8s v1.7.0 #773

Closed
cdickson-sum opened this Issue Jul 14, 2017 · 18 comments

Comments

Projects
None yet
@cdickson-sum

On initial building out of a k8s cluster on ARM system the flannel install fails to start. Goes into a "CrashLoopBackOff"

Expected Behavior

Should move to a Running status in k8s

Current Behavior

Fails to start. Log reports:

$kubectl logs -n kube-system -c install-cni kube-flannel-ds-0s5s4
/bin/sh: error while loading shared libraries: libc.so.6: cannot open shared object file: No such file or directory

Steps to Reproduce (for bugs)

  1. Set up k8s via kubeadm
    2.Using kubectl apply flannel RBAC
    3.Using kubectl apply flannel 0.8.0 image
  2. Flannel pods fail to start

Context

Setting up a small ARM (Raspberry Pi) cluster

Your Environment

  • Flannel version: 0.8.0
  • Kubernetes version (if used): v1.7.0
  • Operating System and version: hypriotos-rpi-v1.4.0
    * latest docker and updates installed
@tomdee

This comment has been minimized.

Show comment
Hide comment
@tomdee

tomdee Jul 14, 2017

Member

I just tried spinning up a free ARM vm from Scaleway (https://www.scaleway.com/instantcloud/) - I ran

sudo apt-get update
sudo apt-get install docker.io
docker run quay.io/coreos/flannel:v0.8.0-arm64

And it appeared to run OK

Member

tomdee commented Jul 14, 2017

I just tried spinning up a free ARM vm from Scaleway (https://www.scaleway.com/instantcloud/) - I ran

sudo apt-get update
sudo apt-get install docker.io
docker run quay.io/coreos/flannel:v0.8.0-arm64

And it appeared to run OK

@cwalker67

This comment has been minimized.

Show comment
Hide comment
@cwalker67

cwalker67 Jul 14, 2017

@tomdee try running the arm version

$ docker run quay.io/coreos/flannel:v0.8.0-arm
standard_init_linux.go:178: exec user process caused "no such file or directory"

@tomdee try running the arm version

$ docker run quay.io/coreos/flannel:v0.8.0-arm
standard_init_linux.go:178: exec user process caused "no such file or directory"
@cwalker67

This comment has been minimized.

Show comment
Hide comment
@cwalker67

cwalker67 Jul 14, 2017

@tomdee
Scaleway's ARMv8 64 bit servers don't equate to a rpi3 ;)

$ docker run quay.io/coreos/flannel:v0.8.0-arm64
standard_init_linux.go:178: exec user process caused "exec format error"

@tomdee
Scaleway's ARMv8 64 bit servers don't equate to a rpi3 ;)

$ docker run quay.io/coreos/flannel:v0.8.0-arm64
standard_init_linux.go:178: exec user process caused "exec format error"
@cdickson-sum

This comment has been minimized.

Show comment
Hide comment
@cdickson-sum

cdickson-sum Jul 14, 2017

@tomdee thank you for starting to look at this, but the issue is with the 32 bit ARM version.

@tomdee thank you for starting to look at this, but the issue is with the 32 bit ARM version.

@jimmycuadra

This comment has been minimized.

Show comment
Hide comment
@jimmycuadra

jimmycuadra Jul 20, 2017

Same problem here. Brand new Kubernetes cluster using kubeadm, both v1.7.1. flannel v0.8.0-amd. Raspberry Pi 3.

$ sudo docker images | grep flannel
quay.io/coreos/flannel                                 v0.8.0-arm          450a18ed92ec        8 days ago          35.14 MB
$ sudo docker ps
CONTAINER ID        IMAGE                                                                                                                          COMMAND                  CREATED              STATUS                  PORTS               NAMES
208d94700252        quay.io/coreos/flannel@sha256:be0d1e3cc5f7dbde66243d3b5e99a7d194755416dcf365a155304193fa6aca7e                                 "/bin/sh -c 'set -e -"   2 seconds ago        Up Less than a second                       k8s_install-cni_kube-flannel-ds-6x7tq_kube-system_1800fd69-6d1c-11e7-b2b2-b827ebfa0d18_3
$ sudo journalctl CONTAINER_ID=208d94700252
-- Logs begin at Thu 2017-07-20 00:05:25 PDT, end at Thu 2017-07-20 00:23:36 PDT. --
Jul 20 00:23:14 redacted dockerd[759]: /bin/sh: error while loading shared libraries: libc.so.6: cannot open shared object file: No such file or directory
$ uname -a
Linux redacted 4.9.35-v7+ #1014 SMP Fri Jun 30 14:47:43 BST 2017 armv7l GNU/Linux
$ cat /etc/os-release
PRETTY_NAME="Raspbian GNU/Linux 8 (jessie)"
NAME="Raspbian GNU/Linux"
VERSION_ID="8"
VERSION="8 (jessie)"
ID=raspbian
ID_LIKE=debian
HOME_URL="http://www.raspbian.org/"
SUPPORT_URL="http://www.raspbian.org/RaspbianForums"
BUG_REPORT_URL="http://www.raspbian.org/RaspbianBugs"

jimmycuadra commented Jul 20, 2017

Same problem here. Brand new Kubernetes cluster using kubeadm, both v1.7.1. flannel v0.8.0-amd. Raspberry Pi 3.

$ sudo docker images | grep flannel
quay.io/coreos/flannel                                 v0.8.0-arm          450a18ed92ec        8 days ago          35.14 MB
$ sudo docker ps
CONTAINER ID        IMAGE                                                                                                                          COMMAND                  CREATED              STATUS                  PORTS               NAMES
208d94700252        quay.io/coreos/flannel@sha256:be0d1e3cc5f7dbde66243d3b5e99a7d194755416dcf365a155304193fa6aca7e                                 "/bin/sh -c 'set -e -"   2 seconds ago        Up Less than a second                       k8s_install-cni_kube-flannel-ds-6x7tq_kube-system_1800fd69-6d1c-11e7-b2b2-b827ebfa0d18_3
$ sudo journalctl CONTAINER_ID=208d94700252
-- Logs begin at Thu 2017-07-20 00:05:25 PDT, end at Thu 2017-07-20 00:23:36 PDT. --
Jul 20 00:23:14 redacted dockerd[759]: /bin/sh: error while loading shared libraries: libc.so.6: cannot open shared object file: No such file or directory
$ uname -a
Linux redacted 4.9.35-v7+ #1014 SMP Fri Jun 30 14:47:43 BST 2017 armv7l GNU/Linux
$ cat /etc/os-release
PRETTY_NAME="Raspbian GNU/Linux 8 (jessie)"
NAME="Raspbian GNU/Linux"
VERSION_ID="8"
VERSION="8 (jessie)"
ID=raspbian
ID_LIKE=debian
HOME_URL="http://www.raspbian.org/"
SUPPORT_URL="http://www.raspbian.org/RaspbianForums"
BUG_REPORT_URL="http://www.raspbian.org/RaspbianBugs"
@cwalker67

This comment has been minimized.

Show comment
Hide comment
@cwalker67

cwalker67 Jul 21, 2017

in examining the images, i've found the the libc.so.6 is substantially smaller than the one it replaces in the copy

  • new libc.so.6 is 894700
  • old libc.so.6 is 1275540
    This may have nothing to do with it but I do notice that the sizes being copied for the arm64 version is larger 1259184

in examining the images, i've found the the libc.so.6 is substantially smaller than the one it replaces in the copy

  • new libc.so.6 is 894700
  • old libc.so.6 is 1275540
    This may have nothing to do with it but I do notice that the sizes being copied for the arm64 version is larger 1259184
@tomdee

This comment has been minimized.

Show comment
Hide comment
@tomdee

tomdee Jul 21, 2017

Member

I'm happy to merge a PR to fix this, but I don't any any hardware to test it on so I'm relying on you guys to provide the fix 😄

Member

tomdee commented Jul 21, 2017

I'm happy to merge a PR to fix this, but I don't any any hardware to test it on so I'm relying on you guys to provide the fix 😄

@stevesloka

This comment has been minimized.

Show comment
Hide comment
@stevesloka

stevesloka Aug 6, 2017

Hey, @tomdee I'm happy to test out any builds. I'm seeing the same error mentioned above on my arm cluster.

Hey, @tomdee I'm happy to test out any builds. I'm seeing the same error mentioned above on my arm cluster.

@ikester

This comment has been minimized.

Show comment
Hide comment
@ikester

ikester Aug 15, 2017

Is there a workaround for using Flannel with K8s v1.7.x on ARM? Using v0.7?

Edit: I was able to deploy v0.7.1 of Flannel with a K8s v1.7.3 cluster. Looking forward to seeing v0.8 working on ARM.

ikester commented Aug 15, 2017

Is there a workaround for using Flannel with K8s v1.7.x on ARM? Using v0.7?

Edit: I was able to deploy v0.7.1 of Flannel with a K8s v1.7.3 cluster. Looking forward to seeing v0.8 working on ARM.

@swestcott

This comment has been minimized.

Show comment
Hide comment
@swestcott

swestcott Aug 16, 2017

I also had an issue with flannel 0.8 on my pi3,

pi@pi3:~ $ kubectl logs kube-flannel-ds-qnjc3 -n kube-system -c kube-flannel
standard_init_linux.go:178: exec user process caused "no such file or directory"

Flannel 0.7.1 is working ok

swestcott commented Aug 16, 2017

I also had an issue with flannel 0.8 on my pi3,

pi@pi3:~ $ kubectl logs kube-flannel-ds-qnjc3 -n kube-system -c kube-flannel
standard_init_linux.go:178: exec user process caused "no such file or directory"

Flannel 0.7.1 is working ok

@jimmycuadra

This comment has been minimized.

Show comment
Hide comment
@jimmycuadra

jimmycuadra Aug 17, 2017

I just switched from flannel to weave and it's been working fine.

I just switched from flannel to weave and it's been working fine.

@asaaki

This comment has been minimized.

Show comment
Hide comment
@asaaki

asaaki Aug 17, 2017

@jimmycuadra Did you set up a fresh cluster or switched within the running one? The latter didn't really work for me. :-/

asaaki commented Aug 17, 2017

@jimmycuadra Did you set up a fresh cluster or switched within the running one? The latter didn't really work for me. :-/

@jimmycuadra

This comment has been minimized.

Show comment
Hide comment
@jimmycuadra

jimmycuadra Aug 18, 2017

@asaaki It was a new cluster. Can't comment about switching from one to another, as I haven't tried that, but I don't see why it wouldn't work in theory. Perhaps if you share specifics of what went wrong, someone can help figure out your problem.

@asaaki It was a new cluster. Can't comment about switching from one to another, as I haven't tried that, but I don't see why it wouldn't work in theory. Perhaps if you share specifics of what went wrong, someone can help figure out your problem.

@bitvector2

This comment has been minimized.

Show comment
Hide comment
@bitvector2

bitvector2 Sep 18, 2017

How can I help with this? I have a cluster of RPIs just waiting to help?

How can I help with this? I have a cluster of RPIs just waiting to help?

@tomdee

This comment has been minimized.

Show comment
Hide comment
Member

tomdee commented Sep 18, 2017

@bitvector2 @stevesloka @swestcott @asaaki @ikester @cwalker67 @cdickson-sum Could you try the new v0.9.0-rc1 images?

@ecliptik

This comment has been minimized.

Show comment
Hide comment
@ecliptik

ecliptik Sep 19, 2017

@tomdee I haven't deployed Flannel v0.9.0-rc1 to my k8s Raspberry Pi 3 cluster, but the image does run now and no longer gives a standard_init_linux.go:178: exec user process caused "no such file or directory that I was seeing with v0.8.0-arm.

HypriotOS/armv7: pirate@navi in /var/tmp
$ uname -a
Linux navi 4.4.50-hypriotos-v7+ #1 SMP PREEMPT Sun Mar 19 14:11:54 UTC 2017 armv7l GNU/Linux

HypriotOS/armv7: pirate@navi in /var/tmp
$ docker run -it --rm quay.io/coreos/flannel:v0.8.0-arm
standard_init_linux.go:178: exec user process caused "no such file or directory"

HypriotOS/armv7: pirate@navi in /var/tmp
$ docker run -it --rm quay.io/coreos/flannel:v0.9.0-rc1-arm
I0919 03:43:26.072450       1 main.go:479] Determining IP address of default interface
I0919 03:43:26.074220       1 main.go:492] Using interface with name eth0 and address 172.17.0.2
I0919 03:43:26.074385       1 main.go:509] Defaulting external address to interface address (172.17.0.2)
I0919 03:43:26.075041       1 main.go:234] Created subnet manager: Etcd Local Manager with Previous Subnet: 0.0.0.0/0
I0919 03:43:26.075134       1 main.go:237] Installing signal handlers
E0919 03:43:26.081125       1 main.go:353] Couldn't fetch network config: client: etcd cluster is unavailable or misconfigured; error #0: dial tcp 127.0.0.1:4001: getsockopt: connection refused
; error #1: dial tcp 127.0.0.1:2379: getsockopt: connection refused
timed out
E0919 03:43:27.083944       1 main.go:353] Couldn't fetch network config: client: etcd cluster is unavailable or misconfigured; error #0: dial tcp 127.0.0.1:4001: getsockopt: connection refused
; error #1: dial tcp 127.0.0.1:2379: getsockopt: connection refused
^CI0919 03:43:27.674273       1 main.go:341] shutdownHandler sent cancel signal...

@tomdee I haven't deployed Flannel v0.9.0-rc1 to my k8s Raspberry Pi 3 cluster, but the image does run now and no longer gives a standard_init_linux.go:178: exec user process caused "no such file or directory that I was seeing with v0.8.0-arm.

HypriotOS/armv7: pirate@navi in /var/tmp
$ uname -a
Linux navi 4.4.50-hypriotos-v7+ #1 SMP PREEMPT Sun Mar 19 14:11:54 UTC 2017 armv7l GNU/Linux

HypriotOS/armv7: pirate@navi in /var/tmp
$ docker run -it --rm quay.io/coreos/flannel:v0.8.0-arm
standard_init_linux.go:178: exec user process caused "no such file or directory"

HypriotOS/armv7: pirate@navi in /var/tmp
$ docker run -it --rm quay.io/coreos/flannel:v0.9.0-rc1-arm
I0919 03:43:26.072450       1 main.go:479] Determining IP address of default interface
I0919 03:43:26.074220       1 main.go:492] Using interface with name eth0 and address 172.17.0.2
I0919 03:43:26.074385       1 main.go:509] Defaulting external address to interface address (172.17.0.2)
I0919 03:43:26.075041       1 main.go:234] Created subnet manager: Etcd Local Manager with Previous Subnet: 0.0.0.0/0
I0919 03:43:26.075134       1 main.go:237] Installing signal handlers
E0919 03:43:26.081125       1 main.go:353] Couldn't fetch network config: client: etcd cluster is unavailable or misconfigured; error #0: dial tcp 127.0.0.1:4001: getsockopt: connection refused
; error #1: dial tcp 127.0.0.1:2379: getsockopt: connection refused
timed out
E0919 03:43:27.083944       1 main.go:353] Couldn't fetch network config: client: etcd cluster is unavailable or misconfigured; error #0: dial tcp 127.0.0.1:4001: getsockopt: connection refused
; error #1: dial tcp 127.0.0.1:2379: getsockopt: connection refused
^CI0919 03:43:27.674273       1 main.go:341] shutdownHandler sent cancel signal...
@danmikita

This comment has been minimized.

Show comment
Hide comment
@danmikita

danmikita Sep 21, 2017

@tomdee I tried the v0.9.0-rc1 image and it fixed the problem. Everything is working perfect!

@tomdee I tried the v0.9.0-rc1 image and it fixed the problem. Everything is working perfect!

@tomdee

This comment has been minimized.

Show comment
Hide comment
@tomdee

tomdee Sep 22, 2017

Member

@danmikita That's great to hear!

Member

tomdee commented Sep 22, 2017

@danmikita That's great to hear!

@tomdee tomdee closed this Sep 22, 2017

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment