Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

k3s + armhf (Raspberry PI 4) = CrashLoopBackOff ??? #780

Closed
PieBru opened this issue Aug 31, 2019 · 9 comments
Closed

k3s + armhf (Raspberry PI 4) = CrashLoopBackOff ??? #780

PieBru opened this issue Aug 31, 2019 · 9 comments

Comments

@PieBru
Copy link

PieBru commented Aug 31, 2019

Describe the bug
After successfully installing k3s on a Raspberry PI 4B with Raspbian Buster Lite, installing other apps via helm fails and the pod goes into CrashLoopBackOff
After following the official helm+tiller install guide, I solved this issue for tiller, but the issue appears again for every app I try: cert-manager, rancher, kubernetes-dashboard, and so on.
At least for tiller, it seems an amd/arm wrong default architecture issue.

To Reproduce

  1. Etch a SD with Raspbian Buster Lite.
  2. Install k3s via curl.
  3. Follow the official guides to install and configure kubectl and helm on the managing PC.
  4. Try to install tiller using helm by following the official guide.

Expected behavior
Raspberry PI 4 succesfull installation of tiller, cert-manager, rancher, kubernetes-dashboard, and so on, carried on by following an official guide with fewer twists and turns.

Screenshots

Additional context
@erikwilson
Copy link
Contributor

Same issue as #703? If so iptables needs to be set to legacy mode.

@PieBru
Copy link
Author

PieBru commented Aug 31, 2019 •
edited
Loading

The iptables-legacy trick did not solve this issue.

After applying the fix, the normal iptables -L command shows the tables with apparently correct values, while before the iptable fix it gave an empty report and this warning:
# Warning: iptables-legacy tables present, use iptables-legacy to see them

Then, after rebooting the master and all the workers, my last install (kubernetes-dashboard) is still stuck in CrashLoopBackOff state.

Should I delete and reinstall it again?
Thanks,
Piero

@PieBru
Copy link
Author

PieBru commented Aug 31, 2019

Forgot a screenshot.

Screenshot from 2019-08-31 19-40-49

@erikwilson
Copy link
Contributor

kubectl describe for that pod will give more info, but it is probably failing on an architecture error where kubernetes-dashboard does not provide a multi-arch image, and helm is not really designed to deal with different architectures. Looking through the dashboard repo they have an arm deployment yaml here: https://github.com/kubernetes/dashboard/blob/v1.10.1/src/deploy/recommended/kubernetes-dashboard-arm.yaml
Probably easiest to just kubectl apply that.

@PieBru
Copy link
Author

PieBru commented Aug 31, 2019

Thanks @erikwilson for the link, I will try it and report tomorrow, but that dashboard wasn't my goal.
My goal is trying to use on-the-Edge (aka RasPI4) k3s and the Rancher vision.

As I said at the beginning, I was having problems during Rancher's dash installation, so I tried the Kubernetes dash just as a test, to see if the issue may be at architecture level, as you pointed out.
I did not find easily arm-compatible repos out there.

Thanks,
Piero

@erikwilson
Copy link
Contributor

Sounds good. The rancher image should be multi-arch, so was probably the initial iptables issue. If you are still having issues please share the contents of kubectl describe of the failing rancher pod.

@ibuildthecloud ibuildthecloud mentioned this issue Sep 1, 2019
Closed
@PieBru
Copy link
Author

PieBru commented Sep 1, 2019

cert-manager failed, CrashLoopBackOff on a minimal freshly installed 1 node cluster on Raspbian Buster Lite.

Installed cet-manager following this guide. By looking at release-notes, arm seems supported since 0.7, while the Rancher doc suggests v0.5.2.

$ kubectl create namespace cert-manager
$ kubectl label namespace cert-manager certmanager.k8s.io/disable-validation=true
$ kubectl apply -f https://github.com/jetstack/cert-manager/releases/download/v0.9.1/cert-manager.yaml

As cert-manager seems a Rancher install prerequisite, I'm stuck here, unless there is a way to intentionally skip security on a internal test Rancher installation on k3s.
It seems also Rancher does not support master+nodes on arm. If that's still true in the latest helm repo I'm afraid I must give up here.

That's the kubectl describe of the first failing pod:

$ kubectl describe pod cert-manager-cainjector -n cert-manager
Name:           cert-manager-cainjector-59d69b9b-gwzlr
Namespace:      cert-manager
Priority:       0
Node:           kmaster/172.16.1.50
Start Time:     Sun, 01 Sep 2019 19:37:20 +0200
Labels:         app=cainjector
                app.kubernetes.io/instance=cert-manager
                app.kubernetes.io/managed-by=Tiller
                app.kubernetes.io/name=cainjector
                helm.sh/chart=cainjector-v0.9.1
                pod-template-hash=59d69b9b
Annotations:    <none>
Status:         Running
IP:             10.42.0.41
Controlled By:  ReplicaSet/cert-manager-cainjector-59d69b9b
Containers:
  cainjector:
    Container ID:  containerd://2d90ea247ff2510770a702027b3818eff3f69a8d92f609692baccb95e788b64a
    Image:         quay.io/jetstack/cert-manager-cainjector:v0.9.1
    Image ID:      sha256:3c4cedb6adfabd76217c92373fd35daa95db43a9320a3e4994b21ecfb29b6251
    Port:          <none>
    Host Port:     <none>
    Args:
      --v=2
      --leader-election-namespace=$(POD_NAMESPACE)
    State:          Waiting
      Reason:       CrashLoopBackOff
    Last State:     Terminated
      Reason:       Error
      Exit Code:    1
      Started:      Sun, 01 Sep 2019 19:44:11 +0200
      Finished:     Sun, 01 Sep 2019 19:44:11 +0200
    Ready:          False
    Restart Count:  6
    Environment:
      POD_NAMESPACE:  cert-manager (v1:metadata.namespace)
    Mounts:
      /var/run/secrets/kubernetes.io/serviceaccount from cert-manager-cainjector-token-5xfg5 (ro)
Conditions:
  Type              Status
  Initialized       True 
  Ready             False 
  ContainersReady   False 
  PodScheduled      True 
Volumes:
  cert-manager-cainjector-token-5xfg5:
    Type:        Secret (a volume populated by a Secret)
    SecretName:  cert-manager-cainjector-token-5xfg5
    Optional:    false
QoS Class:       BestEffort
Node-Selectors:  <none>
Tolerations:     node.kubernetes.io/not-ready:NoExecute for 300s
                 node.kubernetes.io/unreachable:NoExecute for 300s
Events:
  Type     Reason     Age                     From               Message
  ----     ------     ----                    ----               -------
  Normal   Scheduled  7m51s                   default-scheduler  Successfully assigned cert-manager/cert-manager-cainjector-59d69b9b-gwzlr to kmaster
  Normal   Pulling    7m49s                   kubelet, kmaster   Pulling image "quay.io/jetstack/cert-manager-cainjector:v0.9.1"
  Normal   Pulled     6m53s                   kubelet, kmaster   Successfully pulled image "quay.io/jetstack/cert-manager-cainjector:v0.9.1"
  Normal   Pulled     5m19s (x4 over 6m51s)   kubelet, kmaster   Container image "quay.io/jetstack/cert-manager-cainjector:v0.9.1" already present on machine
  Normal   Created    5m18s (x5 over 6m53s)   kubelet, kmaster   Created container cainjector
  Normal   Started    5m17s (x5 over 6m52s)   kubelet, kmaster   Started container cainjector
  Warning  BackOff    2m44s (x22 over 6m49s)  kubelet, kmaster   Back-off restarting failed container

Followed by this:

$ kubectl describe pod cert-manager -n cert-manager
Name:           cert-manager-86f74d9b9f-r6j4z
Namespace:      cert-manager
Priority:       0
Node:           kmaster/172.16.1.50
Start Time:     Sun, 01 Sep 2019 19:37:20 +0200
Labels:         app=cert-manager
                app.kubernetes.io/instance=cert-manager
                app.kubernetes.io/managed-by=Tiller
                app.kubernetes.io/name=cert-manager
                helm.sh/chart=cert-manager-v0.9.1
                pod-template-hash=86f74d9b9f
Annotations:    prometheus.io/path: /metrics
                prometheus.io/port: 9402
                prometheus.io/scrape: true
Status:         Running
IP:             10.42.0.42
Controlled By:  ReplicaSet/cert-manager-86f74d9b9f
Containers:
  cert-manager:
    Container ID:  containerd://8e99f39f87a80454a82761c6599f4189efe311c3e746f63399fb8efacaa292fa
    Image:         quay.io/jetstack/cert-manager-controller:v0.9.1
    Image ID:      sha256:056df3645936c1b05e79ca1a90a742faa5a2ee6ea3e7a02489eec4df822550d9
    Port:          9402/TCP
    Host Port:     0/TCP
    Args:
      --v=2
      --cluster-resource-namespace=$(POD_NAMESPACE)
      --leader-election-namespace=$(POD_NAMESPACE)
    State:          Waiting
      Reason:       CrashLoopBackOff
    Last State:     Terminated
      Reason:       Error
      Exit Code:    1
      Started:      Sun, 01 Sep 2019 19:54:44 +0200
      Finished:     Sun, 01 Sep 2019 19:54:44 +0200
    Ready:          False
    Restart Count:  8
    Requests:
      cpu:     10m
      memory:  32Mi
    Environment:
      POD_NAMESPACE:  cert-manager (v1:metadata.namespace)
    Mounts:
      /var/run/secrets/kubernetes.io/serviceaccount from cert-manager-token-8fzj6 (ro)
Conditions:
  Type              Status
  Initialized       True 
  Ready             False 
  ContainersReady   False 
  PodScheduled      True 
Volumes:
  cert-manager-token-8fzj6:
    Type:        Secret (a volume populated by a Secret)
    SecretName:  cert-manager-token-8fzj6
    Optional:    false
QoS Class:       Burstable
Node-Selectors:  <none>
Tolerations:     node.kubernetes.io/not-ready:NoExecute for 300s
                 node.kubernetes.io/unreachable:NoExecute for 300s
Events:
  Type     Reason     Age                   From               Message
  ----     ------     ----                  ----               -------
  Normal   Scheduled  18m                   default-scheduler  Successfully assigned cert-manager/cert-manager-86f74d9b9f-r6j4z to kmaster
  Normal   Pulling    18m                   kubelet, kmaster   Pulling image "quay.io/jetstack/cert-manager-controller:v0.9.1"
  Normal   Pulled     16m                   kubelet, kmaster   Successfully pulled image "quay.io/jetstack/cert-manager-controller:v0.9.1"
  Normal   Pulled     15m (x4 over 16m)     kubelet, kmaster   Container image "quay.io/jetstack/cert-manager-controller:v0.9.1" already present on machine
  Normal   Created    15m (x5 over 16m)     kubelet, kmaster   Created container cert-manager
  Normal   Started    15m (x5 over 16m)     kubelet, kmaster   Started container cert-manager
  Warning  BackOff    3m10s (x61 over 16m)  kubelet, kmaster   Back-off restarting failed container

Name:           cert-manager-cainjector-59d69b9b-gwzlr
Namespace:      cert-manager
Priority:       0
Node:           kmaster/172.16.1.50
Start Time:     Sun, 01 Sep 2019 19:37:20 +0200
Labels:         app=cainjector
                app.kubernetes.io/instance=cert-manager
                app.kubernetes.io/managed-by=Tiller
                app.kubernetes.io/name=cainjector
                helm.sh/chart=cainjector-v0.9.1
                pod-template-hash=59d69b9b
Annotations:    <none>
Status:         Running
IP:             10.42.0.41
Controlled By:  ReplicaSet/cert-manager-cainjector-59d69b9b
Containers:
  cainjector:
    Container ID:  containerd://176aa9e907a103e2998d1354ceafce8f728947207e63ed44ee8075c6597dcce0
    Image:         quay.io/jetstack/cert-manager-cainjector:v0.9.1
    Image ID:      sha256:3c4cedb6adfabd76217c92373fd35daa95db43a9320a3e4994b21ecfb29b6251
    Port:          <none>
    Host Port:     <none>
    Args:
      --v=2
      --leader-election-namespace=$(POD_NAMESPACE)
    State:          Waiting
      Reason:       CrashLoopBackOff
    Last State:     Terminated
      Reason:       Error
      Exit Code:    1
      Started:      Sun, 01 Sep 2019 19:54:17 +0200
      Finished:     Sun, 01 Sep 2019 19:54:17 +0200
    Ready:          False
    Restart Count:  8
    Environment:
      POD_NAMESPACE:  cert-manager (v1:metadata.namespace)
    Mounts:
      /var/run/secrets/kubernetes.io/serviceaccount from cert-manager-cainjector-token-5xfg5 (ro)
Conditions:
  Type              Status
  Initialized       True 
  Ready             False 
  ContainersReady   False 
  PodScheduled      True 
Volumes:
  cert-manager-cainjector-token-5xfg5:
    Type:        Secret (a volume populated by a Secret)
    SecretName:  cert-manager-cainjector-token-5xfg5
    Optional:    false
QoS Class:       BestEffort
Node-Selectors:  <none>
Tolerations:     node.kubernetes.io/not-ready:NoExecute for 300s
                 node.kubernetes.io/unreachable:NoExecute for 300s
Events:
  Type     Reason     Age                  From               Message
  ----     ------     ----                 ----               -------
  Normal   Scheduled  18m                  default-scheduler  Successfully assigned cert-manager/cert-manager-cainjector-59d69b9b-gwzlr to kmaster
  Normal   Pulling    18m                  kubelet, kmaster   Pulling image "quay.io/jetstack/cert-manager-cainjector:v0.9.1"
  Normal   Pulled     17m                  kubelet, kmaster   Successfully pulled image "quay.io/jetstack/cert-manager-cainjector:v0.9.1"
  Normal   Pulled     15m (x4 over 17m)    kubelet, kmaster   Container image "quay.io/jetstack/cert-manager-cainjector:v0.9.1" already present on machine
  Normal   Created    15m (x5 over 17m)    kubelet, kmaster   Created container cainjector
  Normal   Started    15m (x5 over 17m)    kubelet, kmaster   Started container cainjector
  Warning  BackOff    3m8s (x68 over 17m)  kubelet, kmaster   Back-off restarting failed container

Name:           cert-manager-webhook-cfd6587ff-dnvzh
Namespace:      cert-manager
Priority:       0
Node:           kmaster/172.16.1.50
Start Time:     Sun, 01 Sep 2019 19:37:20 +0200
Labels:         app=webhook
                app.kubernetes.io/instance=cert-manager
                app.kubernetes.io/managed-by=Tiller
                app.kubernetes.io/name=webhook
                helm.sh/chart=webhook-v0.9.1
                pod-template-hash=cfd6587ff
Annotations:    <none>
Status:         Pending
IP:             
Controlled By:  ReplicaSet/cert-manager-webhook-cfd6587ff
Containers:
  webhook:
    Container ID:  
    Image:         quay.io/jetstack/cert-manager-webhook:v0.9.1
    Image ID:      
    Port:          <none>
    Host Port:     <none>
    Args:
      --v=2
      --secure-port=6443
      --tls-cert-file=/certs/tls.crt
      --tls-private-key-file=/certs/tls.key
    State:          Waiting
      Reason:       ContainerCreating
    Ready:          False
    Restart Count:  0
    Environment:
      POD_NAMESPACE:  cert-manager (v1:metadata.namespace)
    Mounts:
      /certs from certs (rw)
      /var/run/secrets/kubernetes.io/serviceaccount from cert-manager-webhook-token-7j595 (ro)
Conditions:
  Type              Status
  Initialized       True 
  Ready             False 
  ContainersReady   False 
  PodScheduled      True 
Volumes:
  certs:
    Type:        Secret (a volume populated by a Secret)
    SecretName:  cert-manager-webhook-webhook-tls
    Optional:    false
  cert-manager-webhook-token-7j595:
    Type:        Secret (a volume populated by a Secret)
    SecretName:  cert-manager-webhook-token-7j595
    Optional:    false
QoS Class:       BestEffort
Node-Selectors:  <none>
Tolerations:     node.kubernetes.io/not-ready:NoExecute for 300s
                 node.kubernetes.io/unreachable:NoExecute for 300s
Events:
  Type     Reason       Age                       From               Message
  ----     ------       ----                      ----               -------
  Normal   Scheduled    18m                       default-scheduler  Successfully assigned cert-manager/cert-manager-webhook-cfd6587ff-dnvzh to kmaster
  Warning  FailedMount  26s (x8 over 16m)         kubelet, kmaster   Unable to mount volumes for pod "cert-manager-webhook-cfd6587ff-dnvzh_cert-manager(260b54c0-ccdf-11e9-9f7b-dca632172163)": timeout expired waiting for volumes to attach or mount for pod "cert-manager"/"cert-manager-webhook-cfd6587ff-dnvzh". list of unmounted volumes=[certs]. list of unattached volumes=[certs cert-manager-webhook-token-7j595]
  Warning  FailedMount  <invalid> (x17 over 18m)  kubelet, kmaster   MountVolume.SetUp failed for volume "certs" : secret "cert-manager-webhook-webhook-tls" not found

I haven't masked any secret, as this is a test-and-destroy setup.
Sorry, I'm a k3s and generally a Kubernetes newbie, thanks so much for your patience.
Piero

@neerdoc
Copy link

neerdoc commented Mar 15, 2020

Hi, I had the exact same problem but with Raspberry Pi 3 B+ in a 10-cluster setup.
Eventually, I managed to get it right, and I believe that the issue is that you have to apply the crds first (but I'm a bit out of my comfort zone, so I might be wrong).

This is what I did to make it work from a brand new k3os cluster:

  • kubectl create namespace cert-manager
  • wget https://raw.githubusercontent.com/jetstack/cert-manager/v0.14.0/deploy/manifests/00-crds.yaml
  • kubectl apply -f 00-crds.yaml
  • helm repo add jetstack https://charts.jetstack.io && helm repo update
  • helm install cert-manager --namespace cert-manager jetstack/cert-manager

Hope it helps!

@brandond
Copy link
Member

brandond commented Dec 5, 2020

Closing due to age.

@brandond brandond closed this as completed Dec 5, 2020
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
4 participants
@brandond @erikwilson @neerdoc @PieBru