Microk8s not starting on ubuntu/odroid #4301
Comments
|
Can you share an inspection report from the cluster? Looks like the Calico CNI fails to initialize for some reason |
|
Thanks for your quick reply, here is the report: |
|
Thanks, I see the following line in Kubelet logs, which causes the services to continuously restart This typically means that cgroups are not configured properly on the system. As a workaround, can you try appending the following kubelet arguments: |
|
Hello,
I tried this but to no avail.
The sbc refuses to boot now.
I wil flash a fresh image tonight and install microk8s again.
I find it strange that a fresh ubuntu and a fresh microk8s do not work
together.
Any suggestions on another linux-flavour?
Kind regards.
Op di 14 nov 2023 om 12:52 schreef Angelos Kolaitis <
***@***.***>
… Thanks, I see the following line in Kubelet logs, which causes the
services to continuously restart
Nov 12 13:46:23 odroid microk8s.daemon-kubelite[359263]: E1112 13:46:23.145422 359263 kubelet.go:1480] "Failed to start ContainerManager" err="failed to initialize top level QOS containers: failed to update top level BestEffort QOS cgroup : open /sys/fs/cgroup/kubepods/besteffort/cpu.weight: no such file or directory"
This typically means that cgroups are not configured properly on the
system.
As a workaround, can you try appending the following kubelet arguments:
echo '
--cgroups-per-qos=false
--enforce-node-allocatable=""
' | sudo tee -a /var/snap/microk8s/current/args/kubelet
sudo snap restart microk8s.daemon-kubelite
—
Reply to this email directly, view it on GitHub
<#4301 (comment)>,
or unsubscribe
<https://github.com/notifications/unsubscribe-auth/AET5SGPDPTDKXQI63NRCRT3YENLPTAVCNFSM6AAAAAA7KSQOJWVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMYTQMJQGA3DCOJVGQ>
.
You are receiving this because you were mentioned.Message ID:
***@***.***>
|
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Summary
when installing microk8s on a clean ubuntu-Mate on an odroid N2+ SBC microk8s does not start
ubuntu:
https://wiki.odroid.com/odroid-n2/os_images/ubuntu/20220622
microk8s 1.28
What Should Happen Instead?
microk8s should start
Reproduction Steps
Install Ubuntu/Mate on Odroid N2+
https://wiki.odroid.com/odroid-n2/os_images/ubuntu/20220622
Install Microk8s using snap
Introspection Report
kubectl get pod -n kube-system gives:
coredns-864597b5fd-nqn2p 0/1 Pending 0 8m2s
calico-kube-controllers-77bd7c5b-fnzxk 0/1 Pending 0 8m2s
calico-node-bf62c 0/1 Init:CrashLoopBackOff 5 (70s ago) 8m2s
kubectl logs calico-node-bf62c -n kube-system gives:
Defaulted container "calico-node" out of: calico-node, upgrade-ipam (init), install-cni (init)
Error from server (BadRequest): container "calico-node" in pod "calico-node-bf62c" is waiting to start: PodInitializing
kubectl describe pod calico-node-bf62c -n kube-system gives:
Name: calico-node-bf62c
Namespace: kube-system
Priority: 2000001000
Priority Class Name: system-node-critical
Service Account: calico-node
Node: odroid/192.168.2.28
Start Time: Tue, 14 Nov 2023 10:19:08 +0000
Labels: controller-revision-hash=6fbb45588b
k8s-app=calico-node
pod-template-generation=1
Annotations:
Status: Pending
IP: 192.168.2.28
IPs:
IP: 192.168.2.28
Controlled By: DaemonSet/calico-node
Init Containers:
upgrade-ipam:
Container ID: containerd://6fdad669f5792ea1618ca5ae284d3725f77b59a7d15907cc17ff34c2a14b54ec
Image: docker.io/calico/cni:v3.25.1
Image ID: docker.io/calico/cni@sha256:9a2c99f0314053aa11e971bd5d72e17951767bf5c6ff1fd9c38c4582d7cb8a0a
Port:
Host Port:
Command:
/opt/cni/bin/calico-ipam
-upgrade
State: Terminated
Reason: Completed
Exit Code: 0
Started: Tue, 14 Nov 2023 10:19:50 +0000
Finished: Tue, 14 Nov 2023 10:19:50 +0000
Ready: True
Restart Count: 0
Environment Variables from:
kubernetes-services-endpoint ConfigMap Optional: true
Environment:
KUBERNETES_NODE_NAME: (v1:spec.nodeName)
CALICO_NETWORKING_BACKEND: <set to the key 'calico_backend' of config map 'calico-config'> Optional: false
Mounts:
/host/opt/cni/bin from cni-bin-dir (rw)
/var/lib/cni/networks from host-local-net-dir (rw)
/var/run/secrets/kubernetes.io/serviceaccount from kube-api-access-wlsfs (ro)
install-cni:
Container ID: containerd://be5e122180d5d88f1765c671d592e62ba0fd33deafc6bdf307d5d8b15d1f22e1
Image: docker.io/calico/cni:v3.25.1
Image ID: docker.io/calico/cni@sha256:9a2c99f0314053aa11e971bd5d72e17951767bf5c6ff1fd9c38c4582d7cb8a0a
Port:
Host Port:
Command:
/opt/cni/bin/install
State: Running
Started: Tue, 14 Nov 2023 10:34:18 +0000
Last State: Terminated
Reason: Error
Exit Code: 1
Started: Tue, 14 Nov 2023 10:28:45 +0000
Finished: Tue, 14 Nov 2023 10:29:17 +0000
Ready: False
Restart Count: 7
Environment Variables from:
kubernetes-services-endpoint ConfigMap Optional: true
Environment:
CNI_CONF_NAME: 10-calico.conflist
CNI_NETWORK_CONFIG: <set to the key 'cni_network_config' of config map 'calico-config'> Optional: false
KUBERNETES_NODE_NAME: (v1:spec.nodeName)
CNI_MTU: <set to the key 'veth_mtu' of config map 'calico-config'> Optional: false
SLEEP: false
CNI_NET_DIR: /var/snap/microk8s/current/args/cni-network
Mounts:
/host/etc/cni/net.d from cni-net-dir (rw)
/host/opt/cni/bin from cni-bin-dir (rw)
/var/run/secrets/kubernetes.io/serviceaccount from kube-api-access-wlsfs (ro)
Containers:
calico-node:
Container ID:
Image: docker.io/calico/node:v3.25.1
Image ID:
Port:
Host Port:
State: Waiting
Reason: PodInitializing
Ready: False
Restart Count: 0
Requests:
cpu: 250m
Liveness: exec [/bin/calico-node -felix-live] delay=10s timeout=10s period=10s #success=1 #failure=6
Readiness: exec [/bin/calico-node -felix-ready] delay=0s timeout=10s period=10s #success=1 #failure=3
Environment Variables from:
kubernetes-services-endpoint ConfigMap Optional: true
Environment:
DATASTORE_TYPE: kubernetes
WAIT_FOR_DATASTORE: true
NODENAME: (v1:spec.nodeName)
CALICO_NETWORKING_BACKEND: <set to the key 'calico_backend' of config map 'calico-config'> Optional: false
CLUSTER_TYPE: k8s,bgp
IP: autodetect
IP_AUTODETECTION_METHOD: first-found
CALICO_IPV4POOL_VXLAN: Always
CALICO_IPV6POOL_VXLAN: Never
FELIX_IPINIPMTU: <set to the key 'veth_mtu' of config map 'calico-config'> Optional: false
FELIX_VXLANMTU: <set to the key 'veth_mtu' of config map 'calico-config'> Optional: false
FELIX_WIREGUARDMTU: <set to the key 'veth_mtu' of config map 'calico-config'> Optional: false
CALICO_IPV4POOL_CIDR: 10.1.0.0/16
CALICO_DISABLE_FILE_LOGGING: true
FELIX_DEFAULTENDPOINTTOHOSTACTION: ACCEPT
FELIX_IPV6SUPPORT: false
FELIX_HEALTHENABLED: true
FELIX_FEATUREDETECTOVERRIDE: ChecksumOffloadBroken=true
Mounts:
/host/etc/cni/net.d from cni-net-dir (rw)
/lib/modules from lib-modules (ro)
/run/xtables.lock from xtables-lock (rw)
/var/lib/calico from var-lib-calico (rw)
/var/log/calico/cni from cni-log-dir (ro)
/var/run/calico from var-run-calico (rw)
/var/run/nodeagent from policysync (rw)
/var/run/secrets/kubernetes.io/serviceaccount from kube-api-access-wlsfs (ro)
Conditions:
Type Status
Initialized False
Ready False
ContainersReady False
PodScheduled True
Volumes:
lib-modules:
Type: HostPath (bare host directory volume)
Path: /lib/modules
HostPathType:
var-run-calico:
Type: HostPath (bare host directory volume)
Path: /var/snap/microk8s/current/var/run/calico
HostPathType:
var-lib-calico:
Type: HostPath (bare host directory volume)
Path: /var/snap/microk8s/current/var/lib/calico
HostPathType:
xtables-lock:
Type: HostPath (bare host directory volume)
Path: /run/xtables.lock
HostPathType: FileOrCreate
sys-fs:
Type: HostPath (bare host directory volume)
Path: /sys/fs/
HostPathType: DirectoryOrCreate
cni-bin-dir:
Type: HostPath (bare host directory volume)
Path: /var/snap/microk8s/current/opt/cni/bin
HostPathType:
cni-net-dir:
Type: HostPath (bare host directory volume)
Path: /var/snap/microk8s/current/args/cni-network
HostPathType:
cni-log-dir:
Type: HostPath (bare host directory volume)
Path: /var/snap/microk8s/common/var/log/calico/cni
HostPathType:
host-local-net-dir:
Type: HostPath (bare host directory volume)
Path: /var/snap/microk8s/current/var/lib/cni/networks
HostPathType:
policysync:
Type: HostPath (bare host directory volume)
Path: /var/snap/microk8s/current/var/run/nodeagent
HostPathType: DirectoryOrCreate
kube-api-access-wlsfs:
Type: Projected (a volume that contains injected data from multiple sources)
TokenExpirationSeconds: 3607
ConfigMapName: kube-root-ca.crt
ConfigMapOptional:
DownwardAPI: true
QoS Class: Burstable
Node-Selectors: kubernetes.io/os=linux
Tolerations: :NoSchedule op=Exists
:NoExecute op=Exists
CriticalAddonsOnly op=Exists
node.kubernetes.io/disk-pressure:NoSchedule op=Exists
node.kubernetes.io/memory-pressure:NoSchedule op=Exists
node.kubernetes.io/network-unavailable:NoSchedule op=Exists
node.kubernetes.io/not-ready:NoExecute op=Exists
node.kubernetes.io/pid-pressure:NoSchedule op=Exists
node.kubernetes.io/unreachable:NoExecute op=Exists
node.kubernetes.io/unschedulable:NoSchedule op=Exists
Events:
Type Reason Age From Message
Normal Pulled 15s (x8 over 14m) kubelet Container image "docker.io/calico/cni:v3.25.1" already present on machine
The text was updated successfully, but these errors were encountered: