Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Quartz64 | Add kernel features required for Kubernetes #6389

Closed
acelinkio opened this issue May 23, 2023 · 2 comments
Closed

Quartz64 | Add kernel features required for Kubernetes #6389

acelinkio opened this issue May 23, 2023 · 2 comments
Assignees
@acelinkio
Labels
Enhancement 💨 Quartz64 Solution available 🥂 Definite solution has been done
Milestone
v8.19

Comments

@acelinkio
Copy link

acelinkio commented May 23, 2023 •
edited

k3s check-config returns two errors: CONFIG_CGROUP_FREEZER: missing (fail) and CONFIG_NETFILTER_XT_MATCH_IPVS: missing (fail)

microk8s based clusters fail running Calico CNI. ipset v7.11: Kernel error received: Invalid argument

Required Information

  • DietPi version | cat /boot/dietpi/.version
G_DIETPI_VERSION_CORE=8
G_DIETPI_VERSION_SUB=17
G_DIETPI_VERSION_RC=2
G_GITBRANCH='master'
G_GITOWNER='MichaIng'
G_LIVE_PATCH_STATUS[0]='applied'
G_LIVE_PATCH_STATUS[1]='not applicable'
  • Distro version | echo $G_DISTRO_NAME $G_RASPBIAN
bullseye
  • Kernel version | uname -a
Linux soquartz0 6.2.14 #1 SMP PREEMPT Wed May  3 17:13:02 UTC 2023 aarch64 GNU/Linux
  • SBC model | echo $G_HW_MODEL_NAME or (EG: RPi3)
Quartz64 (aarch64)

TuringPi v2
SOQuartz Compute Module (3 of them)
32gb of EMMC
https://turingpi.com/product/power-supply/

Additional Information (if applicable)

  • Software title | (EG: Nextcloud)
    k3s
    microk8s

  • Was the software title installed freshly or updated/migrated?
    Fresh install

  • Can this issue be replicated on a fresh installation of DietPi?
    Yes. Replicated on 3 different nodes

  • Bug report ID | echo $G_HW_UUID
3b37e2f9-1dec-47e3-a1c2-19f340d4c25b

Steps to reproduce

  • run dietpi-software
  • select k3s & install
  • run k3s check-config
  • run kubectl get pods -A
    OR
  • run dietpi-software
  • select microk8s & install & reboot
  • run microk8s kubectl get pods -A
  • run microk8s kubectl logs calico-node-4225z

Expected behaviour

  • kubernetes pods should running without crash looping

Actual behaviour

  • kubernetes pods are crashing

Extra details

k3s check-config

Verifying binaries in /var/lib/rancher/k3s/data/4b147cafa965066cd68e04b4e3acce221078156a3b9ba635a653517ce459aa4d/bin:
- sha256sum: good
- links: good

System:
- /usr/sbin iptables v1.8.7 (nf_tables): ok
- swap: disabled
- routes: ok

Limits:
- /proc/sys/kernel/keys/root_maxkeys: 1000000

info: reading kernel config from /proc/config.gz ...

Generally Necessary:
- cgroup hierarchy: cgroups V2 mounted, cpu|cpuset|memory controllers status: good
- CONFIG_NAMESPACES: enabled
- CONFIG_NET_NS: enabled
- CONFIG_PID_NS: enabled
- CONFIG_IPC_NS: enabled
- CONFIG_UTS_NS: enabled
- CONFIG_CGROUPS: enabled
- CONFIG_CGROUP_PIDS: enabled
- CONFIG_CGROUP_CPUACCT: enabled
- CONFIG_CGROUP_DEVICE: enabled
- CONFIG_CGROUP_FREEZER: missing (fail)
- CONFIG_CGROUP_SCHED: enabled
- CONFIG_CPUSETS: enabled
- CONFIG_MEMCG: enabled
- CONFIG_KEYS: enabled
- CONFIG_VETH: enabled (as module)
- CONFIG_BRIDGE: enabled (as module)
- CONFIG_BRIDGE_NETFILTER: enabled (as module)
- CONFIG_IP_NF_FILTER: enabled (as module)
- CONFIG_IP_NF_TARGET_MASQUERADE: enabled (as module)
- CONFIG_NETFILTER_XT_MATCH_ADDRTYPE: enabled (as module)
- CONFIG_NETFILTER_XT_MATCH_CONNTRACK: enabled (as module)
- CONFIG_NETFILTER_XT_MATCH_IPVS: missing (fail)
- CONFIG_NETFILTER_XT_MATCH_COMMENT: enabled (as module)
- CONFIG_NETFILTER_XT_MATCH_MULTIPORT: enabled (as module)
- CONFIG_IP_NF_NAT: enabled (as module)
- CONFIG_NF_NAT: enabled (as module)
- CONFIG_POSIX_MQUEUE: enabled

Optional Features:
- CONFIG_USER_NS: enabled
- CONFIG_SECCOMP: enabled
- CONFIG_BLK_CGROUP: enabled
- CONFIG_BLK_DEV_THROTTLING: missing
- CONFIG_CGROUP_PERF: enabled
- CONFIG_CGROUP_HUGETLB: enabled
- CONFIG_NET_CLS_CGROUP: missing
- CONFIG_CGROUP_NET_PRIO: missing
- CONFIG_CFS_BANDWIDTH: enabled
- CONFIG_FAIR_GROUP_SCHED: enabled
- CONFIG_RT_GROUP_SCHED: missing
- CONFIG_IP_NF_TARGET_REDIRECT: missing
- CONFIG_IP_SET: missing
- CONFIG_IP_VS: missing
- CONFIG_IP_VS_NFCT: missing
- CONFIG_IP_VS_PROTO_TCP: missing
- CONFIG_IP_VS_PROTO_UDP: missing
- CONFIG_IP_VS_RR: missing
- CONFIG_EXT4_FS: enabled
- CONFIG_EXT4_FS_POSIX_ACL: enabled
- CONFIG_EXT4_FS_SECURITY: missing
    enable these ext4 configs if you are using ext4 as backing filesystem
- Network Drivers:
  - "overlay":
    - CONFIG_VXLAN: enabled (as module)
      Optional (for encrypted networks):
      - CONFIG_CRYPTO: enabled
      - CONFIG_CRYPTO_AEAD: enabled
      - CONFIG_CRYPTO_GCM: enabled (as module)
      - CONFIG_CRYPTO_SEQIV: missing
      - CONFIG_CRYPTO_GHASH: enabled (as module)
      - CONFIG_XFRM: missing
      - CONFIG_XFRM_USER: missing
      - CONFIG_XFRM_ALGO: missing
      - CONFIG_INET_ESP: missing
      - CONFIG_INET_XFRM_MODE_TRANSPORT: missing
- Storage Drivers:
  - "overlay":
    - CONFIG_OVERLAY_FS: enabled (as module)

STATUS: 2 (fail)
@acelinkio
Copy link
Author

handful of errors from journalctl -u k3s

May 23 04:57:43 soquartz0 k3s[1855]: E0523 04:57:43.408311    1855 pod_workers.go:965] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"local-path-provisioner\" with CrashLoopBackOff: \"back-off 5m0s restarting failed container=local-path-provisioner pod=local-path-provisioner-76d776f6f9-5d992_kube-system(623396e1-6b43-4cfa-8fcf-d92c0563a6a9)\"" pod="kube-system/local-path-provisioner-76d776f6f9-5d992" podUID=623396e1-6b43-4cfa-8fcf-d92c0563a6a9
May 23 04:57:45 soquartz0 k3s[1855]: E0523 04:57:45.411773    1855 pod_workers.go:965] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"helm\" with CrashLoopBackOff: \"back-off 5m0s restarting failed container=helm pod=helm-install-traefik-crd-q7bks_kube-system(75efe145-2974-4118-a549-f2ee27a0ece8)\"" pod="kube-system/helm-install-traefik-crd-q7bks" podUID=75efe145-2974-4118-a549-f2ee27a0ece8
May 23 04:57:46 soquartz0 k3s[1855]:         error checking rule: exit status 2: ip6tables v1.8.7 (nf_tables): Couldn't load match `conntrack':No such file or directory
May 23 04:57:47 soquartz0 k3s[1855]: E0523 04:57:47.411715    1855 pod_workers.go:965] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"helm\" with CrashLoopBackOff: \"back-off 5m0s restarting failed container=helm pod=helm-install-traefik-vlhgr_kube-system(df730f79-69a5-419f-8a15-bc061a465c7a)\"" pod="kube-system/helm-install-traefik-vlhgr" podUID=df730f79-69a5-419f-8a15-bc061a465c7a
May 23 04:57:55 soquartz0 k3s[1855]: E0523 04:57:55.171047    1855 iptables.go:320] Failed to ensure iptables rules: error checking rule existence: failed to check rule existence: running [/usr/sbin/iptables -t filter -C FORWARD -m comment --comment flanneld forward -j FLANNEL-FWD --wait]: exit status 2: iptables v1.8.7 (nf_tables): Couldn't load match `comment':No such file or directory
May 23 04:57:55 soquartz0 k3s[1855]: E0523 04:57:55.411053    1855 pod_workers.go:965] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"local-path-provisioner\" with CrashLoopBackOff: \"back-off 5m0s restarting failed container=local-path-provisioner pod=local-path-provisioner-76d776f6f9-5d992_kube-system(623396e1-6b43-4cfa-8fcf-d92c0563a6a9)\"" pod="kube-system/local-path-provisioner-76d776f6f9-5d992" podUID=623396e1-6b43-4cfa-8fcf-d92c0563a6a9
May 23 04:57:59 soquartz0 k3s[1855]: E0523 04:57:59.045579    1855 controller.go:113] loading OpenAPI spec for "v1beta1.metrics.k8s.io" failed with: Error, could not get list of group versions for APIService
May 23 04:57:59 soquartz0 k3s[1855]: E0523 04:57:59.048614    1855 controller.go:116] loading OpenAPI spec for "v1beta1.metrics.k8s.io" failed with: failed to retrieve openAPI spec, http error: ResponseCode: 503, Body: service unavailable
May 23 04:57:59 soquartz0 k3s[1855]: E0523 04:57:59.411294    1855 pod_workers.go:965] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"helm\" with CrashLoopBackOff: \"back-off 5m0s restarting failed container=helm pod=helm-install-traefik-crd-q7bks_kube-system(75efe145-2974-4118-a549-f2ee27a0ece8)\"" pod="kube-system/helm-install-traefik-crd-q7bks" podUID=75efe145-2974-4118-a549-f2ee27a0ece8
May 23 04:58:02 soquartz0 k3s[1855]: E0523 04:58:02.411449    1855 pod_workers.go:965] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"helm\" with CrashLoopBackOff: \"back-off 5m0s restarting failed container=helm pod=helm-install-traefik-vlhgr_kube-system(df730f79-69a5-419f-8a15-bc061a465c7a)\"" pod="kube-system/helm-install-traefik-vlhgr" podUID=df730f79-69a5-419f-8a15-bc061a465c7a
May 23 04:58:07 soquartz0 k3s[1855]: E0523 04:58:07.412508    1855 pod_workers.go:965] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"local-path-provisioner\" with CrashLoopBackOff: \"back-off 5m0s restarting failed container=local-path-provisioner pod=local-path-provisioner-76d776f6f9-5d992_kube-system(623396e1-6b43-4cfa-8fcf-d92c0563a6a9)\"" pod="kube-system/local-path-provisioner-76d776f6f9-5d992" podUID=623396e1-6b43-4cfa-8fcf-d92c0563a6a9
May 23 04:58:08 soquartz0 k3s[1855]:         error checking rule: exit status 2: iptables v1.8.7 (nf_tables): Couldn't load match `conntrack':No such file or directory
May 23 04:58:10 soquartz0 k3s[1855]: E0523 04:58:10.411207    1855 pod_workers.go:965] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"helm\" with CrashLoopBackOff: \"back-off 5m0s restarting failed container=helm pod=helm-install-traefik-crd-q7bks_kube-system(75efe145-2974-4118-a549-f2ee27a0ece8)\"" pod="kube-system/helm-install-traefik-crd-q7bks" podUID=75efe145-2974-4118-a549-f2ee27a0ece8
May 23 04:58:11 soquartz0 k3s[1855]: E0523 04:58:11.458308    1855 pod_workers.go:965] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"metrics-server\" with CrashLoopBackOff: \"back-off 5m0s restarting failed container=metrics-server pod=metrics-server-7b67f64457-qgj78_kube-system(f3bde704-05ee-4f6b-aa72-86842b77f4ac)\"" pod="kube-system/metrics-server-7b67f64457-qgj78" podUID=f3bde704-05ee-4f6b-aa72-86842b77f4ac
May 23 04:58:14 soquartz0 k3s[1855]: E0523 04:58:14.411317    1855 pod_workers.go:965] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"helm\" with CrashLoopBackOff: \"back-off 5m0s restarting failed container=helm pod=helm-install-traefik-vlhgr_kube-system(df730f79-69a5-419f-8a15-bc061a465c7a)\"" pod="kube-system/helm-install-traefik-vlhgr" podUID=df730f79-69a5-419f-8a15-bc061a465c7a
May 23 04:58:15 soquartz0 k3s[1855]: E0523 04:58:15.033789    1855 pod_workers.go:965] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"metrics-server\" with CrashLoopBackOff: \"back-off 5m0s restarting failed container=metrics-server pod=metrics-server-7b67f64457-qgj78_kube-system(f3bde704-05ee-4f6b-aa72-86842b77f4ac)\"" pod="kube-system/metrics-server-7b67f64457-qgj78" podUID=f3bde704-05ee-4f6b-aa72-86842b77f4ac
May 23 04:58:16 soquartz0 k3s[1855]:         error checking rule: exit status 2: ip6tables v1.8.7 (nf_tables): Couldn't load match `conntrack':No such file or directory
May 23 04:58:21 soquartz0 k3s[1855]: E0523 04:58:21.411703    1855 pod_workers.go:965] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"local-path-provisioner\" with CrashLoopBackOff: \"back-off 5m0s restarting failed container=local-path-provisioner pod=local-path-provisioner-76d776f6f9-5d992_kube-system(623396e1-6b43-4cfa-8fcf-d92c0563a6a9)\"" pod="kube-system/local-path-provisioner-76d776f6f9-5d992" podUID=623396e1-6b43-4cfa-8fcf-d92c0563a6a9
May 23 04:58:24 soquartz0 k3s[1855]: E0523 04:58:24.410609    1855 pod_workers.go:965] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"helm\" with CrashLoopBackOff: \"back-off 5m0s restarting failed container=helm pod=helm-install-traefik-crd-q7bks_kube-system(75efe145-2974-4118-a549-f2ee27a0ece8)\"" pod="kube-system/helm-install-traefik-crd-q7bks" podUID=75efe145-2974-4118-a549-f2ee27a0ece8
May 23 04:58:25 soquartz0 k3s[1855]: E0523 04:58:25.410329    1855 pod_workers.go:965] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"metrics-server\" with CrashLoopBackOff: \"back-off 5m0s restarting failed container=metrics-server pod=metrics-server-7b67f64457-qgj78_kube-system(f3bde704-05ee-4f6b-aa72-86842b77f4ac)\"" pod="kube-system/metrics-server-7b67f64457-qgj78" podUID=f3bde704-05ee-4f6b-aa72-86842b77f4ac
May 23 04:58:25 soquartz0 k3s[1855]: E0523 04:58:25.444620    1855 iptables.go:320] Failed to ensure iptables rules: error checking rule existence: failed to check rule existence: running [/usr/sbin/iptables -t nat -C POSTROUTING -m comment --comment flanneld masq -j FLANNEL-POSTRTG --wait]: exit status 2: iptables v1.8.7 (nf_tables): Couldn't load match `comment':No such file or directory

@acelinkio
Copy link
Author

Appears to be related to #6192. There is similar errors mentioned inside of the related thread. https://dietpi.com/forum/t/is-anyone-running-dietpi-on-a-pine64-soquartz/14852/24

However those changes appear to be incorporated into the dietpi-software installation. I see during execution several changes made to run update-alternatives for iptables and also installation for apparmor. I've also tried installing the kernel as mentioned https://dietpi.com/downloads/binaries/firmware-soquartz.deb

@acelinkio acelinkio changed the title k3s fails to run as expected on SOQuartz SOQuartz missing Kernel Modules May 24, 2023
@acelinkio acelinkio mentioned this issue May 24, 2023
Closed
@MichaIng MichaIng linked a pull request May 24, 2023 that will close this issue
Feature/add soquartz kernel modules #6390
Closed
@MichaIng MichaIng added this to the v8.18 milestone May 24, 2023
@acelinkio acelinkio mentioned this issue May 26, 2023
Closed
@MichaIng MichaIng linked a pull request May 27, 2023 that will close this issue
Quartz64 | Add kernel features required for Kubernetes #6393
Closed
@MichaIng MichaIng changed the title SOQuartz missing Kernel Modules Quartz64 | Add kernel features required for Kubernetes May 27, 2023
@MichaIng MichaIng added the Solution available 🥂 Definite solution has been done label Jun 3, 2023
@MichaIng MichaIng closed this as completed Jun 3, 2023
@MichaIng MichaIng modified the milestones: v8.18, v8.19 Jun 3, 2023
@MichaIng MichaIng removed the Solution available 🥂 Definite solution has been done label Jun 3, 2023
@MichaIng MichaIng reopened this Jun 3, 2023
MichaIng added a commit that referenced this issue Jun 9, 2023
@MichaIng
v8.19
182937b 
- CHANGELOG | Quartz64: Resolved an issue where some iptables/nftables features did not work as of missing kernel features. Many thanks to @acelinkit for reporting the issue and pointing us at the solution: #6389
@MichaIng MichaIng added the Solution available 🥂 Definite solution has been done label Jun 9, 2023
@MichaIng MichaIng closed this as completed Jun 9, 2023
This was referenced Jun 24, 2023
Merged
Closed
@MichaIng MichaIng mentioned this issue Jul 2, 2023
Merged
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@acelinkio acelinkio

Labels
Enhancement 💨 Quartz64 Solution available 🥂 Definite solution has been done
Projects
None yet
Milestone
v8.19
Development

Successfully merging a pull request may close this issue.

Feature/add soquartz kernel modules
Quartz64 | Add kernel features required for Kubernetes
2 participants
@MichaIng @acelinkio