Job for k3s.service failed because a fatal signal was delivered causing the control process to dump core.

[PoweredByWill]

Environmental Info:

K3s Version: when you run k3s -v it states "illegal instruction" but the version from the install output reports "v1.21.7+k3s1"

Node(s) CPU architecture, OS, and Version:

Linux raspberrypi 5.10.63+ #1488 Thu Nov 18 16:14:04 GMT 2021 armv6l GNU/Linux

Cluster Configuration:

Couldn't get install to complete successfully

Describe the bug:

Install fails with: "Job for k3s.service failed because a fatal signal was delivered causing the control process to dump core."

Steps To Reproduce:

tried to install via: curl -sfL https://get.k3s.io | sh -

• Installed K3s:
  none

Expected behavior:

Install expected to complete successfully

Actual behavior:

Install finished but produces:
"Job for k3s.service failed because a fatal signal was delivered causing the control process to dump core."
Any running of k3s binaries produces: "Illegal instruction"

Additional context / logs:

$ systemctl status k3s.service
k3s.service - Lightweight Kubernetes
Loaded: loaded (/etc/systemd/system/k3s.service; enabled; vendor preset: enabled)
Active: activating (auto-restart) (Result: core-dump) since Sun 2021-12-05 20:04:09 GMT; 3s ago
Docs: https://k3s.io
Process: 1941 ExecStartPre=/bin/sh -xc ! /usr/bin/systemctl is-enabled --quiet nm-cloud-setup.service (code=exited, status=0/SUCCESS)
Process: 1943 ExecStartPre=/sbin/modprobe br_netfilter (code=exited, status=0/SUCCESS)
Process: 1944 ExecStartPre=/sbin/modprobe overlay (code=exited, status=0/SUCCESS)
Process: 1945 ExecStart=/usr/local/bin/k3s server (code=dumped, signal=ILL)
Main PID: 1945 (code=dumped, signal=ILL)

$ journalctl -xe
Dec 05 20:05:47 raspberrypi systemd[1]: Starting Lightweight Kubernetes...
-- Subject: A start job for unit k3s.service has begun execution
-- Defined-By: systemd
-- Support: https://www.debian.org/support
-- 
-- A start job for unit k3s.service has begun execution.
-- 
-- The job identifier is 12727.
Dec 05 20:05:47 raspberrypi sh[2034]: + /usr/bin/systemctl is-enabled --quiet nm-cloud-setup.service
Dec 05 20:05:47 raspberrypi sh[2034]: Failed to get unit file state for nm-cloud-setup.service: No such file or directory
Dec 05 20:05:48 raspberrypi systemd[1]: k3s.service: Main process exited, code=dumped, status=4/ILL
-- Subject: Unit process exited
-- Defined-By: systemd
-- Support: https://www.debian.org/support
-- 
-- An ExecStart= process belonging to unit k3s.service has exited.
-- 
-- The process' exit code is 'dumped' and its exit status is 4.
Dec 05 20:05:48 raspberrypi systemd[1]: k3s.service: Failed with result 'core-dump'.
-- Subject: Unit failed
-- Defined-By: systemd
-- Support: https://www.debian.org/support
-- 
-- The unit k3s.service has entered the 'failed' state with result 'core-dump'.
Dec 05 20:05:48 raspberrypi systemd[1]: Failed to start Lightweight Kubernetes.
-- Subject: A start job for unit k3s.service has failed
-- Defined-By: systemd
-- Support: https://www.debian.org/support
-- 
-- A start job for unit k3s.service has finished with a failure.
-- 
-- The job identifier is 12727 and the job result is failed.

Backporting

• Needs backporting to older releases

[VestigeJ]

I wonder if this is armv7 specific.

$ curl -sfL https://get.k3s.io | sh -

[INFO] Finding release for channel stable
[INFO] Using v1.21.7+k3s1 as release
[INFO] Downloading hash https://github.com/k3s-io/k3s/releases/download/v1.21.7+k3s1/sha256sum-arm64.txt
[INFO] Downloading binary https://github.com/k3s-io/k3s/releases/download/v1.21.7+k3s1/k3s-arm64
[INFO] Verifying binary download
[INFO] Installing k3s to /usr/local/bin/k3s
[INFO] Skipping installation of SELinux RPM
[INFO] Creating /usr/local/bin/kubectl symlink to k3s
[INFO] Creating /usr/local/bin/crictl symlink to k3s
[INFO] Skipping /usr/local/bin/ctr symlink to k3s, command exists in PATH at /usr/bin/ctr
[INFO] Creating killall script /usr/local/bin/k3s-killall.sh
[INFO] Creating uninstall script /usr/local/bin/k3s-uninstall.sh
[INFO] env: Creating environment file /etc/systemd/system/k3s.service.env
[INFO] systemd: Creating service file /etc/systemd/system/k3s.service
[INFO] systemd: Enabling k3s unit
Created symlink /etc/systemd/system/multi-user.target.wants/k3s.service → /etc/systemd/system/k3s.service.
[INFO] systemd: Starting k3s

$ uname -a

Linux mammatus-Cloud 4.9.253-tegra #1 SMP PREEMPT Mon Jul 26 12:13:06 PDT 2021 aarch64 aarch64 aarch64 GNU/Linux

Everything running as expected here. :/

What’s the output of running $ sudo k3s server?

[PoweredByWill]

pi@raspberrypi:~ $ sudo k3s server  
Illegal instruction  
pi@raspberrypi:~ $ 

I also tried apt updating everything.
And looked around for some dumped cores since the installer reported it but nada.

🤷‍♀️

[PoweredByWill]

pi@raspberrypi:~ $ curl -sfL https://get.k3s.io | INSTALL_K3S_VERSION=v0.9.1 sh -
[INFO]  Using v0.9.1 as release
[INFO]  Downloading hash https://github.com/k3s-io/k3s/releases/download/v0.9.1/sha256sum-arm.txt
[INFO]  Downloading binary https://github.com/k3s-io/k3s/releases/download/v0.9.1/k3s-armhf
[INFO]  Verifying binary download
[INFO]  Installing k3s to /usr/local/bin/k3s
[INFO]  Skipping installation of SELinux RPM
[INFO]  Creating /usr/local/bin/kubectl symlink to k3s
[INFO]  Creating /usr/local/bin/crictl symlink to k3s
[INFO]  Creating /usr/local/bin/ctr symlink to k3s
[INFO]  Creating killall script /usr/local/bin/k3s-killall.sh
[INFO]  Creating uninstall script /usr/local/bin/k3s-uninstall.sh
[INFO]  env: Creating environment file /etc/systemd/system/k3s.service.env
[INFO]  systemd: Creating service file /etc/systemd/system/k3s.service
[INFO]  systemd: Enabling k3s unit
Created symlink /etc/systemd/system/multi-user.target.wants/k3s.service → /etc/systemd/system/k3s.service.
[INFO]  systemd: Starting k3s

Version 0.9.1 installs so that's something, but...

pi@raspberrypi:~ $ vmstat 1 5
procs -----------memory---------- ---swap-- -----io---- -system-- ------cpu-----
 r  b   swpd   free   buff  cache   si   so    bi    bo   in   cs us sy id wa st
13  0 102356  39204   6080 111716    5   13    98   113  192  357 13  8 78  1  0
13  0 102356  37536   6400 111732    0    0   308   208 1023 2619 57 43  0  0  0
14  0 102396  37140   6204 111392    0   36   128   340 1365 3312 53 47  0  0  0
18  0 102396  38760   6088 111312    0    0   372   140 1481 3414 33 67  0  0  0
16  0 102396  40260   5916 108612    0   28   672    28 1545 4163 39 61  0  0  0
pi@raspberrypi:~ $ 

Maybe I'll try back a release release or two...

[VestigeJ]

Is this a 3b+ pi or one of the newer pi 4’s? I have a 3b+ i can try and report back.

[brandond]

Yes, K3s requires at least arm v7.

k3s/scripts/build

Line 68 in a8f7e9f

export GOARM="7"

What model Pi do you have? It should work fine on 3 and 4.

[PoweredByWill]

Welp, that must be it.
It's a PiZeroW.

I saw that some folks had it working on 0.9.1 and it did work on mine yesterday via:
curl -sfL https://get.k3s.io | INSTALL_K3S_VERSION=v0.9.1 sh -
so I figured it wouldn't be that, especially with IoT intentions of k3s.
ARMv6 isn't IoT-ish any longer? (Might be a good reason to get the PiZero2w now)

I went through several of the pages of the releases to see if support was dropped or modified at some point but couldn't find anything, but it was sort of needle in the haystack searching.

Is there a release notes or thread that discusses the change?

Appreciate all the help.

[brandond]

We had only ever intended to support armv7; we'd been setting GOARM=7 since v0.1.0-rc3 but for various reasons it wasn't being honored. That was fixed in 1047463 which was released starting with the 1.20 branch.

[PoweredByWill]

Thanks.

I'll probably get in line for a PiZeroW2 (I use my setup for a light travel server) and go from there.

Again, thanks for the input and feedback.