failed to extract layer for installer, can't write /dev/mem

[ManoftheSea]

Hello,
I'm trying to make use of Talos on a cluster made up of a pair of espressobin SBCs. These systems seem to be well-supported in Linux. U-boot can bring them up as UEFI compliant boards, even, though I'm using PXE boot to load the Talos 1.6.7 arm64 image. When I load a configuration to either controlplane or worker, I seem to run into an issue with getting the container image.

There are two lines that appear to be errors. First, the log complains about being unable to write to /dev/mem. Then second, the system is unable to extract image layers:

[ 48.448352] [talos] controller failed {"component": "controller-runtime", "controller": "hardware.SystemInfoController", "error": "failed to open stream: open /dev/mem: no such file or directory"}

[ 83.475570] [talos] retrying error: failed to pull image "ghcr.io/siderolabs/installer:v1.6.7": failed to extract layer sha256:77f5705b72fd07adbe9c333ac28eb95e012d63ea02adcfebba3a5cba3ba80488: write /...

The end of the line is cut off, but I believe it continues with /system/containerd/snapshot/[an incrementing number]

I'll include some logs and configurations in hopes of it being useful. Would anyone know why the system is unable to extract the images? There's clearly network connectivity, as it is able to determine the hashes of the images, and subjectively it feels like it's taking about as long to download the image as my laptop does before giving the first instance of the error. I don't know how container images are compressed - could it be due to some change after the recent XZ issue?

tftp config - note that the espressobin DTBs are taken from a vanilla 6.1 arm64 kernel

LABEL Talos_Linux
  KERNEL /talos/latest/vmlinuz-arm64
  INITRD /talos/latest/initramfs-arm64.xz
  FDTDIR /dtbs/
  APPEND talos.platform=metal slab_nomerge pti=on init_on_alloc=1 init_on_free=1

[  138.224104] [talos] leaving maintenance service {"component": "controller-runtime", "controller": "config.AcquireController"}
[  138.374976] [talos] machine config loaded successfully {"component": "controller-runtime", "controller": "config.AcquireController", "sources": ["maintenance"]}
[  138.375202] [talos] task loadConfig (1/1): done, 1m38.392797129s
[  138.395795] [talos] phase config (10/11): done, 1m38.419081924s
[  138.465358] [talos] kubernetes endpoint watch error {"component": "controller-runtime", "controller": "k8s.EndpointController", "error": "failed to list *v1.Endpoints: Get \"https://coffee-endpoints.lab.seaofdirac.org:6443/api/v1/namespaces/default/endpoints?fieldSelector=meta}
[  139.646454] [talos] kubernetes endpoint watch error {"component": "controller-runtime", "controller": "k8s.EndpointController", "error": "failed to list *v1.Endpoints: Get \"https://coffee-endpoints.lab.seaofdirac.org:6443/api/v1/namespaces/default/endpoints?fieldSelector=meta}
[  141.971787] [talos] kubernetes endpoint watch error {"component": "controller-runtime", "controller": "k8s.EndpointController", "error": "failed to list *v1.Endpoints: Get \"https://coffee-endpoints.lab.seaofdirac.org:6443/api/v1/namespaces/default/endpoints?fieldSelector=meta}
[  143.536677] [talos] initialize sequence: done: 2m7.203429251s
[  148.234698] [talos] kubernetes endpoint watch error {"component": "controller-runtime", "controller": "k8s.EndpointController", "error": "failed to list *v1.Endpoints: Get \"https://coffee-endpoints.lab.seaofdirac.org:6443/api/v1/namespaces/default/endpoints?fieldSelector=meta}
[  148.485399] [talos] controller failed {"component": "controller-runtime", "controller": "k8s.NodeApplyController", "error": "1 error(s) occurred:\n\ttimeout"}
[  148.593468] [talos] install sequence: 11 phase(s)
[  148.598635] [talos] phase env (1/11): 1 tasks(s)
[  148.605000] [talos] task setUserEnvVars (1/1): starting
[  148.610835] [talos] task setUserEnvVars (1/1): done, 5.872238ms
[  148.618487] [talos] phase env (1/11): done, 19.980217ms
[  148.625189] [talos] phase install (2/11): 1 tasks(s)
[  148.630976] [talos] task install (1/1): starting
[  153.707334] [talos] pulling "ghcr.io/siderolabs/installer:v1.6.7"
[  158.970263] [talos] kubernetes endpoint watch error {"component": "controller-runtime", "controller": "k8s.EndpointController", "error": "failed to list *v1.Endpoints: Get \"https://coffee-endpoints.lab.seaofdirac.org:6443/api/v1/namespaces/default/endpoints?fieldSelector=meta}
[  159.175290] [talos] controller failed {"component": "controller-runtime", "controller": "k8s.NodeApplyController", "error": "1 error(s) occurred:\n\ttimeout"}
[  170.310056] [talos] controller failed {"component": "controller-runtime", "controller": "k8s.NodeApplyController", "error": "1 error(s) occurred:\n\ttimeout"}
[  173.436250] [talos] controller failed {"component": "controller-runtime", "controller": "hardware.SystemInfoController", "error": "failed to open stream: open /dev/mem: no such file or directory"}
[  177.335613] [talos] retrying error: failed to pull image "ghcr.io/siderolabs/installer:v1.6.7": failed to extract layer sha256:77f5705b72fd07adbe9c333ac28eb95e012d63ea02adcfebba3a5cba3ba80488: write /system/var/lib/containerd/io.containerd.snapshotter.v1.overlayfs/snapshots/2/n
[  178.532706] [talos] retrying error: failed to pull image "ghcr.io/siderolabs/installer:v1.6.7": failed to extract layer sha256:77f5705b72fd07adbe9c333ac28eb95e012d63ea02adcfebba3a5cba3ba80488: write /system/var/lib/containerd/io.containerd.snapshotter.v1.overlayfs/snapshots/3/n
[  181.559302] [talos] controller failed {"component": "controller-runtime", "controller": "k8s.NodeApplyController", "error": "1 error(s) occurred:\n\ttimeout"}
[  183.525870] [talos] kubernetes endpoint watch error {"component": "controller-runtime", "controller": "k8s.EndpointController", "error": "failed to list *v1.Endpoints: Get \"https://coffee-endpoints.lab.seaofdirac.org:6443/api/v1/namespaces/default/endpoints?fieldSelector=meta}
[  184.681717] [talos] retrying error: failed to pull image "ghcr.io/siderolabs/installer:v1.6.7": failed to extract layer sha256:77f5705b72fd07adbe9c333ac28eb95e012d63ea02adcfebba3a5cba3ba80488: write /system/var/lib/containerd/io.containerd.snapshotter.v1.overlayfs/snapshots/4/n
[  192.936289] [talos] controller failed {"component": "controller-runtime", "controller": "k8s.NodeApplyController", "error": "1 error(s) occurred:\n\ttimeout"}
[  200.846178] [talos] retrying error: failed to pull image "ghcr.io/siderolabs/installer:v1.6.7": failed to extract layer sha256:77f5705b72fd07adbe9c333ac28eb95e012d63ea02adcfebba3a5cba3ba80488: write /system/var/lib/containerd/io.containerd.snapshotter.v1.overlayfs/snapshots/5/n...

[smira]

I wonder if you don't have enough RAM available.

[ManoftheSea]

U-Boot 2023.01-armbian (Mar 10 2023 - 15:39:56 +0000)

DRAM:  1 GiB
Core:  47 devices, 24 uclasses, devicetree: fit
WDT:   Not starting watchdog@8300
Comphy chip #0:
Comphy-0: USB3_HOST0    5 Gbps
Comphy-1: PEX0          5 Gbps
Comphy-2: SATA0         6 Gbps
SATA link 0 timeout.
AHCI 0001.0300 32 slots 1 ports 6 Gbps 0x1 impl SATA mode
flags: ncq led only pmp fbss pio slum part sxs
PCIe: Link down
MMC:   sdhci@d0000: 0, sdhci@d8000: 1
Loading Environment from SPIFlash... SF: Detected is25wp032 with page size 256 Bytes, erase size 4 KiB, total 4 MiB
OK
Model: Globalscale Marvell ESPRESSOBin Board
Net:   eth0: ethernet@30000

U-boot says the board has 1 GB, and other linuxes have indicated a similar amount. The installer image is...

ghcr.io/siderolabs/installer   <none>    e79b48a028a2   3 weeks ago    310MB
ghcr.io/siderolabs/installer   v1.6.7    d3142364cc0e   3 weeks ago    504MB
ghcr.io/siderolabs/talos       v1.6.5    e2105510c08b   6 weeks ago    525MB
ghcr.io/siderolabs/talos       v1.5.5    7570ff60ec8a   5 months ago   526MB

Well, goodness, those are enormous! I'm not certain whether i'm looking at the amd64 or arm64 image, but i expect it is of similar size. Does this indicate that I will be never be able to run on such a small board?

[smira]

yes, I'd say 1 GiB is not enough in general (when including Kubernetes itself), as it won't almost leave anything for the workloads.

the installer size will go down in future versions of Talos