library/rpi #2083

bobmorane83 · 2024-05-16T15:36:56Z

Hi all,
(BTW I'am totally newbee to elemental ...)

elemental-toolkit version:
❯ docker run -it --rm ghcr.io/rancher/elemental-toolkit/elemental-cli:latest version
v1.3.0+g56b2723

CPU architecture, OS, and Version:
On my RPi : Linux pi 6.6.28+rpt-rpi-v8 #1 SMP PREEMPT Debian 1:6.6.28-1+rpt1 (2024-04-22) aarch64 GNU/Linux
Same on my Mac M1

Describe the bug
❯ docker run -it --rm ghcr.io/rancher/elemental-toolkit/elemental-cli:latest build-iso --config-dir=./ rpi
INFO[2024-05-16T13:49:52Z] Starting elemental version v1.3.0
INFO[2024-05-16T13:49:52Z] Reading configuration from './'
INFO[2024-05-16T13:49:52Z] Preparing squashfs root (1 source)...
INFO[2024-05-16T13:49:52Z] Copying rpi:latest source...
ERRO[2024-05-16T13:50:05Z] Failed installing OS packages: GET https://index.docker.io/v2/library/rpi/manifests/latest: UNAUTHORIZED: authentication required; [map[Action:pull Class: Name:library/rpi Type:repository]]

To Reproduce
❯ docker run -it --rm ghcr.io/rancher/elemental-toolkit/elemental-cli:latest build-iso --config-dir=./ rpi

I'am trying to generate a Raspberry Pi 4 image to boot on, but seems that the repositoty is not available anymore.

Did I missed something ?
"docker build --build-arg TOOLKIT_REPO=ghcr.io/rancher/elemental-toolkit/elemental-cli --build-arg VERSION=latest -t rpi ." was ok.

Any clue ?
Thanks,
BR,

frelon · 2024-05-16T16:53:33Z

Hi @bobmorane83!

There is a separate make target for building an RPI disk.

Try running 'make ARCH=arm64 build build-os build-rpi-disk' and it should build the toolkit, green OS container image and finally a raw image that can be written to an sd-card and booted!

Let us know how it goes! 👍

bobmorane83 · 2024-05-16T21:33:06Z

Thanks @frelon,
Now I got :

Building arm64 disk
mkdir -p /home/pi/Developement/elemental/elemental-toolkit/build
docker run --rm -v /var/run/docker.sock:/var/run/docker.sock -v /home/pi/Developement/elemental/elemental-toolkit/examples:/examples -v /home/pi/Developement/elemental/elemental-toolkit/build:/build
--entrypoint /usr/bin/elemental
local/elemental-toolkit:v2.1.0-dev-gc608e0b08 --debug build-disk --platform linux/arm64 --cloud-init-paths /examples/green --unprivileged --expandable -n elemental-green.aarch64 --local
--squash-no-compression --deploy-command elemental,--debug,reset,--reboot,--disable-boot-entry -o /build local/elemental-green:v2.1.0-dev-gc608e0b08
Error: unknown flag: --unprivileged

Removing the --unprivileged flag I got :

docker run --rm -v /var/run/docker.sock:/var/run/docker.sock -v /home/pi/Developement/elemental/elemental-toolkit/examples:/examples -v /home/pi/Developement/elemental/elemental-toolkit/build:/build
--entrypoint /usr/bin/elemental
local/elemental-toolkit:v2.1.0-dev-gc608e0b08 --debug build-disk --platform linux/arm64 --cloud-init-paths /examples/green --expandable -n elemental-green.aarch64 --local
--squash-no-compression --deploy-command elemental,--debug,reset,--reboot,--disable-boot-entry -o /build local/elemental-green:v2.1.0-dev-gc608e0b08
Error: accepts 0 arg(s), received 1

I also tried with tag v2.1.0 but same errors.

Any comment ?
Thx,

bobmorane83 · 2024-05-16T22:59:43Z

Ok, if i use FLAVOR to green-rpi, remove --unprivileged and add --system, and make build-rpi-disk, I got a raw file, flashed on SD card and get the following ;

-Boot 2021.10 (Aug 04 2022 - 12:00:00 +0000)

DRAM: 2 GiB
RPI 4 Model B (0xb03114)
MMC: mmcnr@7e300000: 1, mmc@7e340000: 0
Loading Environment from FAT... Unable to read "uboot.env" from mmc0:1... In: serial
Booting `Elemental recovery'

U-Boot 2021.10 (Aug 04 2022 - 12:00:00 +0000)

DRAM: 2 GiB
RPI 4 Model B (0xb03114)
Booting `Elemental'

error: ../../grub-core/commands/search.c:296:no such device: COS_STATE.
error: ../../grub-core/fs/fshelp.c:258:file `/.snapshots/active' not found.
error: ../../grub-core/net/net.c:1317:no server is specified.
error: ../../grub-core/loader/arm64/efi/linux.c:301:you need to load the kernel
first.

Press any key to continue...

Falling back to `Elemental'

error: ../../grub-core/commands/search.c:296:no such device: COS_STATE.
error: ../../grub-core/fs/fshelp.c:258:file `/.snapshots/active' not found.
error: ../../grub-core/net/net.c:1317:no server is specified.
error: ../../grub-core/loader/arm64/efi/linux.c:301:you need to load the kernel
first.

Press any key to continue...

Falling back to `Elemental recovery'

EFI stub: Booting Linux Kernel...
EFI stub: Using DTB from configuration table
EFI stub: Exiting boot services...

Any doc to read to understand the next steps ?

Thanks,
BR

frelon · 2024-05-17T07:11:40Z

Good catch! If you want to put up a PR with the changes I would be happy to review!

From the logs it looks like it's trying to boot the linux kernel from recovery, which is sort of the expected behavior.. when building expandable images we boot into the recovery partition to expand the partition table correctly and then we reboot into the active system. This might take a while, do you get any more output after the EFI stub: Exiting boot services... log?

bobmorane83 · 2024-05-17T10:22:10Z

Waiting for an hour or so and nothing appends ...
GRUB look like this and key in do not works :

bobmorane83 · 2024-05-17T13:37:52Z

Good news :
After waiting enought :

ssh 192.168.2.194 -l pi
Password:

But what password ? elemental ? nop ...
in user-data I have :

users:
- name: pi
  groups: users,adm,dialout,audio,netdev,video,plugdev,cdrom,games,input,gpio,spi,i2c,render,sudo
  shell: /bin/bash
  lock_passwd: false
  passwd: $5$hJfZB7KyQ3$nZYJJcuk6mlRH/evlBMRmYzgXUrRde3WlAh9.yzYW2B

But what kind of hash ?

frelon · 2024-05-17T13:41:12Z

Awesome! The default login for the examples is root/elemental.. But maybe root login is not enabled for rpi by default. You could probably copy the line here to green-rpi dockerfile.

bobmorane83 · 2024-05-17T13:42:19Z

Was : https://rancher.github.io/elemental-toolkit/docs/customizing/login/

❯ ssh 192.168.2.194 -l root
Password: 
Last failed login: Fri May 17 12:59:57 UTC 2024 from 192.168.2.151 on ssh:notty
There were 2 failed login attempts since the last successful login.
Have a lot of fun...
elemental:~ #

frelon · 2024-05-17T15:09:25Z

Nice find! Yea we recently switched from cos to elemental as the example password.. Glad you got it working! 👍

bobmorane83 · 2024-05-21T12:06:03Z

Ok, now the RPi is booting as expecting : First in recovery, and after a delay( few minutes) rebooting in "elemental" and picking up a DHCP IP (BTW "elemental" is now the normal password as you mentionned).
I have added k3s and k3s-agent to the container and I now creating a cluster with elemental node as worker and standard OS Pi 4 with k3s as master.

Everything is Ok at first :

If I reboot the worker node (elemental1), the node do not reconnect by itself, I get :

msg="Waiting to retrieve agent configuration; server is not ready: Node password rejected, duplicate hostname or contents of '/etc/rancher/node/password' may not match server node-passwd entry, try enabling a unique node name with the --with-node-id flag"

The node has unique hostname and adding --with-node-id add effectively an ID but different after each boot.

I have, in the master node, to delete the worker node elemental1 to get the node back and ready.

BTW I had to add "--snapshotter=native" option to get k3s running on elemental in /etc/systemd/system/k3s-agent.service. All other option (and token) are stored in /etc/systemd/system/k3s-agent.service.env.

Any clue ?
Thanks,

frelon · 2024-05-21T12:25:22Z

Very nice work!

/etc/rancher is probably ephemeral and will be cleared during reboot, in elemental we add that dir as persistent here

So you could probably copy that file and put it in /system/oem and it should persist the /etc/rancher/node/password file 👍

bobmorane83 · 2024-05-21T13:05:25Z

Effectively putting back the original password in /etc/rancher/node/password the node connect itself right away 👍
But I didn't catch how to copy the file in the read only /system/oem (can't remount it in rw) and say to k3s-agent to access there ?

frelon · 2024-05-21T13:39:18Z

You can put it in /oem also, that should be writeable by default!

bobmorane83 · 2024-05-21T19:17:19Z

Adding

               ephemeral:
                type: tmpfs
                size: 25%
                paths: ['/etc', '/var', '/srv']
              persistent:
                mode: bind
                paths:
                - /etc/systemd
                - /etc/rancher
                - /etc/ssh
                - /etc/iscsi
                - /etc/cni
                - /home
                - /opt
                - /root
                - /usr/libexec
                - /usr/local
                - /var/log
                - /var/lib/elemental
                - /var/lib/rancher
                - /var/lib/kubelet
                - /var/lib/NetworkManager
                - /var/lib/cni
                - /var/lib/calico

to pkg/features/embedded/cloud-config-defaults/system/oem/00_layout.yamldid not do the trick.
I think, I'am still have to understand elemental better ...
Any help ?
Thanks,

frelon · 2024-05-22T07:34:12Z

Okay, if you could post the entire file contents here that would help.. It looks like just adding it there as is will only run when booted to recovery because of this conditional: - if: '[ ! -f "/run/elemental/recovery_mode" ]'

bobmorane83 · 2024-05-22T07:50:37Z

I simply copy/past the part of framework/files/system/oem/01_elemental-rootfs.yamlfrom elemental repo :

# Layout Elemental OEM configuration file
#
# This file is part of Elemental and will get reset during upgrades.
#
# Before you change this file manually,
# consider copying this file to /usr/local/cloud-config or
# copy the file with a prefix starting by 90, e.g. /oem/91_custom.yaml
name: "Rootfs Layout Settings"
stages:
  rootfs:
    - if: '[ ! -f "/run/elemental/recovery_mode" ]'
      name: "Layout configuration"
      files: 
      - path: /run/elemental/config.d/layout.yaml
        content: |
          mount:
            write-fstab: true
            ephemeral:
              type: tmpfs
              size: 25%
              paths: ['/etc', '/var', '/srv']
            persistent:
              mode: bind
              paths:
              - /etc/systemd
              - /etc/rancher
              - /etc/ssh
              - /etc/iscsi
              - /etc/cni
              - /home
              - /opt
              - /root
              - /usr/libexec
              - /usr/local
              - /var/log
              - /var/lib/elemental
              - /var/lib/rancher
              - /var/lib/kubelet
              - /var/lib/NetworkManager
              - /var/lib/cni
              - /var/lib/calico

Another option I was thinking of is to add :

name: "Run on initramfs"
stages:
   initramfs:
     - name: "Rancher Setting"
       if: '[ ! -f "/run/cos/recovery_mode" ]'
       commands:
       - |
          ln -s /oem /etc/rancher

frelon · 2024-05-22T08:05:32Z

Could you post the output of mount and journalctl -u elemental-setup-rootfs?

bobmorane83 · 2024-05-22T08:55:06Z

elemental:~ # mount
proc on /proc type proc (rw,nosuid,nodev,noexec,relatime)
sysfs on /sys type sysfs (rw,nosuid,nodev,noexec,relatime)
devtmpfs on /dev type devtmpfs (rw,nosuid,size=4096k,nr_inodes=1048576,mode=755,inode64)
securityfs on /sys/kernel/security type securityfs (rw,nosuid,nodev,noexec,relatime)
tmpfs on /dev/shm type tmpfs (rw,nosuid,nodev,inode64)
devpts on /dev/pts type devpts (rw,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000)
tmpfs on /run type tmpfs (rw,nosuid,nodev,size=385324k,nr_inodes=819200,mode=755,inode64)
tmpfs on /sys/fs/cgroup type tmpfs (ro,nosuid,nodev,noexec,size=4096k,nr_inodes=1024,mode=755,inode64)
cgroup2 on /sys/fs/cgroup/unified type cgroup2 (rw,nosuid,nodev,noexec,relatime,nsdelegate)
cgroup on /sys/fs/cgroup/systemd type cgroup (rw,nosuid,nodev,noexec,relatime,xattr,name=systemd)
pstore on /sys/fs/pstore type pstore (rw,nosuid,nodev,noexec,relatime)
efivarfs on /sys/firmware/efi/efivars type efivarfs (ro,nosuid,nodev,noexec,relatime)
bpf on /sys/fs/bpf type bpf (rw,nosuid,nodev,noexec,relatime,mode=700)
cgroup on /sys/fs/cgroup/memory type cgroup (rw,nosuid,nodev,noexec,relatime,memory)
cgroup on /sys/fs/cgroup/blkio type cgroup (rw,nosuid,nodev,noexec,relatime,blkio)
cgroup on /sys/fs/cgroup/pids type cgroup (rw,nosuid,nodev,noexec,relatime,pids)
cgroup on /sys/fs/cgroup/cpu,cpuacct type cgroup (rw,nosuid,nodev,noexec,relatime,cpu,cpuacct)
cgroup on /sys/fs/cgroup/hugetlb type cgroup (rw,nosuid,nodev,noexec,relatime,hugetlb)
cgroup on /sys/fs/cgroup/misc type cgroup (rw,nosuid,nodev,noexec,relatime,misc)
cgroup on /sys/fs/cgroup/rdma type cgroup (rw,nosuid,nodev,noexec,relatime,rdma)
cgroup on /sys/fs/cgroup/freezer type cgroup (rw,nosuid,nodev,noexec,relatime,freezer)
cgroup on /sys/fs/cgroup/net_cls,net_prio type cgroup (rw,nosuid,nodev,noexec,relatime,net_cls,net_prio)
cgroup on /sys/fs/cgroup/devices type cgroup (rw,nosuid,nodev,noexec,relatime,devices)
cgroup on /sys/fs/cgroup/cpuset type cgroup (rw,nosuid,nodev,noexec,relatime,cpuset)
cgroup on /sys/fs/cgroup/perf_event type cgroup (rw,nosuid,nodev,noexec,relatime,perf_event)
configfs on /sys/kernel/config type configfs (rw,nosuid,nodev,noexec,relatime)
/dev/mmcblk0p4 on /run/initramfs/elemental-state type ext4 (rw,relatime)
/run/initramfs/elemental-state/.snapshots/1/snapshot.img on / type ext2 (ro,relatime)
/dev/mmcblk0p2 on /oem type ext4 (rw,relatime)
/dev/mmcblk0p1 on /run/elemental/efi type vfat (ro,relatime,fmask=0022,dmask=0022,codepage=437,iocharset=iso8859-1,shortname=mixed,errors=remount-ro)
/dev/mmcblk0p5 on /run/elemental/persistent type ext4 (rw,relatime)
tmpfs on /run/elemental/overlay type tmpfs (rw,relatime,size=481656k,inode64)
overlay on /etc type overlay (rw,relatime,lowerdir=/sysroot/etc,upperdir=/run/elemental/overlay/etc.overlay/upper,workdir=/run/elemental/overlay/etc.overlay/work)
overlay on /var type overlay (rw,relatime,lowerdir=/sysroot/var,upperdir=/run/elemental/overlay/var.overlay/upper,workdir=/run/elemental/overlay/var.overlay/work)
overlay on /srv type overlay (rw,relatime,lowerdir=/sysroot/srv,upperdir=/run/elemental/overlay/srv.overlay/upper,workdir=/run/elemental/overlay/srv.overlay/work)
/dev/mmcblk0p5 on /home type ext4 (rw,relatime)
/dev/mmcblk0p5 on /root type ext4 (rw,relatime)
/dev/mmcblk0p5 on /opt type ext4 (rw,relatime)
selinuxfs on /sys/fs/selinux type selinuxfs (rw,nosuid,noexec,relatime)
systemd-1 on /proc/sys/fs/binfmt_misc type autofs (rw,relatime,fd=30,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=18473)
hugetlbfs on /dev/hugepages type hugetlbfs (rw,relatime,pagesize=2M)
mqueue on /dev/mqueue type mqueue (rw,nosuid,nodev,noexec,relatime)
debugfs on /sys/kernel/debug type debugfs (rw,nosuid,nodev,noexec,relatime)
tracefs on /sys/kernel/tracing type tracefs (rw,nosuid,nodev,noexec,relatime)
tmpfs on /tmp type tmpfs (rw,nosuid,nodev,nr_inodes=1048576,inode64)
fusectl on /sys/fs/fuse/connections type fusectl (rw,nosuid,nodev,noexec,relatime)
none on /run/credentials/systemd-sysusers.service type ramfs (ro,nosuid,nodev,noexec,relatime,mode=700)
tmpfs on /run/user/0 type tmpfs (rw,nosuid,nodev,relatime,size=192660k,nr_inodes=48165,mode=700,inode64)

elemental:~ # journalctl -u elemental-setup-rootfs
Jan 08 12:00:24 recovery.lan systemd[1]: Starting Elemental system early rootfs setup...
Jan 08 12:00:24 recovery.lan elemental[662]: INFO[2024-01-08T12:00:24Z] Starting elemental version v2.1.0
Jan 08 12:00:24 recovery.lan elemental[662]: INFO[2024-01-08T12:00:24Z] Reading configuration from '/etc/elemental'
Jan 08 12:00:24 recovery.lan elemental[662]: INFO[2024-01-08T12:00:24Z] Running stage: rootfs.before
Jan 08 12:00:24 recovery.lan elemental[662]: INFO[2024-01-08T12:00:24Z] Processing stage step 'Pull data from provider'. ( commands: 0, files: 0, ... )
Jan 08 12:00:24 recovery.lan elemental[662]: WARN[2024-01-08T12:00:24Z] (conditional) Skip 'Skipping stage (if statement error: failed to run cat /proc/cmdline | grep -q "CDLABEL" || cat /proc/cmdline | grep -q "elemental.disable"  || cat /proc/cmdline | grep -q "rd.cos.disable"
Jan 08 12:00:24 recovery.lan elemental[662]: : exit status 1)' stage name: Identify live mode
Jan 08 12:00:25 recovery.lan elemental[662]: ERRO[2024-01-08T12:00:25Z] no metadata/userdata found
Jan 08 12:00:25 recovery.lan elemental[662]: WARN[2024-01-08T12:00:25Z] (conditional) Skip 'Skipping stage (if statement error: failed to run [ -f "/run/elemental/live_mode" ]: exit status 1)' stage name: Include cloud init files from the ISO root
Jan 08 12:00:25 recovery.lan elemental[662]: INFO[2024-01-08T12:00:25Z] Done executing stage 'rootfs.before'
Jan 08 12:00:25 recovery.lan elemental[662]: INFO[2024-01-08T12:00:25Z] Running stage: rootfs
Jan 08 12:00:25 recovery.lan elemental[662]: INFO[2024-01-08T12:00:25Z] Processing stage step 'Layout configuration'. ( commands: 0, files: 1, ... )
Jan 08 12:00:25 recovery.lan elemental[662]: INFO[2024-01-08T12:00:25Z] Done executing stage 'rootfs'
Jan 08 12:00:25 recovery.lan elemental[662]: INFO[2024-01-08T12:00:25Z] Running stage: rootfs.after
Jan 08 12:00:25 recovery.lan elemental[662]: INFO[2024-01-08T12:00:25Z] Done executing stage 'rootfs.after'
Jan 08 12:00:25 recovery.lan elemental[662]: INFO[2024-01-08T12:00:25Z] Running stage: rootfs.before
Jan 08 12:00:25 recovery.lan elemental[662]: INFO[2024-01-08T12:00:25Z] Done executing stage 'rootfs.before'
Jan 08 12:00:25 recovery.lan elemental[662]: INFO[2024-01-08T12:00:25Z] Running stage: rootfs
Jan 08 12:00:25 recovery.lan elemental[662]: INFO[2024-01-08T12:00:25Z] Done executing stage 'rootfs'
Jan 08 12:00:25 recovery.lan elemental[662]: INFO[2024-01-08T12:00:25Z] Running stage: rootfs.after
Jan 08 12:00:25 recovery.lan elemental[662]: INFO[2024-01-08T12:00:25Z] Done executing stage 'rootfs.after'
Jan 08 12:00:25 recovery.lan elemental[662]: INFO[2024-01-08T12:00:25Z] Some errors found but were ignored. Enable --strict mode to fail on those or --debug to see them in the log
Jan 08 12:00:25 recovery.lan elemental[662]: WARN[2024-01-08T12:00:25Z] 1 error occurred:
Jan 08 12:00:25 recovery.lan elemental[662]:         * no metadata/userdata found
Jan 08 12:00:25 recovery.lan elemental[662]:
Jan 08 12:00:25 recovery.lan systemd[1]: Finished Elemental system early rootfs setup.

frelon · 2024-05-22T09:10:40Z

Hmm, looks like the default configuration on the recovery.. does it look the same when booted to the active snapshot?

bobmorane83 · 2024-05-22T09:38:39Z

Sound the same ...

elemental:~ # mount
proc on /proc type proc (rw,nosuid,nodev,noexec,relatime)
sysfs on /sys type sysfs (rw,nosuid,nodev,noexec,relatime)
devtmpfs on /dev type devtmpfs (rw,nosuid,size=4096k,nr_inodes=1048576,mode=755,inode64)
securityfs on /sys/kernel/security type securityfs (rw,nosuid,nodev,noexec,relatime)
tmpfs on /dev/shm type tmpfs (rw,nosuid,nodev,inode64)
devpts on /dev/pts type devpts (rw,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000)
tmpfs on /run type tmpfs (rw,nosuid,nodev,size=385324k,nr_inodes=819200,mode=755,inode64)
tmpfs on /sys/fs/cgroup type tmpfs (ro,nosuid,nodev,noexec,size=4096k,nr_inodes=1024,mode=755,inode64)
cgroup2 on /sys/fs/cgroup/unified type cgroup2 (rw,nosuid,nodev,noexec,relatime,nsdelegate)
cgroup on /sys/fs/cgroup/systemd type cgroup (rw,nosuid,nodev,noexec,relatime,xattr,name=systemd)
pstore on /sys/fs/pstore type pstore (rw,nosuid,nodev,noexec,relatime)
efivarfs on /sys/firmware/efi/efivars type efivarfs (ro,nosuid,nodev,noexec,relatime)
bpf on /sys/fs/bpf type bpf (rw,nosuid,nodev,noexec,relatime,mode=700)
cgroup on /sys/fs/cgroup/freezer type cgroup (rw,nosuid,nodev,noexec,relatime,freezer)
cgroup on /sys/fs/cgroup/perf_event type cgroup (rw,nosuid,nodev,noexec,relatime,perf_event)
cgroup on /sys/fs/cgroup/memory type cgroup (rw,nosuid,nodev,noexec,relatime,memory)
cgroup on /sys/fs/cgroup/net_cls,net_prio type cgroup (rw,nosuid,nodev,noexec,relatime,net_cls,net_prio)
cgroup on /sys/fs/cgroup/pids type cgroup (rw,nosuid,nodev,noexec,relatime,pids)
cgroup on /sys/fs/cgroup/blkio type cgroup (rw,nosuid,nodev,noexec,relatime,blkio)
cgroup on /sys/fs/cgroup/hugetlb type cgroup (rw,nosuid,nodev,noexec,relatime,hugetlb)
cgroup on /sys/fs/cgroup/rdma type cgroup (rw,nosuid,nodev,noexec,relatime,rdma)
cgroup on /sys/fs/cgroup/devices type cgroup (rw,nosuid,nodev,noexec,relatime,devices)
cgroup on /sys/fs/cgroup/cpu,cpuacct type cgroup (rw,nosuid,nodev,noexec,relatime,cpu,cpuacct)
cgroup on /sys/fs/cgroup/misc type cgroup (rw,nosuid,nodev,noexec,relatime,misc)
cgroup on /sys/fs/cgroup/cpuset type cgroup (rw,nosuid,nodev,noexec,relatime,cpuset)
configfs on /sys/kernel/config type configfs (rw,nosuid,nodev,noexec,relatime)
/dev/mmcblk0p4 on /run/initramfs/elemental-state type ext4 (rw,relatime)
/run/initramfs/elemental-state/.snapshots/1/snapshot.img on / type ext2 (ro,relatime)
/dev/mmcblk0p2 on /oem type ext4 (rw,relatime)
/dev/mmcblk0p1 on /run/elemental/efi type vfat (ro,relatime,fmask=0022,dmask=0022,codepage=437,iocharset=iso8859-1,shortname=mixed,errors=remount-ro)
/dev/mmcblk0p5 on /run/elemental/persistent type ext4 (rw,relatime)
tmpfs on /run/elemental/overlay type tmpfs (rw,relatime,size=481656k,inode64)
overlay on /var type overlay (rw,relatime,lowerdir=/sysroot/var,upperdir=/run/elemental/overlay/var.overlay/upper,workdir=/run/elemental/overlay/var.overlay/work)
overlay on /etc type overlay (rw,relatime,lowerdir=/sysroot/etc,upperdir=/run/elemental/overlay/etc.overlay/upper,workdir=/run/elemental/overlay/etc.overlay/work)
overlay on /srv type overlay (rw,relatime,lowerdir=/sysroot/srv,upperdir=/run/elemental/overlay/srv.overlay/upper,workdir=/run/elemental/overlay/srv.overlay/work)
overlay on /home type overlay (rw,relatime,lowerdir=/sysroot/home,upperdir=/run/elemental/persistent/.state/home.overlay/upper,workdir=/run/elemental/persistent/.state/home.overlay/work)
overlay on /opt type overlay (rw,relatime,lowerdir=/sysroot/opt,upperdir=/run/elemental/persistent/.state/opt.overlay/upper,workdir=/run/elemental/persistent/.state/opt.overlay/work)
overlay on /root type overlay (rw,relatime,lowerdir=/sysroot/root,upperdir=/run/elemental/persistent/.state/root.overlay/upper,workdir=/run/elemental/persistent/.state/root.overlay/work)
overlay on /etc/systemd type overlay (rw,relatime,lowerdir=/sysroot/etc/systemd,upperdir=/run/elemental/persistent/.state/etc-systemd.overlay/upper,workdir=/run/elemental/persistent/.state/etc-systemd.overlay/work)
overlay on /etc/ssh type overlay (rw,relatime,lowerdir=/sysroot/etc/ssh,upperdir=/run/elemental/persistent/.state/etc-ssh.overlay/upper,workdir=/run/elemental/persistent/.state/etc-ssh.overlay/work)
overlay on /var/log type overlay (rw,relatime,lowerdir=/sysroot/var/log,upperdir=/run/elemental/persistent/.state/var-log.overlay/upper,workdir=/run/elemental/persistent/.state/var-log.overlay/work)
selinuxfs on /sys/fs/selinux type selinuxfs (rw,nosuid,noexec,relatime)
systemd-1 on /proc/sys/fs/binfmt_misc type autofs (rw,relatime,fd=30,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=18970)
hugetlbfs on /dev/hugepages type hugetlbfs (rw,relatime,pagesize=2M)
mqueue on /dev/mqueue type mqueue (rw,nosuid,nodev,noexec,relatime)
debugfs on /sys/kernel/debug type debugfs (rw,nosuid,nodev,noexec,relatime)
tracefs on /sys/kernel/tracing type tracefs (rw,nosuid,nodev,noexec,relatime)
tmpfs on /tmp type tmpfs (rw,nosuid,nodev,size=963312k,nr_inodes=1048576,inode64)
fusectl on /sys/fs/fuse/connections type fusectl (rw,nosuid,nodev,noexec,relatime)
none on /run/credentials/systemd-sysusers.service type ramfs (ro,nosuid,nodev,noexec,relatime,mode=700)
tmpfs on /run/user/0 type tmpfs (rw,nosuid,nodev,relatime,size=192660k,nr_inodes=48165,mode=700,inode64)

elemental:~ # journalctl -u elemental-setup-rootfs
Jan 08 12:00:24 recovery.lan systemd[1]: Starting Elemental system early rootfs setup...
Jan 08 12:00:24 recovery.lan elemental[661]: INFO[2024-01-08T12:00:24Z] Starting elemental version v2.1.0
Jan 08 12:00:24 recovery.lan elemental[661]: INFO[2024-01-08T12:00:24Z] Reading configuration from '/etc/elemental'
Jan 08 12:00:24 recovery.lan elemental[661]: INFO[2024-01-08T12:00:24Z] Running stage: rootfs.before
Jan 08 12:00:24 recovery.lan elemental[661]: INFO[2024-01-08T12:00:24Z] Processing stage step 'Pull data from provider'. ( commands: 0, files: 0, ... )
Jan 08 12:00:24 recovery.lan elemental[661]: WARN[2024-01-08T12:00:24Z] (conditional) Skip 'Skipping stage (if statement error: failed to run cat /proc/cmdline | grep -q "CDLABEL" || cat /proc/cmdline | grep -q "elemental.disable"  || cat /proc/cmdline | grep -q "rd.cos.disable"
Jan 08 12:00:24 recovery.lan elemental[661]: : exit status 1)' stage name: Identify live mode
Jan 08 12:00:25 recovery.lan elemental[661]: ERRO[2024-01-08T12:00:25Z] no metadata/userdata found
Jan 08 12:00:25 recovery.lan elemental[661]: WARN[2024-01-08T12:00:25Z] (conditional) Skip 'Skipping stage (if statement error: failed to run [ -f "/run/elemental/live_mode" ]: exit status 1)' stage name: Include cloud init files from the ISO root
Jan 08 12:00:25 recovery.lan elemental[661]: INFO[2024-01-08T12:00:25Z] Done executing stage 'rootfs.before'
Jan 08 12:00:25 recovery.lan elemental[661]: INFO[2024-01-08T12:00:25Z] Running stage: rootfs
Jan 08 12:00:25 recovery.lan elemental[661]: INFO[2024-01-08T12:00:25Z] Processing stage step 'Layout configuration'. ( commands: 0, files: 1, ... )
Jan 08 12:00:25 recovery.lan elemental[661]: INFO[2024-01-08T12:00:25Z] Done executing stage 'rootfs'
Jan 08 12:00:25 recovery.lan elemental[661]: INFO[2024-01-08T12:00:25Z] Running stage: rootfs.after
Jan 08 12:00:25 recovery.lan elemental[661]: INFO[2024-01-08T12:00:25Z] Done executing stage 'rootfs.after'
Jan 08 12:00:25 recovery.lan elemental[661]: INFO[2024-01-08T12:00:25Z] Running stage: rootfs.before
Jan 08 12:00:25 recovery.lan elemental[661]: INFO[2024-01-08T12:00:25Z] Done executing stage 'rootfs.before'
Jan 08 12:00:25 recovery.lan elemental[661]: INFO[2024-01-08T12:00:25Z] Running stage: rootfs
Jan 08 12:00:25 recovery.lan elemental[661]: INFO[2024-01-08T12:00:25Z] Done executing stage 'rootfs'
Jan 08 12:00:25 recovery.lan elemental[661]: INFO[2024-01-08T12:00:25Z] Running stage: rootfs.after
Jan 08 12:00:25 recovery.lan elemental[661]: INFO[2024-01-08T12:00:25Z] Done executing stage 'rootfs.after'
Jan 08 12:00:25 recovery.lan elemental[661]: INFO[2024-01-08T12:00:25Z] Some errors found but were ignored. Enable --strict mode to fail on those or --debug to see them in the log
Jan 08 12:00:25 recovery.lan elemental[661]: WARN[2024-01-08T12:00:25Z] 1 error occurred:
Jan 08 12:00:25 recovery.lan elemental[661]:         * no metadata/userdata found
Jan 08 12:00:25 recovery.lan elemental[661]:  
Jan 08 12:00:25 recovery.lan systemd[1]: Finished Elemental system early rootfs setup.

frelon · 2024-05-22T10:49:06Z

Can you post result of elemental version as well? I think you might be missing this commit. The workaround would be to change the file to:

    - if: '[ ! -f "/run/elemental/recovery_mode" ]'
      name: "Layout configuration"
      files: 
      - path: /etc/elemental/config.d/layout.yaml
    .....

bobmorane83 · 2024-05-22T13:04:37Z

elemental:~ # elemental version
v2.1.0+ga1a22e8

Applying the workaround, can't see real difference and still the same pb with /etc/rancher

elemental:~ # mount
proc on /proc type proc (rw,nosuid,nodev,noexec,relatime)
sysfs on /sys type sysfs (rw,nosuid,nodev,noexec,relatime)
devtmpfs on /dev type devtmpfs (rw,nosuid,size=4096k,nr_inodes=1048576,mode=755,inode64)
securityfs on /sys/kernel/security type securityfs (rw,nosuid,nodev,noexec,relatime)
tmpfs on /dev/shm type tmpfs (rw,nosuid,nodev,inode64)
devpts on /dev/pts type devpts (rw,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000)
tmpfs on /run type tmpfs (rw,nosuid,nodev,size=385324k,nr_inodes=819200,mode=755,inode64)
tmpfs on /sys/fs/cgroup type tmpfs (ro,nosuid,nodev,noexec,size=4096k,nr_inodes=1024,mode=755,inode64)
cgroup2 on /sys/fs/cgroup/unified type cgroup2 (rw,nosuid,nodev,noexec,relatime,nsdelegate)
cgroup on /sys/fs/cgroup/systemd type cgroup (rw,nosuid,nodev,noexec,relatime,xattr,name=systemd)
pstore on /sys/fs/pstore type pstore (rw,nosuid,nodev,noexec,relatime)
efivarfs on /sys/firmware/efi/efivars type efivarfs (ro,nosuid,nodev,noexec,relatime)
bpf on /sys/fs/bpf type bpf (rw,nosuid,nodev,noexec,relatime,mode=700)
cgroup on /sys/fs/cgroup/cpuset type cgroup (rw,nosuid,nodev,noexec,relatime,cpuset)
cgroup on /sys/fs/cgroup/net_cls,net_prio type cgroup (rw,nosuid,nodev,noexec,relatime,net_cls,net_prio)
cgroup on /sys/fs/cgroup/misc type cgroup (rw,nosuid,nodev,noexec,relatime,misc)
cgroup on /sys/fs/cgroup/memory type cgroup (rw,nosuid,nodev,noexec,relatime,memory)
cgroup on /sys/fs/cgroup/perf_event type cgroup (rw,nosuid,nodev,noexec,relatime,perf_event)
cgroup on /sys/fs/cgroup/devices type cgroup (rw,nosuid,nodev,noexec,relatime,devices)
cgroup on /sys/fs/cgroup/hugetlb type cgroup (rw,nosuid,nodev,noexec,relatime,hugetlb)
cgroup on /sys/fs/cgroup/rdma type cgroup (rw,nosuid,nodev,noexec,relatime,rdma)
cgroup on /sys/fs/cgroup/pids type cgroup (rw,nosuid,nodev,noexec,relatime,pids)
cgroup on /sys/fs/cgroup/blkio type cgroup (rw,nosuid,nodev,noexec,relatime,blkio)
cgroup on /sys/fs/cgroup/cpu,cpuacct type cgroup (rw,nosuid,nodev,noexec,relatime,cpu,cpuacct)
cgroup on /sys/fs/cgroup/freezer type cgroup (rw,nosuid,nodev,noexec,relatime,freezer)
configfs on /sys/kernel/config type configfs (rw,nosuid,nodev,noexec,relatime)
/dev/mmcblk0p4 on /run/initramfs/elemental-state type ext4 (rw,relatime)
/run/initramfs/elemental-state/.snapshots/1/snapshot.img on / type ext2 (ro,relatime)
/dev/mmcblk0p2 on /oem type ext4 (rw,relatime)
/dev/mmcblk0p1 on /run/elemental/efi type vfat (ro,relatime,fmask=0022,dmask=0022,codepage=437,iocharset=iso8859-1,shortname=mixed,errors=remount-ro)
/dev/mmcblk0p5 on /run/elemental/persistent type ext4 (rw,relatime)
tmpfs on /run/elemental/overlay type tmpfs (rw,relatime,size=481656k,inode64)
overlay on /var type overlay (rw,relatime,lowerdir=/sysroot/var,upperdir=/run/elemental/overlay/var.overlay/upper,workdir=/run/elemental/overlay/var.overlay/work)
overlay on /etc type overlay (rw,relatime,lowerdir=/sysroot/etc,upperdir=/run/elemental/overlay/etc.overlay/upper,workdir=/run/elemental/overlay/etc.overlay/work)
overlay on /srv type overlay (rw,relatime,lowerdir=/sysroot/srv,upperdir=/run/elemental/overlay/srv.overlay/upper,workdir=/run/elemental/overlay/srv.overlay/work)
overlay on /home type overlay (rw,relatime,lowerdir=/sysroot/home,upperdir=/run/elemental/persistent/.state/home.overlay/upper,workdir=/run/elemental/persistent/.state/home.overlay/work)
overlay on /opt type overlay (rw,relatime,lowerdir=/sysroot/opt,upperdir=/run/elemental/persistent/.state/opt.overlay/upper,workdir=/run/elemental/persistent/.state/opt.overlay/work)
overlay on /root type overlay (rw,relatime,lowerdir=/sysroot/root,upperdir=/run/elemental/persistent/.state/root.overlay/upper,workdir=/run/elemental/persistent/.state/root.overlay/work)
overlay on /etc/systemd type overlay (rw,relatime,lowerdir=/sysroot/etc/systemd,upperdir=/run/elemental/persistent/.state/etc-systemd.overlay/upper,workdir=/run/elemental/persistent/.state/etc-systemd.overlay/work)
overlay on /etc/ssh type overlay (rw,relatime,lowerdir=/sysroot/etc/ssh,upperdir=/run/elemental/persistent/.state/etc-ssh.overlay/upper,workdir=/run/elemental/persistent/.state/etc-ssh.overlay/work)
overlay on /var/log type overlay (rw,relatime,lowerdir=/sysroot/var/log,upperdir=/run/elemental/persistent/.state/var-log.overlay/upper,workdir=/run/elemental/persistent/.state/var-log.overlay/work)
selinuxfs on /sys/fs/selinux type selinuxfs (rw,nosuid,noexec,relatime)
systemd-1 on /proc/sys/fs/binfmt_misc type autofs (rw,relatime,fd=30,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=19469)
hugetlbfs on /dev/hugepages type hugetlbfs (rw,relatime,pagesize=2M)
mqueue on /dev/mqueue type mqueue (rw,nosuid,nodev,noexec,relatime)
debugfs on /sys/kernel/debug type debugfs (rw,nosuid,nodev,noexec,relatime)
tracefs on /sys/kernel/tracing type tracefs (rw,nosuid,nodev,noexec,relatime)
tmpfs on /tmp type tmpfs (rw,nosuid,nodev,size=963308k,nr_inodes=1048576,inode64)
fusectl on /sys/fs/fuse/connections type fusectl (rw,nosuid,nodev,noexec,relatime)
none on /run/credentials/systemd-sysusers.service type ramfs (ro,nosuid,nodev,noexec,relatime,mode=700)
tmpfs on /run/user/0 type tmpfs (rw,nosuid,nodev,relatime,size=192660k,nr_inodes=48165,mode=700,inode64)

elemental:~ # journalctl -u elemental-setup-rootfs
an 08 12:00:24 recovery.lan systemd[1]: Starting Elemental system early rootfs setup...
Jan 08 12:00:24 recovery.lan elemental[658]: INFO[2024-01-08T12:00:24Z] Starting elemental version v2.1.0
Jan 08 12:00:24 recovery.lan elemental[658]: INFO[2024-01-08T12:00:24Z] Reading configuration from '/etc/elemental'
Jan 08 12:00:24 recovery.lan elemental[658]: INFO[2024-01-08T12:00:24Z] Running stage: rootfs.before
Jan 08 12:00:24 recovery.lan elemental[658]: INFO[2024-01-08T12:00:24Z] Processing stage step 'Pull data from provider'. ( commands: 0, files: 0, ... )
Jan 08 12:00:24 recovery.lan elemental[658]: WARN[2024-01-08T12:00:24Z] (conditional) Skip 'Skipping stage (if statement error: failed to run cat /proc/cmdline | grep -q "CDLABEL" || cat /proc/cmdline | grep -q "elemental.disable"  || cat /proc/cmdline | grep -q "rd.cos.disable"
Jan 08 12:00:24 recovery.lan elemental[658]: : exit status 1)' stage name: Identify live mode
Jan 08 12:00:25 recovery.lan elemental[658]: ERRO[2024-01-08T12:00:25Z] no metadata/userdata found
Jan 08 12:00:25 recovery.lan elemental[658]: WARN[2024-01-08T12:00:25Z] (conditional) Skip 'Skipping stage (if statement error: failed to run [ -f "/run/elemental/live_mode" ]: exit status 1)' stage name: Include cloud init files from the ISO root
Jan 08 12:00:25 recovery.lan elemental[658]: INFO[2024-01-08T12:00:25Z] Done executing stage 'rootfs.before'
Jan 08 12:00:25 recovery.lan elemental[658]: INFO[2024-01-08T12:00:25Z] Running stage: rootfs
Jan 08 12:00:25 recovery.lan elemental[658]: INFO[2024-01-08T12:00:25Z] Processing stage step 'Layout configuration'. ( commands: 0, files: 1, ... )
Jan 08 12:00:25 recovery.lan elemental[658]: INFO[2024-01-08T12:00:25Z] Done executing stage 'rootfs'
Jan 08 12:00:25 recovery.lan elemental[658]: INFO[2024-01-08T12:00:25Z] Running stage: rootfs.after
Jan 08 12:00:25 recovery.lan elemental[658]: INFO[2024-01-08T12:00:25Z] Done executing stage 'rootfs.after'
Jan 08 12:00:25 recovery.lan elemental[658]: INFO[2024-01-08T12:00:25Z] Running stage: rootfs.before
Jan 08 12:00:25 recovery.lan elemental[658]: INFO[2024-01-08T12:00:25Z] Done executing stage 'rootfs.before'
Jan 08 12:00:25 recovery.lan elemental[658]: INFO[2024-01-08T12:00:25Z] Running stage: rootfs
Jan 08 12:00:25 recovery.lan elemental[658]: INFO[2024-01-08T12:00:25Z] Done executing stage 'rootfs'
Jan 08 12:00:25 recovery.lan elemental[658]: INFO[2024-01-08T12:00:25Z] Running stage: rootfs.after
Jan 08 12:00:25 recovery.lan elemental[658]: INFO[2024-01-08T12:00:25Z] Done executing stage 'rootfs.after'
Jan 08 12:00:25 recovery.lan elemental[658]: INFO[2024-01-08T12:00:25Z] Some errors found but were ignored. Enable --strict mode to fail on those or --debug to see them in the log
Jan 08 12:00:25 recovery.lan elemental[658]: WARN[2024-01-08T12:00:25Z] 1 error occurred:
Jan 08 12:00:25 recovery.lan elemental[658]:         * no metadata/userdata found
Jan 08 12:00:25 recovery.lan elemental[658]:  
Jan 08 12:00:25 recovery.lan systemd[1]: Finished Elemental system early rootfs setup.

bobmorane83 · 2024-05-22T17:44:44Z

At end here is my working conf :

# Layout Elemental OEM configuration file
#
# This file is part of Elemental and will get reset during upgrades.
#
# Before you change this file manually,
# consider copying this file to /usr/local/cloud-config or
# copy the file with a prefix starting by 90, e.g. /oem/91_custom.yaml
name: "Rootfs Layout Settings"
stages:
  rootfs:
    - if: '[ ! -f "/run/elemental/recovery_mode" ]'
      name: "Layout configuration"
      files: 
      - path: /etc/elemental/config.d/layout.yaml
        content: |
          mount:
            ephemeral:
              type: tmpfs
              size: 25%

  initramfs:
    - if: '[ ! -f "/run/elemental/recovery_mode" ]'
      name: "Rancher Setting"
      commands:
      - |
        mkdir /oem/rancher
        ln -s /oem/rancher /etc

frelon · 2024-05-23T06:33:53Z

I will dust off my raspberry and try to debug this today! You might be hitting a bug with loading the config..

Also the /oem/rancher symlink was inventive! 👍

frelon · 2024-05-27T06:57:48Z

@bobmorane83 I was able to boot my RPi4 and get a persistent path for /etc/rancher using the following config:

# Rootfs Elemental OEM configuration file
#
# This file is part of Elemental and will get reset during upgrades.
#
# Before you change this file manually,
# consider copying this file to /usr/local/cloud-config or
# copy the file with a prefix starting by 90, e.g. /oem/91_custom.yaml
name: "Rootfs Layout Settings"
stages:
  rootfs.before:
    - name: "Pull data from provider"
      datasource:
        providers: ["aws", "gcp", "openstack", "cdrom"]
        path: "/oem"
  initramfs:
    - if: '[ ! -f "/run/elemental/recovery_mode" ]'
      name: "Persist /etc/machine-id"
      commands:
      - |
        # persist machine-id
        if [ -s /run/elemental/persistent/etc/machine-id ]; then
          cat /run/elemental/persistent/etc/machine-id > /etc/machine-id
        else
          mkdir -p /run/elemental/persistent/etc
          cp /etc/machine-id /run/elemental/persistent/etc
        fi
  rootfs:
    - if: '[ ! -f "/run/elemental/recovery_mode" ]'
      name: "Layout configuration"
      files: 
      - path: /run/elemental/config.d/layout.yaml
        content: |
          mount:
            write-fstab: true
            ephemeral:
              type: tmpfs
              size: 25%
              paths: ['/etc', '/var', '/srv']
            persistent:
              mode: bind
              paths:
              - /etc/systemd
              - /etc/rancher
              - /etc/ssh
              - /etc/iscsi
              - /etc/cni
              - /home
              - /opt
              - /root
              - /usr/libexec
              - /usr/local
              - /var/log
              - /var/lib/elemental
              - /var/lib/rancher
              - /var/lib/kubelet
              - /var/lib/NetworkManager
              - /var/lib/cni
              - /var/lib/calico
    - if: '[ -f "/run/elemental/recovery_mode" ]'
      name: "Layout configuration for recovery"
      files: 
      - path: /run/elemental/config.d/layout.yaml
        content: |
          mount:
            write-fstab: true
            ephemeral:
              type: tmpfs
              size: 25%

But used the tumbleweed changes as well.. Let me know if you are still having trouble!

bobmorane83 · 2024-05-29T07:03:20Z

I'll try that.
Thanks.

bobmorane83 mentioned this issue May 17, 2024

Allow to compile for RPi #2084

Merged

frelon added the kind/question Further information is requested label May 22, 2024

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

library/rpi #2083

library/rpi #2083

bobmorane83 commented May 16, 2024 •

edited

Loading

frelon commented May 16, 2024

bobmorane83 commented May 16, 2024

bobmorane83 commented May 16, 2024 •

edited

Loading

frelon commented May 17, 2024

bobmorane83 commented May 17, 2024 •

edited

Loading

bobmorane83 commented May 17, 2024

frelon commented May 17, 2024

bobmorane83 commented May 17, 2024 •

edited

Loading

frelon commented May 17, 2024

bobmorane83 commented May 21, 2024 •

edited

Loading

frelon commented May 21, 2024

bobmorane83 commented May 21, 2024 •

edited

Loading

frelon commented May 21, 2024

bobmorane83 commented May 21, 2024

frelon commented May 22, 2024

bobmorane83 commented May 22, 2024

frelon commented May 22, 2024

bobmorane83 commented May 22, 2024 •

edited

Loading

frelon commented May 22, 2024

bobmorane83 commented May 22, 2024 •

edited

Loading

frelon commented May 22, 2024

bobmorane83 commented May 22, 2024

bobmorane83 commented May 22, 2024 •

edited

Loading

frelon commented May 23, 2024

frelon commented May 27, 2024

bobmorane83 commented May 29, 2024

library/rpi #2083

library/rpi #2083

Comments

bobmorane83 commented May 16, 2024 • edited Loading

frelon commented May 16, 2024

bobmorane83 commented May 16, 2024

bobmorane83 commented May 16, 2024 • edited Loading

frelon commented May 17, 2024

bobmorane83 commented May 17, 2024 • edited Loading

bobmorane83 commented May 17, 2024

frelon commented May 17, 2024

bobmorane83 commented May 17, 2024 • edited Loading

frelon commented May 17, 2024

bobmorane83 commented May 21, 2024 • edited Loading

frelon commented May 21, 2024

bobmorane83 commented May 21, 2024 • edited Loading

frelon commented May 21, 2024

bobmorane83 commented May 21, 2024

frelon commented May 22, 2024

bobmorane83 commented May 22, 2024

frelon commented May 22, 2024

bobmorane83 commented May 22, 2024 • edited Loading

frelon commented May 22, 2024

bobmorane83 commented May 22, 2024 • edited Loading

frelon commented May 22, 2024

bobmorane83 commented May 22, 2024

bobmorane83 commented May 22, 2024 • edited Loading

frelon commented May 23, 2024

frelon commented May 27, 2024

bobmorane83 commented May 29, 2024

bobmorane83 commented May 16, 2024 •

edited

Loading

bobmorane83 commented May 16, 2024 •

edited

Loading

bobmorane83 commented May 17, 2024 •

edited

Loading

bobmorane83 commented May 17, 2024 •

edited

Loading

bobmorane83 commented May 21, 2024 •

edited

Loading

bobmorane83 commented May 21, 2024 •

edited

Loading

bobmorane83 commented May 22, 2024 •

edited

Loading

bobmorane83 commented May 22, 2024 •

edited

Loading

bobmorane83 commented May 22, 2024 •

edited

Loading