talos 1.8.2-alpha.2 imager fails to create image

[rgl]

Bug Report

Description

talos 1.8.2-alpha.2 imager fails to create image.

Logs

At https://github.com/rgl/terraform-libvirt-talos/tree/wip-upgrade-to-1.8 I'm upgrading from talos 1.7.6 to 1.8.0-alpha.2, but the image generation fails:

$ ./do init
### build talos image ###
+ local talos_version_tag=v1.8.0-alpha.2
+ rm -rf tmp/talos
+ mkdir -p tmp/talos
+ cat
+ local talos_libvirt_base_volume_name=talos-1.8.0-alpha.2.qcow2
+ docker run --rm -i -v /home/vagrant/Projects/terraform-libvirt-talos/tmp/talos:/secureboot:ro -v /home/vagrant/Projects/terraform-libvirt-talos/tmp/talos:/out -v /dev:/dev --privileged ghcr.io/siderolabs/imager:v1.8.0-alpha.2 -
assembling the finalized profile...
skipped pulling overlay (no overlay)
profile ready:
arch: amd64
platform: nocloud
secureboot: false
version: v1.8.0-alpha.2
customization:
  extraKernelArgs:
    - net.ifnames=0
input:
  kernel:
    path: /usr/install/amd64/vmlinuz
  initramfs:
    path: /usr/install/amd64/initramfs.xz
  baseInstaller:
    imageRef: ghcr.io/siderolabs/installer:v1.8.0-alpha.2
  systemExtensions:
    - imageRef: ghcr.io/siderolabs/qemu-guest-agent:9.0.2@sha256:f54ef0bf911e6ed0f01982f620ce6d32f6bc16a1220020345696a562d59346e2
    - imageRef: ghcr.io/siderolabs/drbd:9.2.11-v1.8.0-alpha.2@sha256:90b09ab323bbb0294ca6596826f22f59fa76d29f9dc9539187d0e1e2f3907f0b
    - imageRef: ghcr.io/siderolabs/spin:v0.15.1@sha256:87c9f352e5763440df7fc27f76ea4045ca6e049023496db01a2d17ac627e0e0e
output:
  kind: image
  imageOptions:
    diskSize: 2147483648
    diskFormat: raw
  outFormat: raw
rebuilding initramfs with system extensions...
    copying /usr/install/amd64/initramfs.xz to /tmp/imager1335750951/initramfs.xz
rebuilding initramfs with system extensions...
    pulling ghcr.io/siderolabs/qemu-guest-agent:9.0.2@sha256:f54ef0bf911e6ed0f01982f620ce6d32f6bc16a1220020345696a562d59346e2...
rebuilding initramfs with system extensions...
    pulling ghcr.io/siderolabs/drbd:9.2.11-v1.8.0-alpha.2@sha256:90b09ab323bbb0294ca6596826f22f59fa76d29f9dc9539187d0e1e2f3907f0b...
rebuilding initramfs with system extensions...
    pulling ghcr.io/siderolabs/spin:v0.15.1@sha256:87c9f352e5763440df7fc27f76ea4045ca6e049023496db01a2d17ac627e0e0e...
rebuilding initramfs with system extensions...
    discovered system extensions:
rebuilding initramfs with system extensions...
    NAME               VERSION                 AUTHOR
rebuilding initramfs with system extensions...
    qemu-guest-agent   9.0.2                   Markus Reiter
rebuilding initramfs with system extensions...
    drbd               9.2.11-v1.8.0-alpha.2   Alexander Prentki
rebuilding initramfs with system extensions...
    spin               v0.15.1                 Sven Pfennig
rebuilding initramfs with system extensions...
    validating system extensions
rebuilding initramfs with system extensions...
    preparing to run depmod to generate kernel modules dependency tree
rebuilding initramfs with system extensions...
    running depmod to generate kernel modules dependency tree
libkmod: kmod_config_parse: /tmp/ext-modules4228129897/modules/lib/modules/6.6.47-talos/modules.softdep line 3: ignoring bad line starting with 'ftdep'
rebuilding initramfs with system extensions...
    compressing system extensions
rebuilding initramfs with system extensions...
    creating system extensions initramfs archive and compressing it
initramfs ready
kernel command line: talos.platform=nocloud console=tty1 console=ttyS0 net.ifnames=0 net.ifnames=0 init_on_alloc=1 slab_nomerge pti=on consoleblank=0 nvme_core.io_timeout=4294967295 printk.devkmsg=on ima_template=ima-ng ima_appraise=fix ima_hash=sha512
creating disk image...
    creating raw disk of size 2.1 GB
creating disk image...
    attaching loopback device
creating disk image...
    attached loopback device: /dev/loop19
creating disk image...
    created EFI (C12A7328-F81F-11D2-BA4B-00A0C93EC93B) size 104857600 bytes
creating disk image...
    created BIOS (21686148-6449-6E6F-744E-656564454649) size 1048576 bytes
creating disk image...
    created BOOT (0FC63DAF-8483-4772-8E79-3D69D8477DE4) size 1048576000 bytes
creating disk image...
    created META (0FC63DAF-8483-4772-8E79-3D69D8477DE4) size 1048576 bytes
creating disk image...
    formatting the partition "/dev/loop19p1" as "vfat" with label "EFI"
creating disk image...
    formatting the partition "/dev/loop19p2" as "zeroes" with label "BIOS"
creating disk image...
    formatting the partition "/dev/loop19p3" as "xfs" with label "BOOT"
creating disk image...
    formatting the partition "/dev/loop19p4" as "zeroes" with label "META"
creating disk image...
    detaching loopback device /dev/loop19
failed to install: failed to install bootloader: error mounting partitions: error mounting /dev/loop19p3: 2 error(s) occurred:
    invalid argument
    timeout
Error: failed to install: failed to install bootloader: error mounting partitions: error mounting /dev/loop19p3: 2 error(s) occurred:
        invalid argument
        timeout

Here's the contents of imager configuration file, tmp/talos/talos-1.8.0-alpha.2.yml:

arch: amd64
platform: nocloud
secureboot: false
version: v1.8.0-alpha.2
customization:
  extraKernelArgs:
    - net.ifnames=0
input:
  kernel:
    path: /usr/install/amd64/vmlinuz
  initramfs:
    path: /usr/install/amd64/initramfs.xz
  baseInstaller:
    imageRef: ghcr.io/siderolabs/installer:v1.8.0-alpha.2
  systemExtensions:
    - imageRef: ghcr.io/siderolabs/qemu-guest-agent:9.0.2@sha256:f54ef0bf911e6ed0f01982f620ce6d32f6bc16a1220020345696a562d59346e2
    - imageRef: ghcr.io/siderolabs/drbd:9.2.11-v1.8.0-alpha.2@sha256:90b09ab323bbb0294ca6596826f22f59fa76d29f9dc9539187d0e1e2f3907f0b
    - imageRef: ghcr.io/siderolabs/spin:v0.15.1@sha256:87c9f352e5763440df7fc27f76ea4045ca6e049023496db01a2d17ac627e0e0e
output:
  kind: image
  imageOptions:
    diskSize: 2147483648
    diskFormat: raw
  outFormat: raw

Environment

• Talos version: 1.8.2-alpha.2
• Kubernetes version: n/a
• Platform: n/a

[smira]

not quite sure what might be the issue, probably your host Linux can't mount the filesystem, as the kernel is too old.

probably need to look into dmesg, but it's not a bug in the imager itself, it uses still host Linux kernel for mount operations.

one way around is not to build an image is to use an ISO to boot and an installer to install which will not require host Linux kernel mount operations.

[smira]

And certainly way easier is to use Image Factory.

[rgl]

Thanks for the tips!

I'm using Ubuntu 22.04, which uses Linux 5.15.x, and indeed there are a lot of these errors messages in the logs:

set 03 14:48:32 dm1.lan kernel: XFS (loop30p3): Superblock has unknown incompatible features (0x20) enabled.
set 03 14:48:32 dm1.lan kernel: XFS (loop30p3): Filesystem cannot be safely mounted by this kernel.
set 03 14:48:32 dm1.lan kernel: XFS (loop30p3): SB validate failed with error -22.

Which ISO are you recommending? Is that something that Talos provides? Or you mean any ISO or VM with a kernel that matches Talos?

[smira]

I mean just install Talos using an ISO vs. a disk image if you can't generate a disk image due to your host system.

I don't know why you are creating an image yourself vs. using the Image Factory. It looks like you're using official Talos imager and extensions, so you can just use Image Factory.

[rgl]

I prefer using local tools instead of depending on external services. But in this case, I can no longer use imager, at least, until I upgrade my host kernel. I'm now using the Image Factory service and it works as expected.

[kvaps]

I have the same error, in dmesg:

[  953.079320] XFS (loop3p3): Superblock has unknown incompatible features (0x20) enabled.
[  953.079326] XFS (loop3p3): Filesystem cannot be safely mounted by this kernel.
[  953.079331] XFS (loop3p3): SB validate failed with error -22.

building on ubuntu with kernel 5.15.0-124-generic,
I fixed it by updating to latest hwe kernel 6.8.0-47-generic.