Creating an ova image of talos 1.8.3 from a OnPrem Image-factory (version 0.5.0) - crashing

[flpajany]

Bug Report

Description

I have installed a OnPrem Image Factory for Talos and I am using a custom build of talos imager (v1.8.3).

The factory is working just fine with my imagers of talos v1.6.8 and v1.7.7 when I am asking to generate a ova image.

But doing the same for talos 1.8.3, it crashes (logs below).

Could you help me please ?

Logs

{"level":"info","ts":1733840025.344509,"caller":"artifacts/fetch.go:233","msg":"extracted the image","image":"<myregistry>/siderolabs/imager@sha256:e8c8db5b1545039000e49a272107149b98dc34679cc0190044
skipped pulling overlay (no overlay)
profile ready:
arch: amd64
platform: vmware
secureboot: null
version: v1.8.3
customization:
  extraKernelArgs:
    - siderolink.api=https://omni.<myregistry>:8090/?jointoken=6XOe1fGI5KWpfideEDGDZohLKNeagZ4fJyKemg6EToe7
    - talos.events.sink=[fdae:41e4:649b:9303::1]:8091
    - talos.logging.kernel=tcp://[fdae:41e4:649b:9303::1]:8092
input:
  kernel:
    path: /tmp/image-factory3855222849/v1.8.3/amd64/vmlinuz
  initramfs:
    path: /tmp/image-factory3855222849/v1.8.3/amd64/initramfs.xz
  baseInstaller:
    imageRef: ghcr.io/siderolabs/installer:v1.8.3
  systemExtensions:
    - imageRef: ""
      tarballPath: /tmp/image-factory3855222849/schematics/45c3c22cbecec813e866e5a4bc7b9d068b6e103c05197d2b4599234e181c6150-true.tar
output:
  kind: image
  imageOptions:
    diskSize: 8589934592
    diskFormat: ova
  outFormat: raw
rebuilding initramfs with system extensions...
    copying /tmp/image-factory3855222849/v1.8.3/amd64/initramfs.xz to /tmp/imager2335137158/initramfs.xz
rebuilding initramfs with system extensions...
    extracting /tmp/image-factory3855222849/schematics/45c3c22cbecec813e866e5a4bc7b9d068b6e103c05197d2b4599234e181c6150-true.tar...
rebuilding initramfs with system extensions...
    discovered system extensions:
rebuilding initramfs with system extensions...
    NAME        VERSION                                                            AUTHOR
rebuilding initramfs with system extensions...
    schematic   45c3c22cbecec813e866e5a4bc7b9d068b6e103c05197d2b4599234e181c6150   Image Factory
rebuilding initramfs with system extensions...
    validating system extensions
rebuilding initramfs with system extensions...
    compressing system extensions
rebuilding initramfs with system extensions...
    creating system extensions initramfs archive and compressing it
initramfs ready
kernel command line: talos.platform=vmware talos.config=guestinfo console=tty0 console=ttyS0 earlyprintk=ttyS0,115200 net.ifnames=0 init_on_alloc=1 slab_nomerge pti=on consoleblank=0 nvme_core.io_timeout=429
creating disk image...
    creating raw disk of size 8.6 GB
creating disk image...
    attaching loopback device
creating disk image...
    attached loopback device: /dev/loop0
creating disk image...
    created EFI (C12A7328-F81F-11D2-BA4B-00A0C93EC93B) size 104857600 bytes
creating disk image...
    created BIOS (21686148-6449-6E6F-744E-656564454649) size 1048576 bytes
creating disk image...
    created BOOT (0FC63DAF-8483-4772-8E79-3D69D8477DE4) size 1048576000 bytes
creating disk image...
    created META (0FC63DAF-8483-4772-8E79-3D69D8477DE4) size 1048576 bytes
creating disk image...
    detaching loopback device /dev/loop0
panic: runtime error: invalid memory address or nil pointer dereference

        runtime/debug.Stack()
                /usr/local/go/src/runtime/debug/stack.go:26 +0x5e
        golang.org/x/sync/singleflight.newPanicError({0x2549a80, 0x4649dd0})
                /go/pkg/mod/golang.org/x/sync@v0.8.0/singleflight/singleflight.go:44 +0x25
        golang.org/x/sync/singleflight.(*Group).doCall.func2.1()
                /go/pkg/mod/golang.org/x/sync@v0.8.0/singleflight/singleflight.go:193 +0x34
        panic({0x2549a80?, 0x4649dd0?})
                /usr/local/go/src/runtime/panic.go:785 +0x132
        github.com/siderolabs/go-blockdevice/v2/partitioning/gpt.(*Table).syncKernel(0xc0007470e0)
                /go/pkg/mod/github.com/siderolabs/go-blockdevice/v2@v2.0.1/partitioning/gpt/gpt.go:638 +0x1bc
        github.com/siderolabs/go-blockdevice/v2/partitioning/gpt.(*Table).Write(0xc0007470e0)
                /go/pkg/mod/github.com/siderolabs/go-blockdevice/v2@v2.0.1/partitioning/gpt/gpt.go:549 +0xaff
        github.com/siderolabs/talos/cmd/installer/pkg/install.(*Installer).createPartitions(0xc0008753e0, {0x2e51a20, 0xc00119ef30}, 0x2, 0x0, {0x2e33070, 0xc000658720})
                /go/pkg/mod/github.com/siderolabs/talos@v1.8.0-alpha.2/cmd/installer/pkg/install/install.go:516 +0xe56
        github.com/siderolabs/talos/cmd/installer/pkg/install.(*Installer).Install(0xc0008753e0, {0x2e3e190, 0xc000274d20}, 0x2)
                /go/pkg/mod/github.com/siderolabs/talos@v1.8.0-alpha.2/cmd/installer/pkg/install/install.go:315 +0x7a5
        github.com/siderolabs/talos/pkg/imager.(*Imager).buildImage(0xc0002de008, {0x2e3e190, 0xc000274d20}, {0xc000fb5080, 0x2b}, 0xc000c2bb90)
                /go/pkg/mod/github.com/siderolabs/talos@v1.8.0-alpha.2/pkg/imager/out.go:308 +0x9b6
        github.com/siderolabs/talos/pkg/imager.(*Imager).outImage(0xc0002de008, {0x2e3e190, 0xc000274d20}, {0xc000fb5080, 0x2b}, 0xc000c2aae0)
                /go/pkg/mod/github.com/siderolabs/talos@v1.8.0-alpha.2/pkg/imager/out.go:194 +0xd5
        github.com/siderolabs/talos/pkg/imager.(*Imager).Execute(0xc0002de008, {0x2e3e190, 0xc000274d20}, {0xc000fb6040, 0x1a}, 0xc000c2aae0)
                /go/pkg/mod/github.com/siderolabs/talos@v1.8.0-alpha.2/pkg/imager/imager.go:128 +0x48b
        github.com/siderolabs/image-factory/internal/asset.(*Builder).build(_, {_, _}, {{0x0, 0x0}, {0xc0004f77fa, 0x5}, {0xc0004f77f3, 0x6}, {0x0, ...}, ...}, ...)
                /src/internal/asset/asset.go:277 +0xe65
        github.com/siderolabs/image-factory/internal/asset.(*Builder).buildAndCache(_, {_, _}, {{0x0, 0x0}, {0xc0004f77fa, 0x5}, {0xc0004f77f3, 0x6}, {0x0, ...}, ...}, ...)
                /src/internal/asset/asset.go:199 +0xf6
        github.com/siderolabs/image-factory/internal/asset.(*Builder).Build.func1()
                /src/internal/asset/asset.go:170 +0x5e
        golang.org/x/sync/singleflight.(*Group).doCall.func2(0xc000aa9f56, 0xc00055eff0, 0xc000a03f70?)
                /go/pkg/mod/golang.org/x/sync@v0.8.0/singleflight/singleflight.go:198 +0x55
        golang.org/x/sync/singleflight.(*Group).doCall(0xc0001f0300?, 0xc000a03f98?, {0xc000b67280?, 0x71efe5?}, 0xc0001f0300?)
                /go/pkg/mod/golang.org/x/sync@v0.8.0/singleflight/singleflight.go:200 +0x7e
        created by golang.org/x/sync/singleflight.(*Group).DoChan in goroutine 70
                /go/pkg/mod/golang.org/x/sync@v0.8.0/singleflight/singleflight.go:138 +0x2d2


goroutine 92 [running]:
golang.org/x/sync/singleflight.(*Group).doCall.func1.gowrap2()
        /go/pkg/mod/golang.org/x/sync@v0.8.0/singleflight/singleflight.go:167 +0x25
created by golang.org/x/sync/singleflight.(*Group).doCall.func1 in goroutine 77
        /go/pkg/mod/golang.org/x/sync@v0.8.0/singleflight/singleflight.go:167 +0x292

When I am trying to make an ova image directly from the repository talos :

# make PLATFORM=linux/amd64 IMAGE_REGISTRY=<myregistry> image-vmware
v1.8.3: Pulling from siderolabs/imager
Digest: sha256:e8c8db5b1545039000e49a272107149b98dc34679cc0190044dc4a5009100fd6
Status: Image is up to date for <myregistry>/siderolabs/imager:v1.8.3
<myregistry>/siderolabs/imager:v1.8.3
skipped pulling overlay (no overlay)
profile ready:
arch: amd64
platform: vmware
secureboot: false
version: v1.8.3
input:
  kernel:
    path: /usr/install/amd64/vmlinuz
  initramfs:
    path: /usr/install/amd64/initramfs.xz
  baseInstaller:
    imageRef: ghcr.io/siderolabs/installer:v1.8.3
output:
  kind: image
  imageOptions:
    diskSize: 8589934592
    diskFormat: ova
  outFormat: raw
skipped initramfs rebuild (no system extensions)
kernel command line: talos.platform=vmware talos.config=guestinfo console=tty0 console=ttyS0 earlyprintk=ttyS0,115200 net.ifnames=0 init_on_alloc=1 slab_nomerge pti=on consoleblank=0 nvme_core.io_timeout=4294967295 printk.devkmsg=on ima_template=ima-ng ima_appraise=fix ima_hash=sha512
◳ failed to install: failed to install bootloader: error mounting partitions: error mounting /dev/loop0p3: 2 error(s) occurred:
    invalid argument
    timeout
Error: failed to install: failed to install bootloader: error mounting partitions: error mounting /dev/loop0p3: 2 error(s) occurred:
        invalid argument
        timeout
make: *** [Makefile:393: image-vmware] Error 1

Environment

• Talos version: 1.8.3
• Kubernetes version: Unrevelant
• Platform: amd64 (but there could be a problem with arm64)

[smira]

The first error was already fixed a long time ago, please latest main version when reporting issues.

The second error is most probably your host Linux kernel is too old to support xfs features Talos requires, you need to use more up to date Linux.

[flpajany]

The first error was already fixed a long time ago, please latest main version when reporting issues.

The second error is most probably your host Linux kernel is too old to support xfs features Talos requires, you need to use more up to date Linux.

For the first error, which release do I have to use for talos 1.8 ?

[smira]

For the first error, which release do I have to use for talos 1.8 ?

Always use the latest Image Factory release. There is no point in using old ones.

[flpajany]

For the first error, which release do I have to use for talos 1.8 ?

Always use the latest Image Factory release. There is no point in using old ones.

OK thx. I'll try it.

[flpajany]

Tried with version 0.6.2 of the factory and now I have another error (could be related about what you said about xfs though) :

profile ready:
arch: amd64
platform: vmware
secureboot: null
version: v1.8.3
customization:
  extraKernelArgs:
    - siderolink.api=https://omni-test.<mydomain>:8090/?jointoken=ezKjOSkLYjABPw8OaEYBLIPlDQOqz9mFVdODQZDBQBY
    - talos.events.sink=[fdae:41e4:649b:9303::1]:8091
    - talos.logging.kernel=tcp://[fdae:41e4:649b:9303::1]:8092
input:
  kernel:
    path: /tmp/image-factory3201863390/v1.8.3/amd64/vmlinuz
  initramfs:
    path: /tmp/image-factory3201863390/v1.8.3/amd64/initramfs.xz
  baseInstaller:
    imageRef: ghcr.io/siderolabs/installer:v1.8.3
  systemExtensions:
    - imageRef: ""
      tarballPath: /tmp/image-factory3201863390/schematics/5a8af9329409e7fa5ba6f333fe0a642cc1d4c443ced047ee48e694bf380401c5-true.tar
output:
  kind: image
  imageOptions:
    diskSize: 8589934592
    diskFormat: ova
  outFormat: raw
rebuilding initramfs with system extensions...
    copying /tmp/image-factory3201863390/v1.8.3/amd64/initramfs.xz to /tmp/imager2912516937/initramfs.xz
rebuilding initramfs with system extensions...
    extracting /tmp/image-factory3201863390/schematics/5a8af9329409e7fa5ba6f333fe0a642cc1d4c443ced047ee48e694bf380401c5-true.tar...
rebuilding initramfs with system extensions...
    discovered system extensions:
rebuilding initramfs with system extensions...
    NAME        VERSION                                                            AUTHOR
rebuilding initramfs with system extensions...
    schematic   5a8af9329409e7fa5ba6f333fe0a642cc1d4c443ced047ee48e694bf380401c5   Image Factory
rebuilding initramfs with system extensions...
    validating system extensions
rebuilding initramfs with system extensions...
    compressing system extensions
rebuilding initramfs with system extensions...
    creating system extensions initramfs archive and compressing it
initramfs ready
kernel command line: talos.platform=vmware talos.config=guestinfo console=tty0 console=ttyS0 earlyprintk=ttyS0,115200 net.ifnames=0 init_on_alloc=1 slab_nomerge pti=on consoleblank=0 nvme_core.io_timeout=4294967295 printk.devkmsg=on ima_template=ima-ng ima_appraise=fix ima_hash=sha512 siderolink.api=https://omni-test.<mydomain>:8090/?jointoken=ezKjOSkLYjABPw8OaEYBLIPlDQOqz9mFVdODQZDBQBY talos.events.sink=[fdae:41e4:649b:9303::1]:8091 talos.logging.kernel=tcp://[fdae:41e4:649b:9303::1]:8092
creating disk image...
    creating raw disk of size 8.6 GB
creating disk image...
    attaching loopback device
creating disk image...
    attached loopback device: /dev/loop0
creating disk image...
    created EFI (C12A7328-F81F-11D2-BA4B-00A0C93EC93B) size 104857600 bytes
creating disk image...
    created BIOS (21686148-6449-6E6F-744E-656564454649) size 1048576 bytes
creating disk image...
    created BOOT (0FC63DAF-8483-4772-8E79-3D69D8477DE4) size 1048576000 bytes
creating disk image...
    created META (0FC63DAF-8483-4772-8E79-3D69D8477DE4) size 1048576 bytes
creating disk image...
    formatting the partition "/dev/loop0p1" as "vfat" with label "EFI"
creating disk image...
    formatting the partition "/dev/loop0p2" as "zeroes" with label "BIOS"
creating disk image...
    formatting the partition "/dev/loop0p3" as "xfs" with label "BOOT"
creating disk image...
    detaching loopback device /dev/loop0
{"level":"info","ts":1733857311.896146,"caller":"http/http.go:164","msg":"request","frontend":"http","method":"GET","path":"/image/5a8af9329409e7fa5ba6f333fe0a642cc1d4c443ced047ee48e694bf380401c5/v1.8.3/vmware-amd64.ova","error":"error generating asset: failed to install: failed to create partitions: failed to format partition /dev/loop0p3: exit status 1: size 256 of data subvolume is too small, minimum 4096 blocks\nUsage: mkfs.xfs\n/* blocksize */\t\t[-b size=num]\n/* config file */\t[-c options=xxx]\n/* metadata */\t\t[-m crc=0|1,finobt=0|1,uuid=xxx,rmapbt=0|1,reflink=0|1,\n\t\t\t    inobtcount=0|1,bigtime=0|1]\n/* data subvol */\t[-d agcount=n,agsize=n,file,name=xxx,size=num,\n\t\t\t    (sunit=value,swidth=value|su=num,sw=num|noalign),\n\t\t\t    sectsize=num,concurrency=num]\n/* force overwrite */\t[-f]\n/* inode size */\t[-i perblock=n|size=num,maxpct=n,attr=0|1|2,\n\t\t\t    projid32bit=0|1,sparse=0|1,nrext64=0|1]\n/* no discard */\t[-K]\n/* log subvol */\t[-l agnum=n,internal,size=num,logdev=xxx,version=n\n\t\t\t    sunit=value|su=num,sectsize=num,lazy-count=0|1,\n\t\t\t    concurrency=num]\n/* label */\t\t[-L label (maximum 12 characters)]\n/* naming */\t\t[-n size=num,version=2|ci,ftype=0|1]\n/* no-op info only */\t[-N]\n/* prototype file */\t[-p fname]\n/* quiet */\t\t[-q]\n/* realtime subvol */\t[-r extsize=num,size=num,rtdev=xxx]\n/* sectorsize */\t[-s size=num]\n/* version */\t\t[-V]\n\t\t\tdevicename\n<devicename> is required unless -d name=xxx is given.\n<num> is xxx (bytes), xxxs (sectors), xxxb (fs blocks), xxxk (xxx KiB),\n      xxxm (xxx MiB), xxxg (xxx GiB), xxxt (xxx TiB) or xxxp (xxx PiB).\n<value> is xxx (512 byte blocks).\n"}

And now I am unable to build a ova image for talos 1.7.7 too :-(

[flpajany]

In fact now I have the same error as when I tried to build an ova image directly from talos repo (my message above was from an error to accessing to /dev/loop from a container I resolved using -v /dev:/dev) :

profile ready:
arch: amd64
platform: vmware
secureboot: null
version: v1.8.3
customization:
  extraKernelArgs:
    - siderolink.api=https://omni-test.<mydomain>:8090/?jointoken=ezKjOSkLYjABPw8OaEYBLIPlDQOqz9mFVdODQZDBQBY
    - talos.events.sink=[fdae:41e4:649b:9303::1]:8091
    - talos.logging.kernel=tcp://[fdae:41e4:649b:9303::1]:8092
input:
  kernel:
    path: /tmp/image-factory594389154/v1.8.3/amd64/vmlinuz
  initramfs:
    path: /tmp/image-factory594389154/v1.8.3/amd64/initramfs.xz
  baseInstaller:
    imageRef: ghcr.io/siderolabs/installer:v1.8.3
  systemExtensions:
    - imageRef: ""
      tarballPath: /tmp/image-factory594389154/schematics/5a8af9329409e7fa5ba6f333fe0a642cc1d4c443ced047ee48e694bf380401c5-true.tar
output:
  kind: image
  imageOptions:
    diskSize: 8589934592
    diskFormat: ova
  outFormat: raw
rebuilding initramfs with system extensions...
    copying /tmp/image-factory594389154/v1.8.3/amd64/initramfs.xz to /tmp/imager660881459/initramfs.xz
rebuilding initramfs with system extensions...
    extracting /tmp/image-factory594389154/schematics/5a8af9329409e7fa5ba6f333fe0a642cc1d4c443ced047ee48e694bf380401c5-true.tar...
rebuilding initramfs with system extensions...
    discovered system extensions:
rebuilding initramfs with system extensions...
    NAME        VERSION                                                            AUTHOR
rebuilding initramfs with system extensions...
    schematic   5a8af9329409e7fa5ba6f333fe0a642cc1d4c443ced047ee48e694bf380401c5   Image Factory
rebuilding initramfs with system extensions...
    validating system extensions
rebuilding initramfs with system extensions...
    compressing system extensions
rebuilding initramfs with system extensions...
    creating system extensions initramfs archive and compressing it
initramfs ready
kernel command line: talos.platform=vmware talos.config=guestinfo console=tty0 console=ttyS0 earlyprintk=ttyS0,115200 net.ifnames=0 init_on_alloc=1 slab_nomerge pti=on consoleblank=0 nvme_core.io_timeout=4294967295 printk.devkmsg=on ima_template=ima-ng ima_appraise=fix ima_hash=sha512 siderolink.api=https://omni-test.<mydomain>:8090/?jointoken=ezKjOSkLYjABPw8OaEYBLIPlDQOqz9mFVdODQZDBQBY talos.events.sink=[fdae:41e4:649b:9303::1]:8091 talos.logging.kernel=tcp://[fdae:41e4:649b:9303::1]:8092
creating disk image...
    creating raw disk of size 8.6 GB
creating disk image...
    attaching loopback device
creating disk image...
    attached loopback device: /dev/loop0
creating disk image...
    created EFI (C12A7328-F81F-11D2-BA4B-00A0C93EC93B) size 104857600 bytes
creating disk image...
    created BIOS (21686148-6449-6E6F-744E-656564454649) size 1048576 bytes
creating disk image...
    created BOOT (0FC63DAF-8483-4772-8E79-3D69D8477DE4) size 1048576000 bytes
creating disk image...
    created META (0FC63DAF-8483-4772-8E79-3D69D8477DE4) size 1048576 bytes
creating disk image...
    formatting the partition "/dev/loop0p1" as "vfat" with label "EFI"
creating disk image...
    formatting the partition "/dev/loop0p2" as "zeroes" with label "BIOS"
creating disk image...
    formatting the partition "/dev/loop0p3" as "xfs" with label "BOOT"
creating disk image...
    formatting the partition "/dev/loop0p4" as "zeroes" with label "META"
creating disk image...
    detaching loopback device /dev/loop0
{"level":"info","ts":1733904227.547331,"caller":"http/http.go:164","msg":"request","frontend":"http","method":"GET","path":"/image/5a8af9329409e7fa5ba6f333fe0a642cc1d4c443ced047ee48e694bf380401c5/v1.8.3/vmware-amd64.ova","error":"error generating asset: failed to install: failed to install bootloader: error mounting partitions: error mounting /dev/loop0p3: 2 error(s) occurred:\n\tinvalid argument\n\ttimeout"}

But like above, I am unable to build a ova image for talos 1.7.7 (or even 1.6.8 images as a matter of fact) too :-(

Could you tell me what are OS/System prerequisites I need for my host (or from docker) to build my ova images, please ? (what kind of OS/System are you using for the official image factory for example ?)

[smira]

We should probably eventually look into the xfs filesystem creation options, but for now the only way is to use host Linux system which is more up to date. I guess you're running like 5.x Linux kernel.

[flpajany]

# cat /etc/os-release
NAME="AlmaLinux"
VERSION="8.8 (Sapphire Caracal)"
ID="almalinux"
ID_LIKE="rhel centos fedora"
VERSION_ID="8.8"
PLATFORM_ID="platform:el8"
PRETTY_NAME="AlmaLinux 8.8 (Sapphire Caracal)"
ANSI_COLOR="0;34"
LOGO="fedora-logo-icon"
CPE_NAME="cpe:/o:almalinux:almalinux:8::baseos"
HOME_URL="https://almalinux.org/"
DOCUMENTATION_URL="https://wiki.almalinux.org/"
BUG_REPORT_URL="https://bugs.almalinux.org/"

ALMALINUX_MANTISBT_PROJECT="AlmaLinux-8"
ALMALINUX_MANTISBT_PROJECT_VERSION="8.8"
REDHAT_SUPPORT_PRODUCT="AlmaLinux"
REDHAT_SUPPORT_PRODUCT_VERSION="8.8"

# uname -r
4.18.0-477.27.2.el8_8.x86_64

[smira]

That's not a kernel version. Kernel version is in the uname -a output.

[flpajany]

# uname -a
Linux <myhost> 4.18.0-477.27.2.el8_8.x86_64 #1 SMP Fri Sep 29 08:21:01 EDT 2023 x86_64 x86_64 x86_64 GNU/Linux

[smira]

This kernel was released back in 2018, 6 years ago. Certainly we won't support anything below the Linux 5.15.

[flpajany]

This kernel was released back in 2018, 6 years ago. Certainly we won't support anything below the Linux 5.15.

OK we'll try to upgrade our kernel.

[flpajany]

Indeed :

# uname -r
6.12.4-1.el8.elrepo.x86_64

And now it is working ! Thx a lot :-)

[smira]

Moved to #9955