Debian unable to create initrd due out-of-sync repository

[lunix33]

mkosi commit the issue has been seen with

main & v22

Used host distribution

Pop_OS! 22.04

Used target distribution

Debian Bookworm (12)

Linux kernel version used

6.8.0-76060800daily20240311-generic

CPU architectures issue was seen on

x86_64

Unexpected behaviour you saw

I've been trying to create an disk image of Debian Bookworm (12) with a UKI. I can get though the inital rootfs configuration, but by the end of the process when mkosi tries to create the default initramfs it would crash while trying to install the distribution. Here's the flow I can see: (rootfs)build_image > install_kernel > install_uki > finalize_initrds > build_default_initrd > (initrd)build_image > install_distribution

on v22 the process crash with the following python error, but doesn't seem to be the actual root cause:

Reading package lists... Done
Building dependency tree... Done
Reading state information... Done
0 upgraded, 0 newly installed, 0 to remove and 0 not upgraded.
‣  (Fixing ownership of package manager cache directory)
Traceback (most recent call last):
  File "/home/user/projects/image/mkosi/mkosi/run.py", line 55, in uncaught_exception_handler
    yield
  File "/home/user/projects/image/mkosi/mkosi/run.py", line 96, in fork_and_wait
    target(*args, **kwargs)
  File "/home/user/projects/image/mkosi/mkosi/__init__.py", line 4082, in run_build
    build_image(Context(args, config, workspace=workspace, resources=resources))
  File "/home/user/projects/image/mkosi/mkosi/__init__.py", line 3438, in build_image
    install_kernel(context, partitions)
  File "/home/user/projects/image/mkosi/mkosi/__init__.py", line 2247, in install_kernel
    install_uki(context, kver, kimg, token, partitions)
  File "/home/user/projects/image/mkosi/mkosi/__init__.py", line 2179, in install_uki
    initrds += context.config.initrds or [build_default_initrd(context)]
  File "/home/user/projects/image/mkosi/mkosi/__init__.py", line 1687, in build_default_initrd
    build_image(
  File "/home/user/projects/image/mkosi/mkosi/__init__.py", line 3383, in build_image
    install_distribution(context)
  File "/home/user/projects/image/mkosi/mkosi/__init__.py", line 144, in install_distribution
    (context.root / "etc/machine-id").write_text("uninitialized\n")
  File "/usr/lib/python3.10/pathlib.py", line 1154, in write_text
    with self.open(mode='w', encoding=encoding, errors=errors, newline=newline) as f:
  File "/usr/lib/python3.10/pathlib.py", line 1119, in open
    return self._accessor.open(self, mode, buffering, encoding, errors,
FileNotFoundError: [Errno 2] No such file or directory: '/var/tmp/mkosi-workspacepu2u2d_6/root/etc/machine-id'

The build on main crash with the following error from APT, which illustrate the cause of the issue:

E: Unable to locate package base-files

From what I can see, on v22, it fails to create the machine-id because the etc directory dosen't exists in the first place and it seems to be because the call context.config.distribution.install(context) (which from my understanding would be to distributions/debian.py:install) essentially does nothing. When I tried to print the value of essential (in the install function) it just returns an empty list. On main we can clearly see the package database dosen't contain base-files, because the package list hasn't been synced.

When the build fail on main, I could get into a debug shell using the argument, and I was able to validate the package database never got synced. I was able to finish the build by adding Apt.sync(context) before the call to Apt.invoke in the install function for Debian.

I'm not sure if I'm mishandling the cache or doing something improperly. I've been using mkosi-chroot apt-get ... to install my packages in the chroot instead of using the helper in /scripts, since I need my custom repositories and keys (which gets added and dearmored during the prepare (final) script) to be available in the final image, and from what I understand putting them in /etc/apt/... (not within the chroot), which the script seems to be using, would not make them available in the final image. Should I create the files in both? Have I missed somthing in the documentation when it comes to handling cache and package install?

Used mkosi config

[Distribution]
Distribution = debian
Release = bookworm
Architecture = x86-64
Repositories = main,non-free,non-free-firmware,contrib

[Output]
Format = disk
OutputDirectory = dist
SplitArtifacts = true
CacheDirectory = .cache/img
PackageCacheDirectory = .cache/pkg
BuildDirectory = .cache/build

[Validation]
SecureBoot = true

[Host]
@Incremental = true
ToolsTree = default
# Default to testing, but the repo is missing some of the required packages
ToolsTreeRelease = sid

[Content]
Packages =
    systemd,systemd-sysv,systemd-boot,systemd-resolved,systemd-timesyncd
    gnupg,apt-transport-https,ca-certificates,wget
Bootable = true
Bootloader = uki

mkosi output

No response

[DaanDeMeyer]

@lunix33 The provided config does not reproduce the issue.

I'm not sure if I'm mishandling the cache or doing something improperly. I've been using mkosi-chroot apt-get ... to install my packages in the chroot instead of using the helper in /scripts, since I need my custom repositories and keys (which gets added and dearmored during the prepare (final) script) to be available in the final image, and from what I understand putting them in /etc/apt/... (not within the chroot), which the script seems to be using, would not make them available in the final image. Should I create the files in both? Have I missed somthing in the documentation when it comes to handling cache and package install?

You need to add put repositories in a mkosi.skeleton/ tree. Specifically in mkosi.skeleton/etc/apt/sources.list.d. Then they will be taken into account by mkosi and will be available in the final image.

I'm guessing that whatever you're doing in your scripts is getting rid of the repository metadata which we need to build the initrd later on.

[lunix33]

Thanks for the information you gave me, I'll take your recommandation into consideration.

When it comes to the actual issue at hand, after some more test I was able to get things working as expected after reading about that the script actions could affect the initrd generation. I started the project with a prototype on v21, and seems like some of the operations that were working in v21 stopped working with v22 (like deleting the package list within the chroot, though some of the configuration changed since then, and scripts were rewritten, so that might also be it). So removing those operations seem to fix my issue, when it comes to building the default initramfs. I've added a RemoveFiles configuration to delete the content of /var/lib/apt/lists within the final rootfs.