Use of/alignment with Anaconda

[Promaethius]

Add an option to the build command to output an initramfs.img and coreos.img to serve over PXE. This would support the https://github.com/coreos/matchbox project which would lead to tectonic integration. I'd be more than willing to contribute to this.

EDIT: I see a lot of parallels with https://github.com/weldr/lorax/blob/master/docs/livemedia-creator.rst

[cgwalters]

Yes. However, one important architectural point here is that currently lmc/lorax create loopback devices and require SELinux permissive or disabled. If we were to integrate them here, I would prefer we inject them into transient "utility" VMs and extract the output from that.

This way we don't affect the host system - we won't leak loopback devices on failure, we don't require the host to turn off SELinux. And more generally, we can "scale out" builds using e.g. Kubernetes as long as we have virt capabilities.

[cgwalters]

Looking quickly a bit more at LMC, I had forgotten it supports two modes, including one that does the heavy lifting in qemu.

[Promaethius]

Looking quickly a bit more at LMC, I had forgotten it supports two modes, including one that does the heavy lifting in qemu.

Yes that's the impression I got as well. In addition, it creates the squashfs and PXE scripts. It seems to be a nicely rounded tool. Doesn't it also see use in Pungi? As long as it still sees use in upstream fedora could it be useful here?

Reading through creator.py it includes some validation that might be useful:
Kickstart: L595-L638 https://github.com/weldr/lorax/blob/master/src/pylorax/creator.py#L595

Disk Image: https://github.com/weldr/lorax/blob/master/src/pylorax/creator.py#L642 -> https://github.com/weldr/lorax/blob/master/src/pylorax/creator.py#L459 (Where @cgwalters was indicating the dual modes https://github.com/weldr/lorax/blob/master/src/pylorax/creator.py#L478) into virt_install https://github.com/weldr/lorax/blob/master/src/pylorax/creator.py#L484 -> https://github.com/weldr/lorax/blob/master/src/pylorax/installer.py#L486 into mkfsimage_from_disk https://github.com/weldr/lorax/blob/master/src/pylorax/installer.py#L550 -> https://github.com/weldr/lorax/blob/master/src/pylorax/imgutils.py#L508 (function completes and branches back up to creator.py) https://github.com/weldr/lorax/blob/master/src/pylorax/creator.py#L698 into make_live_images https://github.com/weldr/lorax/blob/master/src/pylorax/creator.py#L704 -> https://github.com/weldr/lorax/blob/master/src/pylorax/creator.py#L496 (includes some disk logic between L518-L547) build initramfs rebuild_initrds_for_live https://github.com/weldr/lorax/blob/master/src/pylorax/creator.py#L566 -> https://github.com/weldr/lorax/blob/master/src/pylorax/creator.py#L210 (looks like the dracut build doesn't take params from the rpm-ostree definition, this makes sense given that lorax is ostree specific. I wonder how Pungi handles rpm-ostree to ostree to LMC dracut arguments? L219-L227 Dracut is run on L272. Looking at how dracut is built.)
Branch back up to virt_install into class QEMUInstall https://github.com/weldr/lorax/blob/master/src/pylorax/installer.py#L134 injected KS script at https://github.com/weldr/lorax/blob/master/src/pylorax/installer.py#L208 (If the ks script is anything like https://github.com/rvykydal/anaconda-kickstarts/blob/master/atomic/fedora-atomic-pxe-live.ks then adding ostreesetup commands to the kickstart script can easily rebase; however, this requires the ostree repo to be passed as a URL or installer.py needs to be modified to mount the ostree repo filesystem.)

TLDR;
LMC requires passing in the same dracut commands defined in a normal fedora-atomic-host-base.json rpm-ostree definition.
LMC uses kickstart commands to trigger an ostreesetup.
LMC can access a local ostree repo over http as is.

Should atomic-assembler consider using LMC to build artifacts rather than something else to maintain? Since LMC is used in upstream Fedora/Atomic releases, it might be worthwhile to investigate as artifact creation. As far as I can tell, it uses the same VM process atomic-assembler does.

If we continue using an in-house build system, there's a well documented python api which can generate a fsimage https://github.com/weldr/lorax/blob/master/src/pylorax/imgutils.py#L508, initramfs https://github.com/weldr/lorax/blob/master/src/pylorax/creator.py#L210, and PXE scripts https://github.com/weldr/lorax/blob/master/src/pylorax/creator.py#L572-L575 https://github.com/weldr/lorax/blob/master/src/pylorax/creator.py#L280

[cgwalters]

OK, first: I just put up #113

Now, long story time. I spent quite a while thinking about
this yesterday.

When Atomic/rpm-ostree was first created, I very intentionally
did not try to change (much) how the OS is configured (/etc, config
management, cloud-init) - and most notably, ostreepayload with
Anaconda kickstart works just fine, you can use the timezone
verb, etc.

(Further, ostree being agnostic about underlying block storage
means that you can install using Anaconda with dm-crypt or not,
whatever fs layout, etc.)

And to me, rpm-ostree excels at the "pet system" case - that
blends well with Anaconda which is often used to generate these
types of systems via interactive installs.

So I don't think we're going to delete the rpmostreepayload code
in Anaconda - if for no other reason than we probably still
need it for Silverblue (that's a whole other thing though).

However...the usage of Ignition for Fedora CoreOS
is a very opinionated model for how systems are configured. And
for many of our use cases, this is a good thing - people
who are seeking image-based systems like ostree tend to
want to go full "immutable infrastructure".

Using Ignition has a lot of consequences - it means that
for our users, we won't be exposing Kickstart or cloud-init.
And at a deeper technical level, it just works fundamentally
differently.

For example, see systemd/systemd#8268 (comment)
Because our ostree payload does not have /etc/machine-id,
if someone tried to use Anaconda to install it, the kickstart services
verb would not work.

Now potentially, we could teach anaconda about this - it
might work to simply systemd-machineid-setup in %post.
But the higher level point here stands - Ignition and Anaconda
overlap significantly.

And more importantly, I am convinced now for many reasons
that it's critical for Fedora CoreOS-derived systems
to offer the simple "dd to disk" install method:
coreos/fedora-coreos-tracker#18
We have had several requests for this over time.

Hence, I think the balance weighs towards not using Anaconda
at build time either.

And now here's the thing - all of the current image-building
tools in the Fedora ecosystem (ImageFactory, virt-install,
and lorax-composer/LMC) end up being wrappers around running
Anaconda in a VM.

Another major technical point here is that for ostree-based
systems, our "build" is really an ostree - VM images
are just "wrappers". This is an enormous difference from
people who are just doing "yum install foo bar baz" inside
kickstarts and using that to make AMIs or Vagrant boxes
or whatever.

And consequently, the usage of Anaconda is much much smaller.

As soon as you ship an OSTree-based system it has huge
implications for the delivery model.

The code today in coreos-assembler ties the ostree and
VM images together closely in a way that I'm quite
happy with so far - we never made something as nice as
this back in the Atomic days.

All this to say: I think we should share code with
lorax/LMC - if we end up using Python that will be easier.
And there are a lot of points where we do want to align;
it'd be good in particular to generate the same
PXE layout, as a lot of Fedora/Red Hat ecosystem tools
end up parsing that. But I think our best path here
is to have a custom high level tool rather than try
to wedge the things we do directly into e.g. lorax/LMC.

[cgwalters]

The current situation of using Anaconda both for per-system installs and as a "golden image" building tool creates a lot of tension and problems. For example, when generating a Fedora Cloud or Atomic Host image, we don't want the system go out and do geolocation to try to find the current timezone. We don't want it to persist /var/lib/systemd/random-seed. We don't want whatever network configuration the build-side VM has to be persisted.

Today, we often work around these sorts of things in kickstarts. But things feel a lot cleaner if Anaconda can primarily focus on "per-system" installs, and we have tooling (that again shares code where it makes sense) dedicated to making "unconfigured golden images".

Of course, since coreos-assembler obviously isn't going to be building "classic" images like Fedora Cloud, us not using Anaconda for that increases the likelihood of bugs there. But to me that's the past 😄 There's also the "LiveOS" style systems like current Fedora Workstation. That one falls in the middle, but it seems fine to me to keep using LMC for it. The real "middle" case is Silverblue...whether we use coreos-assembler to build that is a whole other question - whether we support Ignition for it...a messy topic.

[Promaethius]

@cgwalters cheers for the good read. There's a lot about the underlying utilities that I do not understand.

How does the current coreos-assembly differ from being a vm wrapper to install an ostree and ship as an image?
I see that coreos-assembler uses fedora_everything as the base image to install ostree into. Does this circumvent the need for anaconda since anaconda installs per-system? If I'm to understand the LMC correctly, it doesn't need an anaconda disk to use kickstart scripts.

One other difference between LMC and coreos-assembler's virt-install... the network is disabled and the ostree is mounted as a filesystem https://github.com/coreos/coreos-assembler/blob/master/src/virt-install#L112 so I'll mention that in the pull request.

[cgwalters]

I see that coreos-assembler uses fedora_everything as the base image to install ostree into. Does this circumvent the need for anaconda since anaconda installs per-system?

No, it currently just runs Anaconda in a VM (via virt-install) on the build system side, which has all of the issues listed above.

Today, this project (coreos-assembler) does not generate an installer ISO, or a dd-able disk image for bare metal. That's a lot of the discussion here.

[Promaethius]

So, where is this project headed in terms of Ignition vs Anaconda?

Of course, since coreos-assembler obviously isn't going to be building "classic" images like Fedora Cloud, us not using Anaconda for that increases the likelihood of bugs there. But to me that's the past 😄

I totally agree with this. Using an interpreted language as init system scripting feels wrong in terms of predictability. Ditching Anaconda feels like it should have its own issue; however, it ties in by branching the installation process away from LMC and the current coreos-assembly process.

In which case, how will base images be built? If we're not using fedora_everything, will the wrapper pass in build commands via qemu into the VM? Will we have to maintain our own base image?

Additionally, is there any issue with branding?

Removing Anaconda/kickstart might be feasible (oddly) through using Lorax instead of LMC. A quick read of the docs indicate that fedora_everything is built through Lorax. Merging a dnf squashfs with basic kernel to create a system image. Would it be possible to use this process? It would require some creativity with rebasing without initializing.

[cgwalters]

In which case, how will base images be built?

In my opinion, we should use supermin to inject content from the container into "utility VMs" - we'd create filesystems and install the bootloader etc. from there.

A major advantage of this is that we don't need to download a separate installer ISO - there is only the coreos-assembler container content.

And end users don't use it either - they will just have a dd-able disk image that is configured on firstboot via Ignition.

[Promaethius]

In my opinion, we should use supermin to inject content from the container into "utility VMs" - we'd create filesystems and install the bootloader etc. from there.

Looking at the example, supermin looks powerful and light!
The documentation indicates that packages and dependencies are installed from the host system. This creates a lot of flexibility when initiating the appliance given that the ostree package can be "injected" from the build container into the appliance without having to use any ks scripts. Additionally, there won't be any anaconda artifacts.

I guess this sets us back to square one though as far as extracting vmlinuz, initramfs, and squashfs from the appliance.

[cgwalters]

I guess this sets us back to square one though as far as extracting vmlinuz, initramfs, and squashfs from the appliance.

No, the appliance is a server-side only thing. As far as artifacts consumed by end users, I think we'd generate a build/$x/pxe subdirectory with kernel/initramfs, and a build/$x/live.iso or so?

It's interesting, it looks to me like Container Linux today puts the whole OS in the initramfs for iPXE, see https://coreos.com/os/docs/latest/booting-with-ipxe.html

[cgwalters]

In contrast, Anaconda today has only a slightly custom initramfs, and downloads the "stage2" - the much bigger squashfs with Anaconda inside - itself.

[Promaethius]

So supermin --prepare creates an appliance definition with packages, --build -f ext2 --copy-kernel builds vm artifacts (initrd, kernel, hda) with the host kernel, and then coreos-assembler uses guestfish to pass in the arguments to configure the ostree and generate a new initramfs in the qemu booted vm.

Are there any concerns with creating the kernel this way since ignition needs a kernel mod? Or is that up to the rpm-ostree generated dracut initramfs created inside the supermin generated vm? Will copying the device tree through --copy-kernel create any issues?

[cgwalters]

coreos-assembler uses guestfish to pass in the arguments to configure the ostree and generate a new initramfs in the qemu booted vm.

No - the ostree commit contains the initramfs. VM and disk images are just "wrappers" around the ostree commit which has basically everything that isn't the filesystem/disk formats.

[Promaethius]

Ok, maybe I'll get it this time...

The ostree commit is built in the container, it contains the initramfs.
Supermin prepares a minimal VM with the host kernel and some packages including the rpm-ostree toolchain; it packages the kernel into a file, the base filesystem into ext2, and a initramfs.
Qemu-kvm boots the ostree initramfs, supermin kernel, and supermin hda to configure the filesystem with the previous ostree commit.
It reboots to apply changes and removes any leftovers.

PXE artifacts would be the ostree initramfs, supermin kernel, and modified hda?

Update: I've spent some time today messing around with supermin. There are a couple of overlaps and it "seems" that supermin is really relevant as a tool which can create an ext2 image that includes rpm packages such as bash and rpm-ostree. It'll copy the fedora-28-everything x86_64 kernel fine but that can be switched out for other kernel builds easily. The only discardable artifact is the supermin generated initramfs. Swap that out with the rpm-ostree generated dracut initramfs.img? Feels really slick since I'm inferring from initramfs documentation that the generated dracut initramfs will initialize the ostree repo.

[cgwalters]

Supermin prepares a minimal VM with the host kernel

In our case, it's really the kernel in the assembler container, which can be different from the target system defined in the config. Today the container is actually f28 and the default config f29, but e.g. a supported use case would be having us rebase the assembler container to f29, but someone continues to build f28 systems from it - and all of this is distinct from the host system.

Qemu-kvm boots the ostree initramfs, supermin kernel, and supermin hda to configure the filesystem with the previous ostree commit. It reboots to apply changes and removes any leftovers.

No, the supermin VM wouldn't be booted with the target kernel or initramfs, but with the assembler container's version. It would write to a disk image which was a file on the host system.

[ashcrow]

/cc @nhorman for visibility

[cgwalters]

One thing I want to tie in here too is that based on the discussion in coreos/fedora-coreos-tracker#18 (comment)

We will actually need two raw disk images; one for 512b hard drives and one for 4k.

[cgwalters]

For reference, previous code I wrote that uses libguestfs + the libostree API to make disk images: https://gitlab.gnome.org/GNOME/gnome-continuous/blob/master/src/js/libqa.js

However...I think SELinux makes this harder. We will need to take care of labeling all of the files we generate afterwards using the policy in the target root.

[cgwalters]

Retitling this issue as it ended up being a lot of "CoreOS and Anaconda".

[cgwalters]

The things I think we need to keep in mind sharing with Anaconda are:

• Tooling to build bootable media like ISOs, particularly as that relates to multi-arch. On that topic, see installer: use lorax s390x templates, allow adding parameters at boot #732
• Alignment on the storage stack; it's clear that Ignition will need to support LVM at some point. Anaconda's storage stack (and Kickstart) has a lot of that. I could imagine something where we put a JSON frontend in Ignition that just gets "transpiled" to a kickstart fragment that we pass down to blivet...yeah it'd be ugly but...

[cgwalters]

Because our ostree payload does not have /etc/machine-id,
if someone tried to use Anaconda to install it, the kickstart services
verb would not work.

That said I was thinking about this - clearly Anaconda could just create /etc/machine-id in %post if it didn't already exist.

It's probably fairly solvable to have a single OSTree build that could be consumed by Anaconda as well as injected into CoreOS-style systems.

[cgwalters]

Also, another thing related to this is https://bugzilla.redhat.com/show_bug.cgi?id=1061478

[tuan-hoang1]

The things I think we need to keep in mind sharing with Anaconda are:

* Tooling to build bootable media like ISOs, particularly as that relates to multi-arch.  On that topic, see #732

Maybe it's already clear, but just for the record, that #732 is for non-anaconda path.

[tabowling]

@cgwalters as I mentioned last week, @larskarlitski and @teg are working on alternate tooling to build osimages breaking away from anaconda/lorax to be free of simmilar issues. Rpm-ostree are in the future roadmap https://github.com/osbuild/osbuild we should collaborate.

[cgwalters]

Rpm-ostree are in the future roadmap https://github.com/osbuild/osbuild we should collaborate.

OK, a bit ironic though to discuss code sharing with osbuild (which is avoiding Anaconda) in this issue which is about code sharing/alignment with Anaconda 😉

So I moved that to osbuild/osbuild#93

[M4rtinK]

OK, a bit ironic though to discuss code sharing with osbuild (which is avoiding Anaconda) in this issue which is about code sharing/alignment with Anaconda wink

Actually, we have had some preliminary discussions about code sharing between Anaconda and osbuilder, namely bootloader configuration and possibly other things that are shared between system installation & image generation.

[tabowling]

OK, a bit ironic though to discuss code sharing with osbuild (which is avoiding Anaconda) in this issue which is about code sharing/alignment with Anaconda

@cgwalters Yes, you're correct. Apologies for conflating or causing any confusion of this topic. Thanks for initiating the conversation in the more appropriate place.

[dustymabe]

I'm going to close this old issue.