Relationship with https://github.com/osbuild/osbuild/pull/1402

[cgwalters]

If I am understanding things correctly, https://github.com/osbuild/images is basically Go versions of mpp files?

So the work on this branch is basically redoing https://github.com/osbuild/osbuild/pull/1402/files#diff-39b52d4acfa02644f133a682a54270e15f5a8d69de4b1ff0482af589418da8b0 in Go, is that correct?

Now when I look at both of these I think an overall problem is that it's hardcoding distribution and "flavor" specific things in the pipeline code. We have the basic things like UEFI vendor hardcoding in the MPP version...that one will get fixed with the switch to bootupd.

But then the next problem is things like hardcoding ext4. I think in the general case container payload needs to be Source Of Truth for this stuff. This is how bootc install does it - the container ships its own config, so e.g. one might have a base RHEL image using xfs by default, but automotive can override that to ext4 (because they need fsverity) just by changing the container.

That said, a problem with this model (that was also brought up a bit in rhinstaller/anaconda#5298 ) is that it requires fully pulling and "mounting" the container image before we can make the target filesystem.

One thing I think that could help is standardized metadata keys on the container image. What format that should be is obviously a huge bikeshed topic.

[cgwalters]

The other thing with the bootc install flow of course is that the container is its own "install root" in this flow - i.e. the mkfs.xfs being invoked is the one from the container, not the host.

This avoids all host dependencies except (kernel, podman). However, it does mean the container has to carry those tools...not really a problem for most of our initial use cases though.

I guess what I'm getting at here...can you see a reason not to reorient things more towards just invoking bootc install? IOW instead of skopeo copy of the image into a dir store we run it as a container, targeting a loopback device etc?

[cgwalters]

One thing I think that could help is standardized metadata keys on the container image. What format that should be is obviously a huge bikeshed topic.

How about for Fedora derivatives i.e. original Red Hat Linux derivatives container images we say that the default thing we support is kickstart format?

Something like

LABEL com.redhat.kickstart "clearpart --all --initlabel --disklabel=gpt \
part biosboot  --size=1    --fstype=biosboot \
part /boot/efi --size=100  --fstype=efi \
part /boot     --size=1000  --fstype=ext4 --label=boot \
part / --grow --fstype xfs"

?

Though the ugly thing about this is the general architecture specific nature of it when we start talking about ppc64le and s390x...maybe we slightly fix the semantics of autopart so that it can just be:

LABEL com.redhat.kickstart "autopart \
part / --grow --fstype xfs"

or so?

[cgwalters]

OK I ended up hacking up https://github.com/cgwalters/osbuildbootc which takes bits of coreos-assembler (the qemu-for-build parts, split across some horrid shell script and some acceptable Go code) and instead of the create_disk.sh script we just invoke bootc install in the transient VM.

It seems to work well!

I do feel fairly strongly that we need to support a virtualized (i.e. no host privileges) build process. Among other reasons, I also feel very strongly that we should be building out a "developer SDK" type flow and to test things we really want virtualization anyways.

That said there's a lot of stuff here that clearly makes sense to intersect instead with the osbuild pipelines and tooling - things like dealing with cloud images for example.

WDYT about merging these two projects? Move osbuildbootc here and merge the code from this project into it? There's some interesting things here around moving the osbuild pipelines to be automatically virtualized if we don't have privileges; the auto pipeline does this and I think we can do it in a nicer way.

[ondrejbudai]

If I am understanding things correctly, https://github.com/osbuild/images is basically Go versions of mpp files?

So the work on this branch is basically redoing https://github.com/osbuild/osbuild/pull/1402/files#diff-39b52d4acfa02644f133a682a54270e15f5a8d69de4b1ff0482af589418da8b0 in Go, is that correct?

mpp is not used in production, our team considers it as just a helper tool for developing osbuild and new images. osbuild/images is the real thing - it allows us to create manifests:

1. with high-level customizations
2. while sharing as much manifest-generation code between distributions and image types
3. for osbuild-composer that is run in a HA, distributed environment

Nothing from these 3 items is currently possible in mpp. Items 1 and 3 are requirements for running the hosted Image Builder. Item 2 is important for our sanity.

Hope this explains the difference. :)

But then the next problem is things like hardcoding ext4. I think in the general case container payload needs to be Source Of Truth for this stuff. This is how bootc install does it - the container ships its own config, so e.g. one might have a base RHEL image using xfs by default, but automotive can override that to ext4 (because they need fsverity) just by changing the container.

I agree that tying partitioning and the contents of the container absolutely makes sense, otherwise you can end up in a lot of unbootable situations (btrfs on a non-btrfs kernel, LVM without lvm2 installed, etc.)

I guess what I'm getting at here...can you see a reason not to reorient things more towards just invoking bootc install? IOW instead of skopeo copy of the image into a dir store we run it as a container, targeting a loopback device etc?

I think that having one way to create a bootable image from a container would be great. We started with invoking ostree image container deploy because we already had all code for it in osbuild and we wanted to iterate as fast as possible. That proved to be a successful strategy since it started this whole discussion. :)

Note that we should also probably talk about how this new bootable containers initiative relates to the "coreos built by osbuild" initiative that @dustymabe and others were working on. If we want bootc everywhere, then osbuild/osbuild#1402 is basically obsolete and we should stop all efforts going into ostree image container deploy.

I do feel fairly strongly that we need to support a virtualized (i.e. no host privileges) build process. Among other reasons, I also feel very strongly that we should be building out a "developer SDK" type flow and to test things we really want virtualization anyways.

Note that the current Image Builder deployment doesn't support virtualization at all. Everything is AWS-based which sadly doesn't have KVM support (well, expect for metal instances ofc). So one requirement from our side is to be able to run everything without KVM.

[cgwalters]

mpp is not used in production, our team considers it as just a helper tool for developing osbuild and new images. osbuild/images is the real thing - it allows us to create manifests:

(This is somewhat of a tangent, it relates the below in that I think in order to scale in general we really do want something more declarative/extensible than fixed Go code; but I understand the overall point you're making)

Note that we should also probably talk about how this new bootable containers initiative relates to the "coreos built by osbuild" initiative that @dustymabe and others were working on. If we want bootc everywhere, then osbuild/osbuild#1402 is basically obsolete and we should stop all efforts going into ostree image container deploy.

That's the tricky bit! CoreOS doesn't use bootc today (in any way) because...well, it was more of a side project until recently. And bootc install in particular definitely overlaps with Ignition and all the stuff going on there (see also coreos/fedora-coreos-config#2141 ). It may be we need to target CoreOS with bootc and if that's the case we can definitely align things much more.

We have some smaller details being debated like the aleph version and whether it makes sense to carry forward that "coreos stuff".

I agree that tying partitioning and the contents of the container absolutely makes sense, otherwise you can end up in a lot of unbootable situations (btrfs on a non-btrfs kernel, LVM without lvm2 installed, etc.)

We had a realtime chat about this and a general consensus:

• It definitely makes sense to have metadata in the container
• There was not an immediate proposal for what that metadata is

So one requirement from our side is to be able to run everything without KVM.

Yes, agreed (also for Power not supporting KVM in the future).

[cgwalters]

Anyways, any objections to trying to merge these two projects and start to make it a shared devenv/sdk that runs as a container (both unprivileged with virt and privileged)? Something like github.com/osbuild/osbuild-bootc-devenv ? Or osbuild-bootc-assembler ? Or maybe go short with osbuildc?

[cgwalters]

BTW I think a another point of tension here that we didn't touch on is: in the current README.md this project suggests things like defining users to inject outside of the container image.

But I do think we actually want to push more towards a place where that type of customization is done in the container.

[ondrejbudai]

Anyways, any objections to trying to merge these two projects and start to make it a shared devenv/sdk that runs as a container (both unprivileged with virt and privileged)? Something like github.com/osbuild/osbuild-bootc-devenv ? Or osbuild-bootc-assembler ? Or maybe go short with osbuildc?

I think we still don't have a good understanding of what you want to merge. Which parts you want to take from cgwalters/osbuildbootc and which ones from this repo? :)

[achilleas-k]

But I do think we actually want to push more towards a place where that type of customization is done in the container.

Do we really want users defined in the "immutable" (sorry for the loaded term) part of the system? That's something I always considered to be a bad idea for security and management reasons. For example, revoking an ssh key would require updating the system. Or is this not the way it works now?

[ondrejbudai]

We also saw some issues when a user id changed in a updated ostree commit, right, @achilleas-k?

[achilleas-k]

Anyways, any objections to trying to merge these two projects and start to make it a shared devenv/sdk

To follow-up on Ondrej's question:
I know I asked the same question in our chat earlier but I'm still not clear on what merging means here. This project (this repository) is just a quick Dockerfile and build script for building and distributing the osbuild-deploy-container binary that's defined on this branch and this PR.

Do you want to use that osbuild-deploy-container tool and add some code to it so that it does things more in a bootc-compatible way?

[achilleas-k]

But then the next problem is things like hardcoding ext4. I think in the general case container payload needs to be Source Of Truth for this stuff. This is how bootc install does it - the container ships its own config, so e.g. one might have a base RHEL image using xfs by default, but automotive can override that to ext4 (because they need fsverity) just by changing the container.

(I think we went over this already but I'd like to write it up here as well for clarify and posterity)

I'd like to have the same kind of flexibility in osbuild/images or osbuild-deploy-container (or whatever we end up with as a tool for this, separate or integrated into osbuild-*). The hard-coded partition tables you linked to are the base partition tables for this image we're building and something I added to quickly get that PoC out last week. But I agree that it would be great for the container itself to define its own partitioning (and other stuff). For one, it gets around the problem of creating partitions that the container can't boot or work with (either because the kernel doesn't support it or the tooling isn't in the image). So this is where we have the opportunity to come up with a specification of what that container metadata should look like.

[achilleas-k]

So this is where we have the opportunity to come up with a specification of what that container metadata should look like.

Thinking about this more: We could even go as far as having the container define its partition table and also declare what it can support as optional alternatives (for example, it defines an ext4-based partition table but also supports xfs-on-LVM) which could give users the ability to modify the defaults without breaking their deployment.

[cgwalters]

Do we really want users defined in the "immutable" (sorry for the loaded term) part of the system? That's something I always considered to be a bad idea for security and management reasons. For example, revoking an ssh key would require updating the system.

A big part of the idea here though is that "source of truth" is the container image. For some deployments, having changes only happen through the container is actually very much a feature.

Now SSH keys are among the messiest things in this area because openssh's default config doesn't want things to work this way, but see https://github.com/containers/bootc/blob/main/docs/install.md#injecting-ssh-keys-in-a-container-image
Is this beautiful? No. But it means that the state of the system is more fully captured by the container image, which some users will definitely want.

I mean the other problem with the "out of band install time config" (kickstart, blueprints) is actually that it then defers to some other "day 2" management story. See also containers/bootc#190 which proposes a blueprint/kickstart equivalent that is stored in a registry, but then also containers/bootc#22

[achilleas-k]

This is a bigger discussion of course and it would go beyond what we do here in this corner of the whole project. I tend to lean towards separating the "base image" and "deployment-time" configurations, but in the end it only matters in terms of how this affects us here.

1. If we go with everything in the container and it's part of the base image, then we don't really need to do anything. We just deploy the container and it's all there.
2. If we go with install/deploy-time config, with configmaps or similar, then we'd have to support it in any alternative tooling we end up using, or pass it down to bootc through our interfaces at the least.

[cgwalters]

This project (this repository) is just a quick Dockerfile and build script for building and distributing the osbuild-deploy-container binary that's defined on this branch and osbuild/images#250.
Do you want to use that osbuild-deploy-container tool and add some code to it so that it does things more in a bootc-compatible way?

What I'm thinking is really just:

• We literally merge the git repositories while keeping all the tooling separate, keeping both tools working exactly as they are today
• Ensure that relevant people have access to the repo
• Start iterating towards aligning things incrementally

[achilleas-k]

Let's do it then. Are we putting it under the CentOS org to live alongside the centos-boot repo?

[cgwalters]

Well...I was thinking it's more like osbuild (tooling) than it is operating system definitions (centos-boot) right?

Let's do it then.

Awesome, I'm working on it right now! Mega PR incoming

[cgwalters]

Well...I was thinking it's more like osbuild (tooling) than it is operating system definitions (centos-boot) right?

Well I guess these things cross in the osbuild/images space right now...this is a tricky topic. But I think we're more agreeing to shift that stuff to the OS definitions longer term right?

[cgwalters]

Hmm sorry diving into this slightly more I see what you guys were getting at around "this repository is a shell"...leads to a space where we probably want to expose the functionality from your fork of osbuild/images into a Go API right? And then we just vendor osbuild/images here?

[achilleas-k]

Well...I was thinking it's more like osbuild (tooling) than it is operating system definitions (centos-boot) right?

Right, that makes sense. I was thinking it might be more high level project specific but let's bring it here.

Well I guess these things cross in the osbuild/images space right now...this is a tricky topic. But I think we're more agreeing to shift that stuff to the OS definitions longer term right?

Yes, but I think there are two things we might want to keep separate (if it makes sense):

1. osbuild/images as a library/sdk for defining images with:
   a. a bunch of types and functions for (ideally declaratively) defining an image, and
   b. domain knowledge for what defines a distro, what's needed for a specific machine architecture, what changes are required for a different cloud/virt environment.
2. Tooling specifically for building container-based images, or interacting with bootc, or making a container like the one we have in here.

So 1. is what osbuild/images is now, though there's some work to be done to reach the ideal version of 1a.
Also 1b. exists in osbuild/images, but there's some work to be done to make it more useful "from the outside".
As for 2., it can (and in some current analogous cases does) live in osbuild/images, but it might make sense to have it separately. We could always merge them down the line. It might also make sense to be in osbuild-composer in the end, depending on what we need to do for the image builder service.

[achilleas-k]

Hmm sorry diving into this slightly more I see what you guys were getting at around "this repository is a shell"...leads to a space where we probably want to expose the functionality from your fork of osbuild/images into a Go API right? And then we just vendor osbuild/images here?

Right, yes. I think what would make sense is to have the osbuild-deploy-container code here and vendor osbuild/images. My thinking is that if we use osbuild-deploy-container as a starting point and iterate on it, it will help us figure out what we need from osbuild/images that either isn't exposed yet as a public API, or isn't clean/tidy/flexible enough to be useful. And we take it from there.

[dustymabe]

There's a lot going on in this thread :) - I'd be down for a sync session to discuss the various topics as I see some opportunity for alignment/agreement.

[ondrejbudai]

A month passed, and the repository looks much different nowadays. I don't think there's anything actionable here, so lemme just close this. Feel free to object/reopen.

Note that #18 has also some interesting discussion regarding the high-level architecture of image builds (and is actually actionable).