enhancements: Propose CoreOS layering

[cgwalters]

This enhancement proposes:

• A fundamental new addition to ostree/rpm-ostree, which is support for directly pulling and updating the OS from container images (while keeping all existing functionality, per-node package layering and our existing support for pulling via ostree on the wire).
• Documentation for generating derived (layered) images from the pristine CoreOS base image.
• Support for switching a FCOS system to use a custom image on firstboot via Ignition
• zincati will continue to perform upgrades by inspecting the upgrade graph from the base image.

---

Work tracker

• ostree-ext 0.4.0
• ostree-ext 0.5.0
• Use ostree-rs-ext 0.5.0 rpm-ostree#3139
• Add deploy-via-container key for image.yaml coreos-assembler#2523
• proxy: Add GetConfig, add manifest list support, add an integration test containers/skopeo#1495
• Fast-track skopeo-1.5.1-1.fc35 fedora-coreos-config#1331
• release: Copy oscontainer to quay.io fedora-coreos-pipeline#383
• Support rpm-ostree install in container builds rpm-ostree#3227

Trying this out

You'll need rpm-ostree v2021.14 or newer.

• Fedora: https://bodhi.fedoraproject.org/updates/FEDORA-2021-5e6a48669c
• EL8 COPR

Current snapshot container base image: quay.io/cgwalters/fcos

---

New rpm-ostree architecture diagram

            ┌─────────────────┐
            │                 │
         ┌──┤   rpm-ostree    ├────┐
         │  │                 │    │
         │  └─────────────────┘    │
         │                         │
         │                         │
┌────────▼────────┐       ┌────────▼────────┐
│                 │       │                 │
│     libdnf      │       │  ostree-rs-ext  ├────────────┐
│                 │       │                 │            │
└────────┬────────┘       └────────┬────────┘            │
         │                         │                     │
┌────────▼────────┐       ┌────────▼────────┐   ┌────────▼─────────┐
│                 │       │                 │   │                  │
│     librpm      │       │    ostree-rs    │   │  imageproxy-rs   │
│                 │       │                 │   │                  │
└─────────────────┘       └────────┬────────┘   └───────────┬──────┘
                                   │                        │
                          ┌────────▼────────┐   ┌───────────▼──────────────────┐
                          │                 │   │ (forked process)             │
                          │     ostree      │   │                              │
                          │                 │   │     ┌──────────────────┐     │
                          └─────────────────┘   │     │                  │     │
                                                │     │     skopeo       │     │
                                                │     │                  │     │
                                                │     └────────┬─────────┘     │
                                                │              │               │
                                                │     ┌────────▼─────────┐     │
                                                │     │                  │     │
                                                │     │ containers/image │     │
                                                │     │                  │     │
                                                │     └──────────────────┘     │
                                                │                              │
                                                │                              │
                                                │                              │
                                                └──────────────────────────────┘

[jlebon]

Nice, this is an interesting idea!

I wonder if we should have an intermediate enhancement before this one which is about officially shipping FCOS as a container and supporting rebasing to it and updating. And then from there we could have this enhancement focus more precisely on the idea of derivation.

[jlebon]

Worth discussing also the intersection between derivation and Ignition. There's clearly a lot of overlap there in terms of configuration.

[cgwalters]

I wonder if we should have an intermediate enhancement before this one which is about officially shipping FCOS as a container and supporting rebasing to it and updating.

Not opposed, but in the end "encapsulation" wasn't intended to be a user-visible thing until perhaps the very end, and it's not that interesting. This is much more consequential and acts as a driver for it, and while they're logically separate from the user PoV I think we might as well do them together.

[LorbusChris]

fyi @vrutkovs, I think this'll be great for building okd-machine-os

[cgwalters]

https://mail.gnome.org/archives/ostree-list/2021-November/msg00000.html

[vrutkovs]

+1 from me, we're certainly interested in this in OKD

[jlebon]

Would be nice if there was a halfway approach where we do support the Dockerfile approach but the only supported operation is still to just apply a Butane config, potentially with some locally referenced files that were copied from a previous build stage.

Personally, my primary concerns with the "freehand" Dockerfile approach is that:

1. It's too easy to make drastic changes. E.g. in rpm-ostree, we have this conscious decision to separate out pure package layering and base package overrides. This distinction does not exist here, and removing these guardrails means it becomes much easier for users to shoot themselves in the foot. I guess one thing we could do is have rpm-ostree detect when content was modified instead of purely added and require a --allow-base-overrides or something?
2. It makes the configuration story messier. With this, you can now configure at both image build time and Ignition boot time. And they each use completely different languages. So I think there's potential there to cause user confusion. It also impacts support, which now needs to understand more types of inputs.

Focusing on Butane configs as the layering mechanism in my opinion helps with both of those because it's more declarative and so easier to analyze, and it's the same configuration language we've been speaking so far.

[jlebon]

To be clear, I'm OK with this too, and I know we can always tweak things going forward (though it's always harder to add restrictions than it is to remove them). Just wanted to point out what IMO are some non-negligible risks of this approach.

[cgwalters]

we have this conscious decision to separate out pure package layering and base package overrides. This distinction does not exist here,

That's not quite true, because we will still have the original base image. Reverting back to the "golden FCOS base" will also be a just an rpm-ostree rebase away.

Additionally, because we will have yum|dnf -> rpm-ostree in this image (xref https://coreos.github.io/rpm-ostree/cliwrap/) we can actually still impose the install vs override semantic.

Further, tooling can perform filesystem and rpmdb diffing between layers - we have all that code today.

[cgwalters]

And they each use completely different languages.

Also, today one can use any tool to generate Ignition, which definitely happens (e.g. openshift-install generates Ignition via some custom Terraform stuff) and we support that.

More broadly I think we are in a position of needing to support "low level" as well as arbitrary configuration mechanisms, but we also try to have opinions on higher level tooling.

So here Dockerfile is that lowest common denominator for arbitrary, low level (kinda) mechanism - but we can streamline higher level workflows inside of that.

[jlebon]

we have this conscious decision to separate out pure package layering and base package overrides. This distinction does not exist here,

That's not quite true, because we will still have the original base image. Reverting back to the "golden FCOS base" will also be a just an rpm-ostree rebase away.

Right, but the power a user wields in their Dockerfile sees it all as just one filesystem. E.g. nothing stops them from doing COPY my-libc.so /usr/lib64/libc.so and the build system will be happy to build that. Whereas if it's Butane based, we can immediately tell they're modifying base content and fail the build. So the failure happens at image build time instead of deploy and reboot time. Of course, we do want to provide flexibility to modify base content, but IMO it should be explicitly opted in.

Combining this with ostreedev/ostree-rs-ext#159, maybe we could have this ostree container finalize step take that switch? E.g. ostree container finalize --allow-overrides.

Additionally, because we will have yum|dnf -> rpm-ostree in this image (xref coreos.github.io/rpm-ostree/cliwrap) we can actually still impose the install vs override semantic.

Hmm, can you expand on the UX you're thinking of here? dnf install will automatically upgrade an already installed package so we'd have to require some added switch (related: coreos/rpm-ostree#2844 (comment)).

[cgwalters]

Hmm, can you expand on the UX you're thinking of here? dnf install will automatically upgrade an already installed package so we'd have to require some added switch (related: coreos/rpm-ostree#2844 (comment)).

Well, the cliwrap status quo today errors actually when someone types dnf install. I know this is a perpetually confusing thing but so far cliwrap does not involve necessarily invoking traditional yum|dnf logic. It opens the door to that of course.

So specifically if e.g. someone times RUN yum install usbguard in their Dockerfile, we would error out, and they'd have to do RUN rpm-ostree install usbguard which would run through all of the same logic that exists today which would only layer, not upgrade dependencies. (There's...a lot implied in this whole "rpm-ostree in container builds" flow, really worth splitting out to a separate issue; will look at that)

[cgwalters]

The real status quo right now to be clear is that dnf install usbguard will give "command not found" and rpm-ostree install usbguard will error out because you're in a container. And rpm -Uvh https://example.com/usbguard.rpm will work.

[cgwalters]

coreos/rpm-ostree#3227

[cgwalters]

> podman run --rm -ti quay.io/cgwalters/fcos bash
bash-5.1# dnf install usbguard
bash: dnf: command not found
bash-5.1# rpm-ostree install usbguard
error: This system was not booted via libostree; found $container=podman environment variable.
rpm-ostree is designed to manage host systems, not containers.

bash-5.1# 

[jlebon]

So specifically if e.g. someone times RUN yum install usbguard in their Dockerfile, we would error out, and they'd have to do RUN rpm-ostree install usbguard which would run through all of the same logic that exists today which would only layer, not upgrade dependencies.

Ack, I like that.

What I'm trying to make sure here is that OCP customers who will be interacting with this feature are still subject to basic rules by default so that our QA efforts aren't trivially invalidated. I.e. I don't want them to be able to change /usr base content just as easily as they can /etc content. If the only way OCP customers interact with this is through Butane or MCs for example, then I think that's fine. Otherwise, I think we should make them opt into modifying base content explicitly (and maybe e.g. we only support them doing this if they have a support exception).

Taking this feature in isolation (e.g. for FCOS users), typing rpm-ostree rebase $my_pullspec already implies some risk acceptance (though there's still value in keeping guardrails there too of course). But in the OCP case, the MCO will be what's running that command.

[jlebon]

Anyway, don't want to block on this necessarily, but I think we should keep in mind this idea of distinguishing between modifying vs adding when designing and implementing this and higher levels. Ideally we'd mention it somewhere in the proposal at least.

[cgwalters]

https://bodhi.fedoraproject.org/updates/FEDORA-2021-5e6a48669c is out!

[cgwalters]

Do you have any suggested text on this? Pasting as a comment works, or a diff. I added some text in the Dockerfile section.

There's really a whole lot that could be written here around debuggability/introspection. One thing that ostree can provide much faster than the default containers/storage stack is filesystem level diffs, because it's all content addressed.

Hmm. I guess in theory too we could actually require something like
LABEL ostree.override=true to support overwriting content in previous layers?
That could be somewhat easily enforced in the ostree-container stack.

[cgwalters]

Anyway, don't want to block on this necessarily, but I think we should keep in mind this idea of distinguishing between modifying vs adding when designing and implementing this and higher levels. Ideally we'd mention it somewhere in the proposal at least.

One thing I want to say is this is a super useful discussion! I mean, I have opinions on a lot of this stuff, but for sure there is a vast space here filled with nuance and important details.

I think this enhancement is just the start to a story, not an end.