RPM-OSTree - git for operating system binaries

[schaefi]

With rpm-ostree, you can create versioned filesystem trees to provide atomic, reliable upgrades. "git-like" management of appliances would be pretty sweet.

[schaefi]

https://wiki.gnome.org/Projects/OSTree

[Conan-Kudo]

RPM-OSTree package for openSUSE Leap 15.0 / Tumbleweed: https://build.opensuse.org/package/show/home:Pharaoh_Atem:SUSE_RPMOSTree/rpm-ostree

Docs and examples:

• https://github.com/projectatomic/rpm-ostree/blob/master/docs/manual/treefile.md
• https://pagure.io/fedora-atomic/tree/master
• https://pagure.io/workstation-ostree-config/tree/master
• https://pagure.io/fork/siosm/workstation-ostree-config/tree/f28-kinoite

[davidcassany]

@Conan-Kudo I am still wondering two things:

1. What actually means supporting rpm-ostree in KIWI? Probably this caused by my lack of knowledge regarding rpm-ostree. I assume this is a new package manager right? or at most an extension of the feature set of DNF and/or any other that has some layer of cooperation.

2. Which is the use case? According to what I understand this is not much different of building a root-tree over an ostree, thus building rootfs in some container like layered filesystem, correct? So I am wondering what is actually the result of using rpm-ostree as a package manager for an e.g. Fedora build. Mostly because if rpm-ostree is a package manager (from KIWI POV) then it should not have impact on the build procedure. It should still be possible to build isos, vms, oem, etc.

I can imagine features like usign the derived_from attribute to import an ostree and create a new images out of it by adding a new transaction on top with rpm-ostree. Am I pointing to the right direction?

Also stuff like making each <packages> section as an individual transaction, even I am having hard time to see the use case and potential benefits of it.

We need to refine this support to narrow what actually means and also understand the use cases we are covering with it. @Conan-Kudo would be appreciated if you can help us on that regard.

[Conan-Kudo]

RPM-OSTree produces an OSTree, which is a special rootfs type representing a largely immutable operating system environment. The rpm-ostree program has features for producing and managing OSTrees from RPM content. It does leverage DNF stack libraries, but it does not have the same interface as DNF.

What it would mean from a KIWI perspective is that it would be able to orchestrate creation of OSTrees and applying them to the desired medium so that they can be used in the same way traditional Linux system trees are.

Today, there aren't any tools that make it easy to produce RPM-OSTree based things other than VM disk images with cosa. As KIWI can produce way more types and is easier to leverage (think OBS and similar), that would give it a major leg up for producing immutable infrastructure OSes. As it is, I currently cannot create a live ISO of an rpm-ostree based system, nor can I easily create layered OSTree repositories for feeding update repos or derivative distros.

@cgwalters and @jlebon would have a better idea of the exact mechanics here, as they're the current developers of rpm-ostree.

[cgwalters]

I currently cannot create a live ISO of an rpm-ostree based system,

cosa buildextend-live is used to generate the "live ISO" linked from the FCOS builds.

[schaefi]

Another player in the field of "packages" is luet

• https://luet-lab.github.io/docs/docs/

the term package in this regard means a container+plus_metadata

the term package in the context of rpm-ostree is an extended format for rpm (iirc)

In any case kiwi has no support for the sophisticated new idioms of packages which are actually sub-systems.
As of now I still don't know if there is a high demand to natively support these type of package managers in kiwi.

kiwi itself is flexible enough to use any os-tree you will give it. Therefore it's possible to combine rpm-ostree
or luet or whatever else will come providing "systems" to let those tools create/manage a new root tree and let
kiwi take over only for the task to create images out of it

From the perspective of a stacked solution I currently don't see the need to add work into kiwi.

However it might be a good idea to discuss a change in kiwi that makes the "installation of packages" as we see
it now an optional step. Currently the following is possible with kiwi:

$ kiwi-ng system build --description ... --target-dir ... --allow-existing-root

This will re-use an eventually existing root tree no matter who created it, luet, rpm-ostree, etc...
But the provided description still needs:

• at least one old school package repository from the distribution
• at least on <packages> section even if you don't install anything in addition to the existing tree

This tells us that kiwi itself expects some connection to repositories and packages as they are provided by the distribution vendors. The discussion we can have here is if we want to weaken this concept if customers can provide a system root tree to work with ?

Your feedback is welcome

Thanks

[davidcassany]

However it might be a good idea to discuss a change in kiwi that makes the "installation of packages" as we see
it now an optional step. Currently the following is possible with kiwi:

$ kiwi-ng system build --description ... --target-dir ... --allow-existing-root

This will re-use an eventually existing root tree no matter who created it, luet, rpm-ostree, etc...
But the provided description still needs:

* at least one old school package repository from the distribution

* at least on `<packages>` section even if you don't install anything in addition to the existing tree

This tells us that kiwi itself expects some connection to repositories and packages as they are provided by the distribution vendors.

I have thinking a quite a lot within the last few days regarding these topics. First things regarding the demand that there might be regarding these sort of ostree/luet/container like packages do think there is no demand on adding support at package manager level. It is certainly possible (I already did a successful PoC for Luet as a hobby thing) but even I believe it might make sense in some specific use cases I also think this is not best approach to take advantatge of these sort of technologies from an appliance building PoV.

What is appealing from containers? IMHO from developers PoV, beyond the isolation and the DevOps trends, something that is nice is the stacking or layering concept. You never start from scratch and you can always add some little salt to anything and turn something that was close to what you wanted to something else that really matches your needs or expectations. With all that you probably don't really need to care much about how your base layer was created, what is actually including and all configuration details. Damn easy to get started.

What if we make use of container technologies to build the idea of derived images for any type of image. Imagine that after the prepare step the created root tree (including all KIWI artifacts) is kept in some ostree or OCI storage/registry. So we could eventually rebuild the exact same image by just pulling in this stored prepare step (it would already contain the XML description and any other artifact that KIWI requires to create an image). Alternatively, we could recreate the whole tree (re run the prepare step on top of the already provided tree so, for instance, it could upgrade packages and rebuild again the provided image). But what if we also provide a delta of a new XML description? What if in that procedure we allow to include XML description artifacts that are simply merged with the base XML, so on the fly a new full XML gets created and applied on top of the imported rootfs.

Imagine we have a Live image, with a <type> like:

<type image="iso" flags="overlay" firmware="efi" kernelcmdline="splash" hybridpersistent_filesystem="ext4" hybridpersistent="true">
   <rootfs-base keep="oci:myliveimage:latest"/>
</type>

With all the packages and so one. So the prepare step of this image could eventually push the root-tree into an OCI image myliveimage:latest and the end of it. So a kiwi-ng system build call, as part of the build process, stores the WORKING root-tree in some OCI storage for any potential later reuse.

Then imagine I want to tweak this image and build a new one on top of this one. Imagine we could do something like:

<image>
<derived from="myliveimage:latest"/>
<packages type="image">
   <package name="my-new-branding"/>
</packages>
<packages type="uninstall">
   <package name="my-old-branding">
</packages>
</image>

And a kiwi-ng system build call on that system could eventually result into the former base XML with these additional packages sections appended and executed the build over the already provided root-tree as it happens with --allow-existing-root. Taking it one step further this could even be used to build a different type of image or build the same image but using dmsquash instead of overlay... The obvious problem here is how to define the XML merge and some details of the build steps (create only vs prepare and create). So some questions to figure out, but I believe it should be possible to find some simple and comprehensible procedure for those.

The discussion we can have here is if we want to weaken this concept if customers can provide a system root tree to work with ?

So, what if there is no need to weaken this concept, what if we enforce the imported rootfs to be a fully qualified KIWI rootfs, including the XML, config.sh, etc.

Something like that provides 100 ways to shoot yourself, of course, but you can already shoot yourself quite easily at the moment if you start using all the schema possibilities randomly.

This is probably a dumb idea, but I believe there should be a way to build images on top of another image and at the moment this is the only "sane" procedure that came into my mind :P

Any thoughts?

[cgwalters]

If it helps, iit's somewhere on the roadmap for me to better integrate (rpm-)ostree with container images, and make the experience support something much more like what you're talking about - a Dockerfile-style FROM fedoracoreos then do stuff. A lot of stuff involved there though.

[schaefi]

I like the idea to preserve a kiwi root tree in a registry. Correct me when wrong, the stored root tree would be a docker container and can also be used as such but its main use case is to preserve a root tree from a kiwi stage for further operations as you mentioned, right ?

I think this is a great concept to implement the "reproducable build feature" you had in mind earlier and allows for more use cases.

From a docker registry perspective the pushed data is probably not very useful as a container. Thus I was thinking if providing the tree as an ostree would fit in better. However I must say that using a docker registry and the handling of data as a docker container is pretty straight forward and easy compared to an rpm ostree. But this could also because of my lack of knowledge on the ostree end

But what if we also provide a delta of a new XML description?

I had played with this in the past and I must say it was very hard to achieve this on the XML level. A generic approach that allows kind of a XML merge is possible but ends in a cumbersome to use concept. The delta part will not validate the schema only the combined information will be valid. The model I came up then as an alternative was the profiled sections, but this doesn't play well on a delta concept. So the idea is nice but for a delta concept I suggest to use a meta language on top which can be composed into a complete XML description for validation. Similar to the C prepocessing

[davidcassany]

Closing the issue since we are definitely deviating from the original topic and no progress is expected. Also related ideas are discussed in #1771

[Conan-Kudo]

I was talking to @travier today in the Fedora KDE SIG meeting, and he said he'd be interested in helping us understand how to add RPM-OSTree support to KIWI.

[dcermak]

So, I have been playing around a teeny-tiny bit with ostree and unfortunately I am still far from really understanding it, but this is roughly how I got it:

1. ostree itself will build a ostree commit, which is effectively just a FS snapshot that ostree can grab and deploy on an existing system
2. you setup a http server and serve this ostree commit to your running system, point ostree to that webserver and run a ostree upgrade
3. ostree will apply that commit and make it available after a reboot

What I have not found out yet, is how you create a disk/iso image from this.

However, an ostree commit is afaik just a tarball of the filesystem, so if we can create a VM/iso from a tarball, then we should be able to build a system with ostree enabled. But admittedly, this is all very hand-wavy at the moment.

[Conan-Kudo]

This code from osbuild shows how they generate an rpm-ostree commit, and that might be useful for coming up with a way to do it in kiwi.

[nkadel]

Relying on deep kernel functionality to introduce unncessary "immutable architecture" to a build environment just seems painful. Almost everything rpm-ostree provides can be done with distinct directories with hardlinks between them, a much older and more stable technology with no requirement for introducing potentially fragile new technologies. If it's necessary to make the components of such an environmentimmutable, use "chattr". But don't hold up desirable image building tools on complex technologies which don't actually help perform the task.

[Conan-Kudo]

That is pretty much how rpm-ostree works, actually. It's heavily managed hardlink/reflink farm.

[cgwalters]

Yep, though I'd clarify rpm-ostree uses ostree, which is what does all that. ostree is really a very overgrown wrapper for the link() system call along with the ability to fetch via http.

[Conan-Kudo]

FYI: unfortunately, RPM-OSTree is now fundamentally incompatible with openSUSE, so I have stopped attempting to make it available for the distribution. Since openSUSE now uses /usr/etc, it is not compatible with how rpm-ostree uses it. Development of the RPM-OSTree backend cannot be done on openSUSE at this time.

[ericcurtin]

If anyone is looking for a reference implementation, the Automotive variant is basically boiled down to a one liner:

https://sig.centos.org/automotive/building/

git clone https://gitlab.com/CentOS/automotive/sample-images.git
sudo sample-images/auto-image-builder.sh cs9-qemu-minimal-ostree.x86_64.qcow2

If you grep the output:

$ grep ^org.osbui.*ostree ~/cs9-ridesx4-minimal-ostree.aarch64.aboot.simg1711378035.txt
org.osbuild.ostree.genkey: 039b0760153259cd15a5c28ded1f190fbf6e828f76ff9925a4a205ebf7a9dfb6 {
org.osbuild.ostree.preptree: 865c268aca38255b56d4dc7726297c6db8f4f837e0cd37b176a442bd7fef4a07 {
org.osbuild.ostree.init: 8704ff9ae85760c85475f06fe26683a9d518ea103699b1ac0462b978bba28703 {
org.osbuild.ostree.commit: 3236b1465f7e3ec24618a6350532276fe055419033c9d4a9fb77b93811a91e75 {
org.osbuild.ostree.sign: bef42772d505ad28285e67fae9796a69f904e9a83f95bceb172cbe0c7aaf8bd6 {
org.osbuild.ostree.init-fs: baf0238cd955d4ebff78feba4360da63cd0a3c0e6a6c284163eaeab2db9888fd {}
org.osbuild.ostree.pull: d4c638e615651f975bbe62a3395f3aa5dda4ab099db33df7fd71c4ebfadb7a09 {
org.osbuild.ostree.os-init: 77bb8ccfc5c1959ed4b424cbb31a4f13d6253077b57254fbb9ea15ddcf8e493f {
org.osbuild.ostree.config: 90430db80ed476ca8a8c709822908dd5f78762e144b8a1db3b58505e1980d583 {
org.osbuild.experimental.ostree.config: 9a12f564fc854ed001e98aaa69eaa83579d917dcb10f08b3725442b6229645b2 {
org.osbuild.ostree.remotes: 675cff32c80db8f17a54a254f94ec317eabb814cac2ce3c06bf356011dec6d74 {
org.osbuild.ostree.deploy: db3f60947577dd6c09b7f496f2b69e97c4d599aa297a1e32cfa2edcf10e6213b {
org.osbuild.ostree.fillvar: 5f63f87ccfe85cf26e22980ad6b907092b18891fc3e02c7aa8917baf93a410d8 {
org.osbuild.ostree.selinux: 25e239b11a1d3b1e1c3b09f486a78deb7a379d8dfbb0fb9b118fd8a43b4e27e8 {

you should be able to see the general flow.

These python scripts that power this are around here basically:

/usr/lib/osbuild/stages/org.osbuild.ostree.preptree

[cgwalters]

Since openSUSE now uses /usr/etc

Link/more info?

[Conan-Kudo]

Since openSUSE now uses /usr/etc

Link/more info?

openSUSE now uses /usr/etc to store vendor configs of stuff that is in /etc, and my understanding of the situation is that this is the location RPM-OSTree uses to divert and archive pristine copies of /etc. At least, I think that's what you told me when you removed /usr/etc from Fedora's filesystem package years ago.

Since RPM-OSTree does not use a private hierarchy for this, it effectively has become incompatible with openSUSE (not that there weren't already other issues with its usage of %verifyscript and a few other things, but this is the final nail in the coffin).

[travier]

I've been reading on how kiwi works and I think that we should do the following:

• Add a prepare step that takes a ostree native container image and deploys it to a sysroot folder.
  • https://osinside.github.io/kiwi/concept_and_workflow.html#the-prepare-step
  • It will be an "alternative" to the package-based options that are in kiwi right now
• Then figure out if the existing create step works for the image types that we care about:
  • https://osinside.github.io/kiwi/image_types_and_results.html
  • Ideally for ISO Hybrid Live Image and Virtual Disk Image

We will add support for building ostree native containers to kiwi later. We will also need to figure out where bootupd fits in there but we can do that later as well.

Let me know what you think.

[Conan-Kudo]

This seems like a reasonable plan. So then the end state we should be able to create live media and disk images that are RPM-OSTree backed?

[schaefi]

I was wondering if the kiwi stackbuild plugin could already be helpful here. See the following documentation

• https://osinside.github.io/kiwi/plugins/stackbuild.html#turn-a-container-into-a-vm-image

Let's say I take this and replace the example container with an ostree native container I should be able to produce an RPM-OSTree backed image. I would be eager to try that out. Where can I find ostree containers ?

Thanks

[travier]

So then the end state we should be able to create live media and disk images that are RPM-OSTree backed?

Ideally yes.

I was wondering if the kiwi stackbuild plugin could already be helpful here

I don't think this will work. Ostree needs special handling to deploy a version to a system rootfs.

Overall, we'll have to duplicate the logic from https://github.com/coreos/coreos-assembler/blob/main/src/create_disk.sh#L312..L379 (https://manpages.debian.org/testing/ostree/ostree-admin-deploy.1.en.html).

[travier]

I would be eager to try that out. Where can I find ostree containers ?

https://quay.io/repository/fedora-ostree-desktops/kinoite?tab=tags

[schaefi]

I don't think this will work. Ostree needs special handling to deploy a version to a system rootfs.

yeah I see and that matches my results. So here is what I did

• install the kiwi-stackbuild-plugin from pip or the package as documented here: https://osinside.github.io/kiwi/plugins/stackbuild.html
• mkdir ostree-to-live
• Add the relevant image description to build a live image from an ostree container, vi ostree-to-live/config.kiwi

<?xml version="1.0" encoding="utf-8"?>

<image schemaversion="8.0" name="Rawhide-Live">
    <description type="system">
        <author>The Author</author>
        <contact>user@example.org</contact>
        <specification>
            Live Image Build OS-tree backed using rawhide
        </specification>
    </description>
    <preferences>
        <type image="iso" flags="dmsquash" firmware="uefi" hybridpersistent_filesystem="ext4" hybridpersistent="true"/>
        <version>1.42.1</version>
        <packagemanager>dnf4</packagemanager>
        <locale>en_US</locale>
        <keytable>us</keytable>
        <timezone>UTC</timezone>
    </preferences>
    <repository type="rpm-md" alias="Rawhide" sourcetype="metalink">
        <source path="https://mirrors.fedoraproject.org/metalink?repo=fedora-rawhide&amp;arch=x86_64"/>
    </repository>
    <packages type="bootstrap">
        <package name="grub2"/>
        <package name="grubby"/>
        <package name="kernel"/>
        <package name="grub2-efi-x64-cdboot"/>
        <package name="dracut-live"/>
    </packages>
    <users>
        <user password="$1$wYJUgpM5$RXMMeASDc035eX.NbYWFl0" home="/root" name="root" groups="root"/>
    </users>
</image>

• build the image selecting an ostree rawhide container

sudo kiwi-ng system stackbuild --stash kinoite:rawhide --from-registry quay.io/fedora-ostree-desktops --target-dir /tmp/my-ostree-live --description ostree_to_live

This works and builds a live image, but it's huge :)

I was looking into the rootfs and I found the specials when using ostree.

total 24
dr-xr-xr-x.   2 root root    6 Mar 13 01:00 afs
lrwxrwxrwx.   1 root root    7 Mar 13 01:00 bin -> usr/bin
dr-xr-xr-x.   5 root root 4096 May 14 17:47 boot
drwxr-xr-x    3 root root   92 May 14 17:49 dev
drwxr-xr-x. 136 root root 8192 May 14 17:49 etc
drwxr-xr-x.   2 root root    6 Mar 13 01:00 home
drwxr-xr-x.   3 root root   38 May 14 17:49 image
lrwxrwxrwx.   1 root root    7 Mar 13 01:00 lib -> usr/lib
lrwxrwxrwx.   1 root root    9 Mar 13 01:00 lib64 -> usr/lib64
drwxr-xr-x.   2 root root    6 Mar 13 01:00 media
drwxr-xr-x.   2 root root    6 Mar 13 01:00 mnt
drwxr-xr-x.   2 root root    6 Mar 13 01:00 opt
lrwxrwxrwx.   2 root root   14 Jan  1  1970 ostree -> sysroot/ostree
drwxr-xr-x    2 root root    6 May 14 17:46 proc
dr-xr-x---.   3 root root   18 May 14 17:47 root
drwxr-xr-x.  34 root root 4096 May 14 17:47 run
lrwxrwxrwx.   1 root root    8 Mar 13 01:00 sbin -> usr/sbin
drwxr-xr-x.   2 root root    6 Mar 13 01:00 srv
drwxr-xr-x    2 root root    6 May 14 17:46 sys
drwxr-xr-x.   3 root root   20 Jan  1  1970 sysroot
drwxrwxrwt.   2 root root    6 May 14 17:49 tmp
drwxr-xr-x.  12 root root  144 May 14 17:47 usr
drwxr-xr-x.  23 root root 4096 May 14 17:47 var

in /sysroot/ostree I could see all the different objects that makes up that ostree.

At this point we need to add the deploy functionality that you pointed me to, but thanks for sharing the containers and the link, I understood a little bit more on the concept :)