sysupdate: Support major OS upgrades

[AdrianVovk]

Many distributions have multiple update streams. For instance, Fedora has:

• Fedora 42, the next upcoming stable release. This stream will first contain the Fedora 42 beta, but then eventually will have the stable releases
• Fedora 41, the current stable release
• Fedora 40, the old stable release, which continues to receive security updates until Fedora 42 is released
• Fedora Rawhide, the unstable rolling stream, which receives updates all the time

This PR makes sysupdate aware of this functionality, so that this situation can be represented. Update streams are defined by the OS vendor by creating /usr/lib/sysupdate@<stream>.d/ with transfer and optional feature definitions. You can switch streams via systemd-sysupdate --stream=<stream>

Switching the stream replaces the /usr partition, and thus will bring new stream definitions. Effectively, you can think of /usr/lib/sysupdate.d/ as the "current" stream (which is anonymous as far as the system is concerned), and all of /usr/lib/sysupdate@*.d/ as streams you can switch to. Thus: each /usr partition in each stream should define available streams to switch to. This gives the distributor lots of flexibility. For instance, let's say you're on Fedora 40 (the f40 stream) and switch to rawhide, which will effectively upgrade the system past the Fedora 41 branchpoint. The rawhide version of /usr will not contain f40 nor f41 streams, but will contain an f42 stream. Users can switch to the f42 stream to recieve updates for what will eventually become Fedora 42, but they cannot downgrade to Fedora 41 or 40. Once f42 branches off of rawhide, then the stream can be removed from rawhide, removing the upgrade path.

Streams have strange config load paths. They're loaded from /usr/lib/sysupdate@<stream>.d/, but then administrator overrides come from /etc/sysupdate.d/. This seems strange, but it's implemented like this to prevent footguns. Basically, if you use the transfers in /usr/lib/sysupdate@<stream>.d., then once the update is successful those transfers move to /usr/lib/sysupdate.d/. This means that the administrator overrides would be inactive while switching streams, but then activate themselves once that's done. Ultimately, this is a footgun if the administrator overrides cause problems: switching to a different stream might work fine, but the moment that's done updates may break.

Streams are also loaded from /var/cache/systemd/sysupdate@<stream>.d/. This allows for stream definitions to be dynamically downloaded from a server, instead of hard-coding them into /usr. This allows new streams to be pushed to existing installations without requiring a full OS upgrade.

This will be hooked into sysupdated and updatectl in subsequent PRs.

Partial fix of #33345

[poettering]

we typically suffix dirs with / in our docs. i.e. /usr → /usr/ and /etc → /etc/. It conveys a tiny bit of additional information to readers.

[poettering]

i am fine with warning, but I find this a bit too strong. i.e. the "extreme" and stuff is a bit too much.

What about this: let's use docbook's native constructs for this. i.e. <caution><title>Warning!</title><para>…</para></caution>

And then remove the <emphasis>extreme</emphasize> and the `You have been warned.

or in otherwords, let the typesetter solve this problem, not the text contents so much.

By toning this a bit down but at the same time offsetting this a bit from the rest of the text I think we get the message across quite well and appear a bit more professional I guess ;-)

[poettering]

the commit msg has some "SQUASHME" thing in it

[poettering]

maybe call this "release streams" instead of just "streams"?

[poettering]

i figure there are a bunch of </para><para> missing here.

[poettering]

as mentioned elsewhere, please suffix the path with /

[poettering]

cache? uh?

[AdrianVovk]

Yeah. It's something that needs to be cleaned up and go away after the OS update. It's not supposed to be persistent. Frankly, maybe I need to build in the IMAGE_VERSION into the path to make sure that it's not considered anymore after an upgrade. So something like /var/cache/$IMAGE_VERSION/systemd/sysudpate@*.d/, and then we can have a tmpfile rule that cleans those up based on age.

[poettering]

hmm, i am not too much of a fan of this model tbh.

hmm, wouldn#t it be nicer if we'd model this much closer to the features model. i.e.:

1. always have all .transfer files for all streams in the same sysupdate.d/ dir, but the transfers for the different streams have different names.
2. introduce a new .stream file that is very similar to .feature
3. however, the difference is that you only pick one stream, but multiple features.
4. one stream would be special: "default.stream" which is what we'd use by default. But it could be symlinked onto something else to switch default stream.
5. .transfer files would gain a Stream= setting where you list one or more streams a transfer belongs to

[AdrianVovk]

hmm, wouldn#t it be nicer if we'd model this much closer to the features model. i.e.:

1. always have all .transfer files for all streams in the same sysupdate.d/ dir, but the transfers for the different streams have different names.

The whole point of having separate release streams is that you have distinct transfer definitions for each stream. Different update URLs. Possibly different servers. Different delta updates (nightlies might not have deltas). And so on.

A stream is a different place you can get different updates from. So in practice, there will be no sharing of transfers between streams.

So ultimately you're asking that this:

/usr/lib/sysupdate.d/
    10-usr.transfer
    11-usr-verity.transfer
    50-devel-sysext.transfer
    90-uki.transfer
    devel.feature

/usr/lib/sysupdate@nightly.d/
    10-usr.transfer
    11-usr-verity.transfer
    50-devel-sysext.transfer
    51-devel-addon.transfer
    90-uki.transfer
    devel.feature

turns into this:

/usr/lib/sysupdate.d/
    10-nightly-usr.transfer
    10-stable-usr.transfer
    11-nightly-usr-verity.transfer
    11-stable-usr-verity.transfer
    50-nightly-devel-sysext.transfer
    50-stable-devel-sysext.transfer
    51-nightly-devel-addon.transfer
    90-nightly-uki.transfer
    90-stable-uki.transfer
    default.stream -> stable.stream
    devel.feature
    nightly.stream
    stable.stream

To me that's a lot more complicated to understand, and for unclear benefit.

Another wrench in the works: features will vary between streams too! They may appear or they may disappear between updates. I'm not sure of the best way to represent that. Maybe a feature that has no transfer definitions (for a given stream) just doesn't exist on that stream? Or the features themselves need a Stream= setting? Again, more complexity for unclear benefit.

And one more wrench in the works: how are we supposed to handle streams for sysupdate components? Components don't have their own streams; they use the host's streams (since the whole component definition is replaced when you switch to a different stream). Defining streams inside of the sysupdate.d/ directory opens that can of worms, because it enables you to define streams in sysupdate.$component.d/. I guess the solution is to just bail if the component has stream definitions? Why not just make the situation un-representable in the first place?

2. introduce a new .stream file that is very similar to .feature

I think I like the idea of defining a .stream file, but not necessarily for the purposes of putting all streams together into a single directory. There is some metadata that we've shoehorned into transfer definitions that really doesn't belong there. Specifically: appstream metadata, and the changelog.

Right now, we put that into the transfers because we had nowhere else. But if we define these .stream files, we could put that metadata in there. Then we'll always return just one appstream URL or just one changelog URL via the DBus API. We'd just have a "one-.stream-file-per-directory" rule.

An alternate idea:

Maybe we can also let .stream files carry the defaults for the [Source] section of .transfer files? For instance, the .stream file would define the Path= that all transfers share, and each transfer would just have to define MatchNames=. In other words: if a .transfer file doesn't define something under its [Source] section, then sysupdate would go check the stream's [Source] section for the default value. This way, the .stream file defines where you get updates from and the .transfer files define what gets updated (which files to match on the server, and where to put them on the system).

This would make me a lot more open to the single-directory idea, because it would enable us to share all the .transfer files between all of the streams. This is kinda similar to what I was hoping to implement over on the packaging side: I was going to have a single shared collection of .transfer files with templating in them, and then each stream would be a collection of values to place in there. Putting the functionality directly into sysupdate removes the templating step.

However it doesn't solve all problems. For one, it doesn't solve what to do with features, nor does it solve the components issue. It also introduces its own problems: it wouldn't allow different streams to have different sets of .transfer files (maybe some streams drop additional files next to the UKI on the ESP, but others don't!). Transfers don't necessarily all pull from the network either (maybe you have a mix of url-file and local-file transfers, though I can't think of a real use-case for this) - how do we handle that?

Ultimately, I think having individual directories per stream is most flexible. GNOME OS might not need quite so much flexibility and we could get away without it, but others might not have it so lucky.

3. however, the difference is that you only pick one stream, but multiple features.
4. one stream would be special: "default.stream" which is what we'd use by default. But it could be symlinked onto something else to switch default stream.

I don't think we should allow redefining which stream default.stream is pointing to. Allowing this makes switching streams non-atomic: first you have to update the symlink in /etc, and then you need to actually apply the update. Things can go bad in between there.

It's also possible for streams to be empty; the existence of a stream doesn't imply that there are updates available there. We could release GNOME OS 47 with a GNOME OS 48 stream already baked in, before any GNOME OS 48 release has been made. If the user tries tries to switch to the 48 stream, they'll first have to redefine default.stream to 48.stream. Since there are no updates in that stream, they're now always up-to-date. This user won't receive any GNOME OS 47 security patches!

If, on top of that, the stream definition is broken and unable to actually find or apply the GNOME 48 updates, this user will never get the patch to fix the broken definitions, and they'll never get any more updates on this system. They'd have to know to redirect default.stream back to 47.stream, then apply a bunch of updates, then try again.

Ultimately, switching streams switches the contents of /usr, and thus replaces /usr/lib/sysupdate.d/default.stream; so the active stream is always the one in /usr and that's what we must rely on to make this atomic.

The one situation I can think of where a symlink makes sense is when two different streams are actually the exact same stream, and the user wants to pick which one they'd like to keep using if the streams diverge. Let's say that we're on GNOME OS Nightly, which currently is a development snapshot of GNOME OS 48. Right now, the 48 stream and the Nightly stream are one in the same. But eventually, GNOME 48 will branch from GNOME Nightly, and the stream definitions will diverge. At that point, the symlink can be used to decide which one the user intended to stay on: 48 or nightly. I don't know if supporting this use-case is worth the footguns the symlink introduces...

5. .transfer files would gain a Stream= setting where you list one or more streams a transfer belongs to

Ultimately I think this is what I dislike the most about the single-directory idea. I don't mind a single directory if transfers take some of their settings from the stream definition and all streams share the same transfers. But if we're going to have stream-specific transfer definitions I see no reason to put them all into one directory.