pluggable backends for systemd-sysupdate

[abderrahim]

Component

No response

Is your feature request related to a problem? Please describe

Currently systemd-sysupdate downloads a whole file (or tarball) and installs it. This is however wasteful to download a whole image on every update.

Describe the solution you'd like

A few options exist that can more efficiently download files and images. Some options include:

• ostree (to download into a directory)
• casync (and its go rewrite desync)
• jigdo
• Some custom backend implementing RAUC's block-hash-index

Instead of having a blessed backend for systemd-sysupdate (and having to argue which one of the above is objectively better), we want to support pluggable backends instead.

These backend could implement the same interface as systemd-pull so that they can be used as a drop-in replacement from sysupdate's view. Another possibility is to define a simpler interface and let systemd-pull call the backend and provide additional features.

The idea is to have a simple enough interface that can be implemented by external tools (such as casync) directly, or by writing a small wrapper around the external tool.

Describe alternatives you've considered

No response

The systemd version you checked that didn't have the feature you are asking for

256

[obbardc]

another one to add to the options list could be zchunk. swupdate makes use of it for the same reasons in the op and it's quite effective.

[abderrahim]

A couple of weeks ago, I discussed with @poettering a possible implementation plan. Here is a summary of our discussion.

• Each backend would be one executable file.
  • This could be a shell script wrapper around the actual tool to implement the systemd-sysupdate interface.
• We should use essentially the same interface as systemd-{import,pull}
  • The command line parameters should be similar to systemd-pull
  • Progress reporting should happen via sd-notify
    • systemd-{import,pull} currently sends its progress as X_IMPORT_PROGRESS, we might want to define some standard progress message in sd_notify instead.
• Any extra info that is needed by the tool should be passed as environment variables
  • This includes currently installed versions of the transfer target
  • Some tools will require additional (tool specific) configuration to be passed

There are still a few (minor) open questions:

• How should we tell systemd-sysupdate to use a specific tool?
  • One idea is the url scheme: use casync+https://... to tell systemd-sysupdate to use casync (maybe an executable named systemd-import-casync?)
  • Would it be better to have a separate config option for what tool to use?
• How to provide extra configuration to the tool
  • We can add a new option to the transfer definition file to provide extra environment variables.
• Should the tool be called at vacuum time to clean up after itself?
  • Could be useful for some tools to keep some data around.
• What if a tool doesn't implement all the features of systemd-{import,pull}(sync, btrfs subvolumes, offset/max-size) ?
  • Ultimately this is the tool's problem, but it can pipe its output to systemd-import

[AdrianVovk]

So at All Systems Go! this year, I spent a while talking to @jluebbe and here's an architecture we came up with:

RAUC is an updater popular in the embedded world. It has this fancy HTTP Streaming feature. TL;DR:

• The server must support HTTP range requests
• On the server side, when producing an update you take the finished bundle and append some metadata and a dm-verity tree
• The client downloads just the metadata (which contains the dm-verity roothash), then sets up a network block device that only fetches blocks as they're read (using range-requests). The dm-verity tree is set up similarly, and then the whole thing is activated with cryptsetup. This gives you one partition with some very nice properties: you only download what's absolutely necessary, and all data has its checksums verified before it's written to disk.

So what we came up with:

• Rauc learns to shove a whole DDI into their bundle format - so we can ship multiple partitions at once
• systemd-sysupdate learns to copy relevant partitions out of a block device. So basically given a block device with an update from anywhere, sysupdate could pick it up and then pull an update from it.
  • The primary purpose for this is to mount the DDI using Rauc's NBD support, then pull updates out of there
  • The secondary purpose is to also support things like pulling updates out of a USB device w/ an update bundle dd'd onto it.

However, there are some remaining questions:

• How does sysupdate communicate with Rauc to set up this NBD device?
  • This needs to happen before sysupdate starts
  • Research question: how does RAUC do this? Answer: Seems like it doesn't! Rauc is just given an image to install from and it does the installing
• In the USB stick scenario, how would sysupdate enumerate updates? It would somehow need to know that a stick was plugged in, check the contents of the USB stick, and pull the update from it.
• Another thought that struck me later: HTTP Streaming isn't actually enough to implement adaptive updates. You need an additional step of generating and comparing the block-hash indices, etc. And this isn't something that we can kinda-sorta pull out of rauc like HTTP Streaming

Anyways, I have some ideas how to address some of these things, that will come in a follow-up comment

[jluebbe]

• Rauc learns to shove a whole DDI into their bundle format - so we can ship multiple partitions at once

On the RAUC side, this would need:

• a new bundle format ("raw", "ddi" or something similar), which auto-generates a minimal manifest (without images)
• a new command, similar to rauc mount, which only does the streaming and NBD setup

[shocklateboy92]

Can one of those backends be libtorrent (or another torrent library)?

that would give us chunking (downloading only deltas), parallel downloads, automatic mirror selection, error checking, and a whole host of other features for free.

[AdrianVovk]

So basically, we want to achieve RAUC-like adaptive updates + HTTP streaming but with semantics that fit systemd-sysupdate. I'd also like to avoid needing to generate special bundles - we could put the metadata next to the file.

The RAUC solution is incredibly clever and perfect for something super flexible like RAUC. You can distribute a squashfs with a bunch of scripts, indexes, maybe even packages, and loads of other content, and use portions of it. However I wonder if it's a little too clever for our use-case where we always want to stream a resource from the network directly to disk.

Separately, I'd like to minimize duplication. In the RAUC case, the dm-verity built into the bundle makes lots of sense: again the bundle contains lots of content that doesn't necessarily end up on disk. In our case, this isn't true: ultimately the entire content of the bundle ends up on the disk, every time. In our case, then, the dm-verity of the bundle is actually completely redundant, because we'd need a separate block-hash-index that already includes hashes of each chunk.

So, taking inspiration from RAUC's technologies, here's what a solution could look like for sysupdate:

• For each file you publish on the server, generate a hash-index file. This is what's listed in SHA256SUMS, and it's ultimately what sysupdate passes to the download backend.
  • The hash-index file also contains a reference to the data file you'll be pulling from.
  • The hash-index file is downloaded in full, since the entire file is necessary to apply an update anyways. So we can get away with a simple checksum of the file, instead of a merkle tree.
  • This solves the enumeration question: the hash-index file is what gets enumerated and "downloaded", similarly to how sysupdate works today
• The download-backend would be a custom plugin, exactly as described in this issue. It would be relatively simple, as in would require no coordination or pre-setup via NBD - instead it would act kinda just like curl. It's given:
  • The checksum of the hash-index file (from SHA256SUMS)
  • The URL of the hash-index file (to be downloaded by the download backend)
  • The URL of the data file (in the hash-index file, or just derived from it. So https://example.com/foo.raw.idx -> https://example.com/foo.raw)
  • A list of available instances, including the instance it's about to overwrite
  • The hash-index file of each available instance, either cached on disk or re-computed on-the-fly
  • The destination file/block/folder to overwrite
• The download backend is relatively simple, requiring no setup or coordination (via something like NBD or whatever). It can just go through block-by-block from the hash-index file, and either download the block via HTTP range request from the data file, or copy the block from one of the available instances. The block is verified against the expected hash, and then written to disk.

Regarding my mention above of hash-indices being cached: sysupdate would need to provide instance-specific cache dirs for the backend, so that simultaneously the backend can cache the hash-indices of existing instances (to avoid recomputing each time), but sysupdate can still vacuum up these cache dirs when necessary. So, for instance: the cache dir could be: /var/cache/sysupdate/$TARGET-instance-cache/$DEFINITION/$VERSION/. The download backend would be given this directory to work with, and it could cache the hash-indices there to avoid recomputing each time. When the instance is vacuumed, sysupdate would delete the cache dir without the involvement of the download backend. This could also be a place the default curl download backend stores things like etags and in-progress download information so that an interrupted download could be resumed later (without much loss if the cache were to accidentally disappear)

As far as I can tell, this scheme I describe here provides the benefits of RAUC's adaptive updates + HTTP streaming, but fits into sysupdate's model and into this pulggable backends proposal.

[AdrianVovk]

Can one of those backends be libtorrent (or another torrent library)?

I don't see why not. Your server would need to act as a torrent seed, though, not just a basic HTTP server. Which means you can't put it behind a CDN, unless you've got lots of seedboxes 🤷.

Also, slight concern is users seeding update downloads for other users - would they be comfortable with that, and how would that be done? Probably out-of-scope for sysupdate.

libtorrent seems to have an HTTP backend, where it treats an HTTP server as a big seedbox. Which seems great. However, it seems like it doesn't use HTTP range requests? Maybe I'm misunderstanding the docs...

It describes another feature: it'll sort and combine chunks in a way that minimizez the number of requests. That's a good strategy we should copy.

[AdrianVovk]

re: USB. Details are still a bit unsolved. We could maybe let sysupdate check multiple URLs in some order? The USB stick would contain what's essentially a RAUC bundle with the contents of an update server on it (a SHA256SUMS file, plus the necessary data files, plus the hash-index files). A udev rule could mount this USB stick upon insertion to some special dir, like /run/sysupdate/offline-update or similar. Then sysupdate could be configured look for updates at multiple places: file:///run/sysupdate/offline-uptate and https://updates.example.org/.

[AdrianVovk]

Returning to the concrete work that would need to happen in sysupdate specifically, I think this proposal is mostly what we need. Except, I would switch everything to Varlink, instead of trying to use CLI parsing and environment variables. Varlink is perfect for a job like this!

I don't mean to have a download-service with some socket listening somewhere. That's probably overkill for this use-case. I mean having executables in a directory, and sysupdate can just pick which one to run and pass in a socket for it to consume.

This is nice and extensible, and lets us pass in some more structured data to the update backend. Here's the example I explained above, for the proposed download backend:

{
    "version": "47",
    "mode": "raw/tar",
    "fsync": true,
    "checuksum": "<sha256 of hash-index file>",
    "source": "https://update.example.com/usr-47-<uuid of usr partition>.raw.idx",
    "destination": "/dev/disk/by-diskseq/1-part3",
    "cachedir": "/var/cache/sysupdate/host-instance-cache/50-usr/47/",
    "instances": [
        {
            "version": "46",
            "cachedir": "/var/cache/sysupdate/host-instance-cache/50-usr/46/",
            "location": "/dev/disk/by-diskseq/1-part5"
        },
        {
            "version": "45",
            "cachedir": "/var/cache/sysupdate/host-instance-cache/50-usr/45/",
            "location": "/dev/disk/by-diskseq/1-part3"
        }
    ]
}

Then instead of using sd-notify to pass back progress notifications, we'd just pass them back through Varlink. We've already got it, so why not (plus sd-notify isn't trivial to set up on the receiving end)

[AdrianVovk]

How should we tell systemd-sysupdate to use a specific tool? One idea is the url scheme: use casync+https://... to tell systemd-sysupdate to use casync (maybe an executable named systemd-import-casync?) Would it be better to have a separate config option for what tool to use?

I like the URL solution, because it's extensible to things outside of sysupdate too. Imagine systemd-importd using it one day too. Of course, that feels less important for the delta-updates usecase. But for the libtorrent usecase, it might be nice to be able to importctl pull-raw torrent+magnet://<magnet link here>.

Notably, I think it makes sense for the backend to be protocol-agnostic on the URL part. Basically everything after the + would get passed into the backend. That way, a url-file with foo+bar://whatever URL would tell the foo import backend to fetch bar://whatever. Again, this makes a lot of sense for the torrent usecase.

How to provide extra configuration to the tool? We can add a new option to the transfer definition file to provide extra environment variables.

Would be nice to have some concrete ideas of what these custom options would look like before we decide anything about them. I can't think of a great usecase for custom options.

BUT: If we're using varlink, we could pass the options directly as command-line options rather than via an env var. Since the cli options wouldn't be used for anything else in the Varlink case, we can use them as properly intended.

Should the tool be called at vacuum time to clean up after itself? Could be useful for some tools to keep some data around.

Seems like this is fixed with my cache dir idea. sysupdate would just clean up left-over cache dirs whenever it vacuums.

What if a tool doesn't implement all the features of systemd-{import,pull}(sync, btrfs subvolumes, offset/max-size)? Ultimately this is the tool's problem, but it can pipe its output to systemd-import

Yeah I agree it's the tool's problem. But maybe we can version the varlink interface, or have a method to list supported features. Then we can pre-emptively fail if the backend doesn't support the feature we're asking for.

Or, alternatively, the backend would just bail if it recieves an unknown/unsupported field in the Varlink request. So for example let's say we introduce an offset field that sysupdate needs now; whenever that's used the backend would recieve an unknown field in the method call and it would error out.

In other words: I think Varlink fixes this for us.

[mcassaniti]

This sounds quite interesting. Can you provide some detail on how this would work to do delta updates? Some sort of overview of what calls/executes what and some details on file contents on the backend along with when they are parsed would be great.

[AdrianVovk]

This sounds quite interesting. Can you provide some detail on how this would work to do delta updates? Some sort of overview of what calls/executes what and some details on file contents on the backend along with when they are parsed would be great.

I'm not entirely certain how to correctly interpret the question, so I'll interpret as "how would pluggable backends be used to implement traditional diff-based delta updates with systemd-sysupdate?". Please let me know if that's incorrect.

Basically the scheme is up to you. It's whatever you want to put into the download backend. sysupdate scans SHA256SUMS for some file and its hash, and then gives that to the download backend (along with a download destination, and a working directory for caching some data). What the backend does is up to the backend. sysupdate wouldn't be doing any parsing of the delta updates, which I think is part of the benefit here.

If I were implementing a delta-updater, though, I'd do something simple like:

• you'd change the download URL from https://dl.example.com/usr_@v_@u.raw to delta+https://dl.example.com/usr_@v_@u.raw in the transfer definition file
• There would be a sysupdate download backend named delta that gets invoked to handle the download
• The backend is started and then receives the following request:

{
    "version": "123",
    "mode": "raw",
    "fsync": true,
    "checuksum": "<sha256 checksum of usr_123_<uuid>.raw>",
    "source": "https://dl.example.com/usr-123-<uuid of usr partition>.raw",
    "destination": "/dev/disk/by-diskseq/1-part3",
    "cachedir": "/var/cache/sysupdate/host-instance-cache/50-usr/123/",
    "instances": [
        {
            "version": "122",
            "cachedir": "/var/cache/sysupdate/host-instance-cache/50-usr/122/",
            "location": "/dev/disk/by-diskseq/1-part5"
        },
        {
            "version": "121",
            "cachedir": "/var/cache/sysupdate/host-instance-cache/50-usr/121/",
            "location": "/dev/disk/by-diskseq/1-part3"
        }
    ]
}

• To find the delta blob, the download backend tries to append .$VERSION-delta (or whatever) for each version it can use as a delta. sysupdate told the backend that it has v122 and v121 available. Let's assume that the user is currently booted into v122. So the backend can safely read from that version (Careful! If the system was booted into v121 and the backend tries to apply a v122 delta, depending on the patch algorithm it might be reading and writing from the same partition. Which might lead to corrupted updates). Having decided that v122 is OK, the download backend downloads https://dl.example.com/usr_123_<uuid>.raw.122-delta, possibly into the /var/cache directory if appropriate (again, depends on usecase)
• The file format can be whatever is appropriate for the downloader. bsdiff, zstd patches, whatever. I would add a header containing a checksum of the binary patch, though
• The backend verifies the integrity of the downloaded binary patch using the prepended checksum. And then it combines the patch with the image of v122 to create an image of v123. The downloader writes this to the requested destination
• The backend checksums the destination to make sure that the patch applies correctly, and the hash matches. So the correct contents are now in the expected location.
• The backend exits successfully.
• sysupdate continues, eventually marking the partition as containing v123

Let's say that there's no .122-delta file on the remote. Well then maybe the downloader can fall back to .121-delta (if that's safe to do with the chosen binary diff algorithm), or failing that it can just directly download the requested file w/ no fancy delta downloading and call it a day.