live updates design

[cgwalters]

Background

Some people may wonder - why does rpm-ostree install require a reboot today? The answer is because all of the internal infrastructure is oriented around:

1. Being safe by default - power loss/kernel crashes won't result in a corrupted system
2. Supporting "offline" content pulls.

Diving into point 2) is interesting here. (Updated 2018-06-14) - Today one can systemctl enable rpm-ostree-automatic.timer with auto-updates, or simply script rpm-ostree upgrade as part of a cron job or other automation, and have confidence that it has no impact on the running system.

This "offline pull" model can greatly reduce total operational downtime versus a default "live apply" model as implemented by yum/apt etc. The reason is that any time one is doing a live update, it's a "sysadmin pagers at the ready" event. Whereas with rpm-ostree's offline capabilities, one can fearlessly set up a systemd timer/cron job to do pulls in the background, and only take a single reboot to have everything ready.

Of course, one can with some effort script yum/apt into just downloading the packages into /var/cache, but you still have downtime and potential instability when they're being applied.

Use cases

New package installs:

rpm-ostree install cowsay
rpm-ostree live-apply cowsay

Security updates - take just the updated openssl from the new update, and apply it live:

rpm-ostree live-apply openssl  # This *just* updates the filesystem
systemctl restart sshd NetworkManager  # This is an *admin* decision

We can obviously generalize this to support more than one package:

rpm-ostree live-apply 
systemctl restart foo bar baz

Let's think a little bit about what rpm-ostree status should show. There are some prior musings in #118

# rpm-ostree status
State: idle
Deployments:
 atomic-ws:atomicws/fedora/x86_64/continuous
             Version: 25.2017.41 (2017-02-22 08:32:22)
          BaseCommit: ee73b4cc419c26e0b32c3358e3bbde71709567188619732ce67a7fc0e92123a4
              Commit: bea41aaf5691eb368dcbced902509d75d4cce8da42ce3b184cf11c378f927aaa
              OSName: fedora
            Packages:  emacs fuse-sshfs gdb keepassx krb5-workstation powerline

● atomic-ws:atomicws/fedora/x86_64/continuous
             Version: 25.2017.40 (2017-02-22 02:32:22)
          BaseCommit: 1709567188619732ce67a7fc0e92123a4ee73b4cc419c26e0b32c3358e3bbde7
              Commit: 4cce8da42ce3b184cf11c378f927aaabea41aaf5691eb368dcbced902509d75d
              OSName: fedora
            Packages: emacs fuse-sshfs gdb keepassx krb5-workstation powerline
           LiveUpdates: openssl

And maybe LiveUpdates: all? This type of stuff is really going to want a smarter status display (something diff-based like I mused on a bit #295 ?)

What happens if one does e.g. rpm-ostree upgrade again before rebooting? Or for that matter rpm-ostree cleanup -p?

Also, do we use transient overlays like ostree admin unlock, or do we push a rollback, then mutate in place?

[cgwalters]

Now thinking of something like this:

● atomic-ws:atomicws/fedora/x86_64/continuous
             Version: 25.2017.40 (2017-02-22 02:32:22)
          BaseCommit: 1709567188619732ce67a7fc0e92123a4ee73b4cc419c26e0b32c3358e3bbde7
        BootedCommit: 4cce8da42ce3b184cf11c378f927aaabea41aaf5691eb368dcbced902509d75d
          LiveCommit: 1d532ea4e6ce1ac5f953ab82b47291fd65caf35fde5143f2b8e0eb05d9f97dd7
              OSName: fedora
            Packages: emacs fuse-sshfs gdb keepassx krb5-workstation powerline
        LivePackages: openssh-server-7.4p1-3.fc25.x86_64

Where Commit → BootedCommit, and LiveCommit is the result of us computing a new commit based on what the user explicitly requested cherry-picking from the pending deployment.

Going back to the example case where we have a pending base update with new kernel, NetworkManager, and a security update for openssh-server, and we just want to cherry-pick the latter via rpm-ostree livefs openssh-server, we actually do a new layer with just that package.

The WIP I have for rpm-ostree livefs is oriented around the filesystem right now, but the next step I think is to make it package based by default per this design, with the filesystem bits being safety checks.

Something to solve as well is - what happens with (and how do we display) the result of multiple livefs applications?

[vtolstov]

Nice feature, does it possible to live update current tree ? For example i have booted system with some services running, now i know that my new composed tree have updates to libvirt, docker and other stuff. Kernel can be updated via kpatch (i'm working on automatic building packages with patches for kernel). But software can be updated only via ostree admin unlock and install needed packages.....

I want feature that can overlay on current composed tree new tree (optionally reload services to get updates)

[cgwalters]

Hi @vtolstov, indeed that's the goal, and what's supported by the initial WIP pull request.

[vtolstov]

Ok, i'm happy to test it =)

[cgwalters]

The case for /usr is pretty easy. We'll need special handling for everything else, which is:

• /etc: Do the 3 way merge; need to lift the ostree logic into a public API?
• /var: Run systemd-tmpfiles --prefix=/var ?
• /boot: Notice that we're installing a kernel, and warn about this

[rdeusser]

IMHO, any install/update to a client system should look at the current version of the tree and the old version of the tree and "live-update" the differences. So say I update openssh server side. When clients get that new tree, rpm-ostree should compare the new tree with the currently running tree and compare the differences (think git diff). At this point, rpm-ostree should see that the only thing that changed is something in openssh and so it essentially performs a git merge with the currently running tree bringing the client in sync with what's in the server without requiring a reboot due to the fact that nothing critical needs to be updated (so kernel or init). If something critical like the kernel (assuming you're not using kpatch or other) needs updated, rpm-ostree should be able to discern that the system will require a reboot and perform a rpm-ostree pull so it could be rebooted at a later date (maybe an app deploy is being rolled out and now isn't a good time to reboot a server). I just started looking into using rpm-ostree so I'm no expert, but speaking as someone who would deploy rpm-ostree in a large infrastructure this is how I believe it should work if it's possible.

[cgwalters]

Hi @iamthemuffinman - Yep, the work-in-progress here does something like that indeed.

[rdeusser]

@cgwalters Good to hear! Thanks for the update!

[jlebon]

There are details we would need to resolve to do this, e.g.:

• /usr: we can't just mount the deployment's /usr since it can get wiped, so we'd have to checkout elsewhere and mount that -- there's also the issue of repeated livefs applications causing remounts

Hmm, could we avoid the "stacking mounts" problem by just modifying the checkout itself, so e.g.:

$ ostree checkout $pending/usr /ostree/repo/tmp/livefs-overlay
$ do the `mount --bind` dance to get it r-o on /usr
$ directly modify /ostree/repo/tmp/livefs-overlay for repeated applications

?

Then we don't have to teach cleanup code to avoid the pending deployment.

I don't think so, because of the kernel case.

Heh, right, that's understood. Just like you're still using the older glibc until you reboot.

But down that path leads to less flexibility. IMO, it should be possible to just cherry-pick packages, especially "leaf" ones. For example services like openssh.

We should still be able to implement that using the same trick above. For wholesale replace, we checkout $pending/usr and mount, for package specific, we checkout $current/usr, modify, and mount (we could use overlayfs, though having the same path for both wholesale and pkg-specific cases would keep things simple).

Another thing I don't like about the "mount /usr" approach is I think the libostree stack benefits from not generally messing around at the VFS or block layers. The more we cross those layers, the more we end up having to manage them.

What do you think of the single /usr approach wrt these concerns? It shouldn't be any more complex than rpm-ostree usroverlay today, right?

[cgwalters]

ostree checkout $pending/usr /ostree/repo/tmp/livefs-overlay

That dir is going to be subject to repo cleanup; even an ostree pull. A variant of this is ostree checkout $pending/usr ./usr-$pending - basically check out a versioned usr into the deployment root itself. This ensures that it's lifecycle bound together.

[cgwalters]

What do you think of the single /usr approach wrt these concerns?

The model of having per-deployment usr-$checksum dirs avoids creating cross-deployment dependencies which was a big concern. We also avoid having any persistent mounts across reboots, so again the rest of libostree doesn't really need to manage the mount stack. The only issue I can think of here is that any diffing approach would then need to be aware to skip those dirs. I'd probably advocate driving down the usr-$checksum approach into libostree itself, and e.g. having information in the deployment about the status of live updates.

[jlebon]

Nesting it in the current deployment root makes sense, yeah. Though we should avoid embedding a checksum in the name if we want to support e.g. cleanup -p && upgrade && livefs --replace, at which point we want that directory (usr-livefs?) to have the contents of the new pending deployment.

[jlebon]

So thinking about this some more, one concern that comes to mind is that only the first livefs operation will be atomic, since that's when we do the initial mount. Follow-up livefs operations would be modifying usr-livefs directly, which means errors halfway through modifying the directory could leave us in an undefined state.

One way we could make it atomic all the time would be to use symlinks on the /usr subdirs, e.g.

/usr-livefs/
  bin --> ../.usr-livefs.ln/bin
  lib64 --> ../usr-livefs.ln/lib64
  ...
/.usr-livefs.ln --> /.usr-livefs.0

Then just play the 0 <--> 1 switcheroo game.
Is there a simpler to do this?

[cgwalters]

will be atomic,

The thing is though, unless software goes to the effort of doing e.g. openat(AT_FDCWD, "/usr", O_DIRECTORY) and doing all further lookups from there, it's going to be racy. Think about just running e.g. python3 (or the dynamic linker, or really nearly everything) and in the middle we do the new bind mount.

One thing that would be possible but a whole lot of work would be to serialize systemd unit file startup on this, and have it maintain a private bind mount per-service for /usr that stays persistent, and we basically only GC the old root when all services referencing it have stopped.

Which, at that point we've basically gone really far down the path of having a container framework...

[jlebon]

Yeah, that's true, it'd be a lot of work to try to somehow cover those cases. Though I'm more concerned with indeterminately being stuck with a broken /usr if a failure happens halfway through. In the end, livefs will definitely compromise on correctness vs. always rebooting, though we can still do a lot better than dnf update.

[cgwalters]

Though I'm more concerned with indeterminately being stuck with a broken /usr if a failure happens halfway through.

Should be pretty easy to "roll back" by overmounting the original /usr on failures right? And detect this situation on daemon startup and do that too.

[jlebon]

(Side note: going by #1504, it seems like even a regular ex livefs without --replace also has the potential to break configuration.)

[jlebon]

Should be pretty easy to "roll back" by overmounting the original /usr on failures right? And detect this situation on daemon startup and do that too.

So, remounting on top of the /usr-livefs mount? Sure, but that's not feasible if we want to support repeated applications though, right?

Basically, what I mean is this:

• on the first application of livefs, we create e.g. /usr-livefs and mount it on top of /usr. The mounting is the last step, so that if it fails, we never even mount
• on the second application of livefs, we want to modify /usr-livefs, but if it fails, we want to revert back to the state of /usr-livefs after the first application, not the original one, right? That's why I was proposing using the symlinks trick.

[cgwalters]

WIP rewrite of current livefs in #2293

[jlebon]

A lot of discussions in this ticket, but I think the approach we're going with now was implemented in #2293. Let's do any follow-up design tweaks in a separate ticket based on that.