Strengthen notion of "update driver" in rpm-ostree status

[jlebon]

This is kinda similar to #1406, but taking the idea further. In some downstream distributions, we want rpm-ostree updates to be driven by a separate client. E.g. in FCOS, we're discussing an agent that talks to rpm-ostree for downloading, applying, and rebooting into updates. Similarly, this could be used in Silverblue by GNOME Software when automatic updates are turned on.

In such cases, users will likely still be interacting with rpm-ostree directly, e.g. for package layering, rolling back, etc... Rather than having downstream users wrap status and confusing users with two CLIs, we should establish an interface by which rpm-ostree can extract information about the state of the update driver and display it in status. E.g. something like:

State: idle
AutomaticUpdates: driven by GNOME Software
  Last checked: 45m ago
Deployments:
● ostree://fedora:fedora/29/x86_64/silverblue
                   Version: 29.20190129.0 (2019-01-29T01:23:33Z)
                BaseCommit: 24963d8d5ffb03f47d1b5518ca7ce556ab97febf128ad6e9d15f68e3d6341dc5
              GPGSignature: Valid signature by 5A03B4DD8254ECA02FDA1637A20AA56B429476B4
           LayeredPackages: compat-ffmpeg28 gnome-calculator gnome-tweak-tool krb5-workstation libvirt-client libvirt-daemon-kvm libvirt-nss mosh oci-kvm-hook pcsc-lite rpmfusion-free-release strace tmux vim-X11 virt-install virt-manager
                            xsel

Or:

State: idle
AutomaticUpdates: driven by Cincinnati Client
  State: waiting for permission to reboot
Deployments:
  ostree://fedora-coreos:fedora/29/x86_64/coreos
                   Version: 29.32 (2019-01-30T15:34:08Z)
                    Commit: 1f96809b6dce4091a8b2ceecb8b3e2fa977ff83553cf2f6900b41c8743397322

● ostree://fedora-coreos:fedora/29/x86_64/coreos
                   Version: 29.31 (2018-12-11T15:34:08Z)
                    Commit: f98ec8a1977ff83553cf2f6900b41c87433973225f1c1babb0ccc166d1808225

We could have that data come out of a file in /run or something, or a helper script in /etc/rpm-ostreed.conf we run and paste its output directly in there.

[dustymabe]

seems like a good idea but it would be cool if the lines could link us to where we could find more information on the controlling update client.. The suggestion I would make here is to mention the systemd unit that controls that client:

AutomaticUpdates: driven by Cincinnati Client (cincinnati.service) 

[cgwalters]

we should establish an interface by which rpm-ostree can extract information about the state of the update driver

A lot of interesting design questions there.

We could have that data come out of a file in /run or something, or a helper script in /etc/rpm-ostreed.conf we run and paste its output directly in there.

Hm...yeah, I kind of like the low-tech approach of a file in /run. Defined to be unicode text suitable for display on a terminal?

[lucab]

Prior art from locksmith: https://github.com/coreos/locksmith/blob/master/pkg/coordinatorconf/conf.go

[lucab]

Expanding some details from my previous comment.
ContainerLinux was using the /run status file to show the reboot strategy in the "login welcome" message (from profile script). Relevant information there was the name of the agent in charge of driving the updates, and its status/strategy.

Depending on the usecase here, I think we have two options on the low-tech static side:

• if we need to handle this in some shell script, stick to the environment file format
• if we only want to source this in rpm-ostree or similar, use a simple JSON hashmap.

In both cases, the protocol dictated by rpm-ostree should cover the location of such file and well-known key names.

[jlebon]

if we need to handle this in some shell script, stick to the environment file format

If I'm understanding this, you're suggesting the "agent" script write to some environment file referenced in rpm-ostreed.service's unit? But then, the output won't change for the duration of that daemon run, right? (We do exit after a while, but still.) Or were you suggesting an environment file that rpm-ostree status reads in?

I like the idea of a simple JSON format. E.g. something like:

# /run/rpm-ostree/agent.json
{
  "name": "Cincinnati Client",
  "status": "waiting for permission to reboot",
  "unit": "cincinnati-client.service"
}

Hmm, another slightly less low-tech idea is to leverage the journal since we're already using that a bit. Dictate a msg id, and look for it. One advantage of this is that it's probably something the agents would want to do anyway (logging to the journal), and we transparently get a message, the unit name in charge of the agent, and a timestamp, so we can print e.g.

AutomaticUpdates: driven by $unit_description ($unit_name)
  State: $message ($relative_message_timestamp)

# e.g.

AutomaticUpdates: driven by Cincinnati Client (cincinnati-client.service)
  State: waiting for permission to reboot (12m ago)

# or

AutomaticUpdates: driven by Cincinnati Client (cincinnati-client.service)
  State: no updates available (8m ago)

And then a key in /etc/rpm-ostreed.conf to point to the agent identifier?

[lucab]

@jlebon the latter. rpm-ostreed can read it directly (on-request every time, or periodically cached), but having in env-file format makes it easy for other script/units to source directly. On the other hand, JSON is a better data-interchange format (also my preference) but can't be directly shell-sourced, in case we cared for that.

[jlebon]

Should probably tag @kalev here since this affects GNOME Software too.

[jlebon]

Thoughts on the journal entry idea in #1747 (comment)? One thing that I really like from it is that it implicitly exports the description, name, and status of the unit handling updates. ISTM it also feels more natural from the app's POV rather continuously updating a JSON file.

[kalev]

I don't have a strong opinion either way. I'd be happy to implement what's needed on gnome-software side though to support this.

[lucab]

@jlebon that's an interesting approach. Is your plan to only consider the most recent journal entry with a well-known message-id? Do you want to detect/react if there happen to be multiple conflicting drivers?

[jlebon]

Is your plan to only consider the most recent journal entry with a well-known message-id?

I hadn't fully thought it through, but yes I think so.

Do you want to detect/react if there happen to be multiple conflicting drivers?

We discussed this in video chat too, but just to write it down. I think we could detect this to be nice. Probably the most obvious thing to detect is whether rpm-ostree's automatic updates feature itself is turned on and an external update driver is detected (maybe just the stage mode though, because check mode I think is fine to have on while still being driven by an external updater). In practice, I don't think this will be much of a problem so I wouldn't block on it.

[kelvinfan001]

Also on board with using the one-shot journal entry to claim "ownership" and let everything else be communicated through systemd unit state.

[cgwalters]

I am concerned though that this is too implicit/deductive. The primary goal of this ticket is that we want the update driver to explicitly tell rpm-ostree that it "owns" updates.

Right, something I forgot is that zincati won't ask rpm-ostree to update until it knows there is one, whereas e.g. gnome-software defers to rpm-ostree directly.

We could add a simple AgentClaim DBus API that does the same thing that happens when an update is initiated?

One concern I have with parsing journal entries is access control; any process on the system can log a journal message. We can't filter for _UID=0 because both zincati and gnome-software run unprivileged.

[jlebon]

We could add a simple AgentClaim DBus API that does the same thing that happens when an update is initiated?

Hmm, that implies the daemon is always running, no?

One concern I have with parsing journal entries is access control; any process on the system can log a journal message. We can't filter for _UID=0 because both zincati and gnome-software run unprivileged.

Yeah, good point. Though it's mostly informational, not something rpm-ostree really needs to act upon. Also, we use the journal to get to the systemd service. So at least it's a process that has to be able to create and start a systemd service.

[cgwalters]

Hmm, that implies the daemon is always running, no?

Which daemon?

Either way I think we'd serialize this claim rpm-ostree side into say /run/rpm-ostree/state.json or whatever. When an update is actually performed on the next boot we would query the journal from rpm-ostreed and then the claim is complete.

So at least it's a process that has to be able to create and start a systemd service.

Not sure what you mean; any process can log a journal message.

[jlebon]

Which daemon?

Either way I think we'd serialize this claim ...

rpm-ostreed. Or right exactly, we'd have to serialize it somewhere.

So at least it's a process that has to be able to create and start a systemd service.

Not sure what you mean; any process can log a journal message.

We would ignore the message if it's for a systemd service which doesn't exist.

But again, note this is all informational. We also already print journal messages in rpm-ostree status (specifically, the ostree-finalize-staged logs if it failed). I'm sure this too could be fooled pretty easily.

Can you describe what the concern is? Adding a D-Bus API for this feels like overengineering at this point (also Zincati today doesn't use D-Bus, so we'd have to also add a CLI entrypoint for this). If we want to do things other than just displaying to the user, then yeah a new API which requires privileges seems warranted.

[cgwalters]

We would ignore the message if it's for a systemd service which doesn't exist.

Every process is running under some systemd unit. Now we could restrict things to e.g. .service but that still means that say a compromised but otherwise sandboxed service (say chronyd) could register as an update driver.

But again, note this is all informational.

Yes, but it's fairly important information and confusing an admin about updates seems bad.

We also already print journal messages in rpm-ostree status (specifically, the ostree-finalize-staged logs if it failed). I'm sure this too could be fooled pretty easily.

We should have strong journal source checks, see e.g.:
#2445

[cgwalters]

Adding a D-Bus API for this feels like overengineering at this point (also Zincati today doesn't use D-Bus, so we'd have to also add a CLI entrypoint for this).

I wouldn't call it overengineering; it'd involve some boilerplate C/C++ code but that's not really hard, neither is proxying it to a CLI.

On a thread very related to this; a while ago in the MCO I tried to use the systemd journal to log why reboots were being initiated, and it had the same problem of needing to filter by privileged processes.

A good way to think about this is that we're using the journal as IPC - much like if one has two processes where one reads from a file and the other writes it, you have all the usual concerns about access control/ownership etc.

[jlebon]

Sure OK, new D-Bus API WFM.

[kalev]

For what it's worth, I also think new D-Bus API sounds cleaner :)