profile: add an upper/lower relationship to overlay vendor & user profile

[lucab]

torcx currently processes a single profile, containing an ordered list of image names+references. This can be either a vendor (RO) or a user (RW) profile, but switching to a user profile makes opting into vendor updates/changes harder to track.

This problem is spelled out in open-questions docs where we proposed either some arbitrary chaining (via includes directive) or just a flat profile but introducing well-known vendor references.

After some discussion here we distill a middle-ground solution, consisting of an implicit 2-layers profile overlaying: the lower layer is the vendor one, which is always applied if present; the upper layer is an optional user layer, which is merged (overriding) on top of the vendor one.
The final result is an ordered list of images where:

• the one specified in vendor profile comes first (lower priority)
• an image specified in the user profile overrides the same if present in vendor profile
• an empty/nil version tag in the user profile means that the archive

This introduces some implicit/merging state, so few thoughts come to mind:

• when we record the profile which we just applied, do we store the user profile or final flattened list? (i.e. are we more interested in making this re-usable later as-is or more in capturing the full current state?) - IMHO the latter
• nil-versioning an image can become stale (eg. when we remove archives from the distribution), should the merging alogrithm be relaxed to skip such cases? - IMHO yes, but reduces strictness

Brainstorming ticket, open for feedback! /cc @euank @crawford @dm0- @squeed

[squeed]

As per our offline discussion, I think we've found the right answer.

We should probably also add a command that displays the full set of packages to be installed. Perhaps as a part of torcx profile check.

[lucab]

@squeed that's probably not a good idea, as the result will be skewed compared to the actual next boot, as the vendor set/profile can change in the next boot (problem we already have right now for check). Perhaps we can fit this in the future process of signaling out-of-band which images/profiles are available in the next image, and feed such manifest/list to check as well. I'll need some more thoughts on this, I'll capture them in a separate ticket (#48).

[lucab]

Tracked at #55.

[dm0-]

It's probably too late to reconsider now but after toying with this, I think I'd prefer to see some include or parent option for profiles so the user has more control, instead of one always-enabled profile. I saw that was mentioned above but not chosen. What problems does it have?

I guess my end concern is to support knowledgeable users writing their /etc/torcx/next-profile to start from a clean slate. So, if the worry was users not realizing they need to include vendor, maybe go the opposite way and support some novendor option that allows them to only use the upper profile.

If we move more projects into torcx like OEM agents, etcd, SSSD, and other random packages, we would have to ship a default profile that enables them for compatibility, but most users would probably not want those things. There is also the increased memory usage from tmpfs, with Docker alone unpacking to ~80MiB.

If we stick with this two-layer configuration, we'll also need to figure out a sensible way to customize OEM-specific profiles while not getting in the way of user configuration from Ignition. For example, the GCE agent is planned to move from rkt to torcx, which we'd have to configure outside the vendor path in read-only /usr, but I think we'd still like the agent to be enabled by default as if it was in the vendor profile.

I'm not sure of a good way to handle all this in general, short of implementing systemd-style drop-in directory behavior and masking for profiles.

[lucab]

@dm0- thanks for the feedback. It is not too late, we are sorting out integration details right now.

What problems does it have?

It has a negative tradeoff between complexity and features. It introduces an arbitrary chain of dependencies, which will be harder to reason about and debug (re-ordering, duplicate images, missing profiles, duplicate profiles, including vendor profiles). This sounds like using the wrong tool to solve a much simpler problem of overlaying vendor/user data.

So, if the worry was users not realizing they need to include vendor, maybe go the opposite way and support some novendor option that allows them to only use the upper profile.

This was the previous status quo, and @euank was concerned becase it was an all-or-nothing config and a too easy way of losing all auto updating features. Masking at an image level introduces a bit more granularity, at the same time making a complete opt-out harder. However, if we want to go that route we can either:

1. add a vendor equal false or "" field in the generator config to completely disable the lower profile
2. let the user have an empty/null-symlinked profile named vendor in torcx confdir, with higher priority than our own (this may already unintentionally work)

(Barred the above concerns, I'm in favor of just documenting the behavior in 2 and making sure it properly works)

If we move more projects into torcx like OEM agents, etcd, SSSD, and other random packages, we would have to ship a default profile that enables them for compatibility, but most users would probably not want those things. There is also the increased memory usage from tmpfs, with Docker alone unpacking to ~80MiB.

This is a separate concern, which is on-spot here but exists anyway. I've splitted it at #62.

[squeed]

Indeed, Luca and I first considered an arbitrary chain of profiles, but we decided not to do that. The primary reasoning behind that was decreasing the possible failure surface - of course, at the expense of flexibility.

We couldn't come up with a compelling enough use case for arbitrary inheritance. If you think you have one, shoot.

[dm0-]

I think the empty file or null symlink at /etc/torcx/profiles/vendor.json is a good solution to disabling it.

I've split the OEM stuff into #68.