♲📦 Implement rpm-ostree rojig

[cgwalters]

Update 2018-02-22: This effort has been renamed to "rojig" to clarify the distinction from the original "jigdo" project.

---

Fetching content via both ostree and libdnf (RPM) ends up mixing the tradeoffs of both, requires release engineering to manage both, and makes it harder to mirror content. Not to mention the fact that there's the whole OCI/Docker thing which also works in a completely different way, and admins need to manage/mirror that too.

Now while the "obvious" thing would be to try to align with OCI in some way, the complete lack of wire deltas there is very problematic for uses outside of server clusters, and given that we already have lots of extensive linkage to RPM via libdnf, it makes the most sense to move that direction.

Hence, let's experiment with doing ostree-in-RPM, basically the "compose" process injects additional data (SELinux labels for example) in an "ostree image" RPM, like fedora-atomic-host-27.8-1.x86_64.rpm. That "ostree image" RPM will have extra data, including information of the "rojig set" of the other RPMs that are needed to download - basically we only download changed RPMs, like a traditional yum system. But SELinux labeling, depsolving, etc. are still done server side, and we still have a reliable OSTree commit checksum.

We're adopting the idea of Jigdo, hence the terminology. Though to be clear, much of the design and all of the implementation is fundamentally different, so we call this "rojig".

First PR: #1103

Current status: Proof of concept code works: we can take a set of RPMs + extra data (postprocessing), generating an OSTree commit (with reliable checksum), then convert that back into a "rojig set", which is the original RPMs plus a new "rojig RPM". Then a client side can take the "rojig set", just transferring RPM on the wire, reassembling bit for bit the original OSTree commit.

External discussion:

• CentOS: https://lists.centos.org/pipermail/centos-devel/2017-December/016345.html
• Fedora releng: mailing list post and pagure ticket
• atomic-devel: https://lists.projectatomic.io/projectatomic-archives/atomic-devel/2017-December/msg00003.html
• ostree-list: https://mail.gnome.org/archives/ostree-list/2017-December/msg00001.html

Followup issues:

• #1127

External work trackers:

• Anaconda
• Cockpit

---

Implementation: server side

When doing a compose tree, import the packages, and after import check out the final tree and SELinux relabel the imports so that they're reliably updated (currently depends on some unified core 🌐 work). Then, we can scan the pkg trees for objects, and find net-new objects that need to go in the oirpm.

I implemented that pretty quickly and hit on the issue that we have the initramfs 3 times in the tree (due to selinux labels), so I then implemented code to detect that; while we have 3 copies on disk, we can easily avoid that on the wire.

The big fundamental issue is SELinux - without that this would be way easier. The basic problem there is that we need to carry the SELinux labels in the oirpm, because the packages aren't labeled. So we need to carry a mapping from (pkg, file) → xattrs in the oirpm, and (on the client side) apply those when unpacking.

Implementation: client side

• Find OIRPM name (ostree origin: rpmostree-rojig:fedora-atomic-host)
• Look in repo for new NVR of OIRPM with that name (question: do we want to somehow pin to origin repo?)
• Download OIRPM, which contains list of packages to import (note this is a subset of the actual rpmdb due to e.g. emacs-filesystem and rootfiles etc., though we could just import all of them)
• Import each referenced package, using xattrs from OIRPM map
• Import net-new objects from OIRPM (commit,dirtree,dirmeta)
• All of the above replaces ostree pull - after this we proceed with the rest of the logic ideally unchanged

---

Task list

• Determine a story for GPG; pull the key from the "origin rpm-md repo"
• Client side UI design
• Client side implementation

[cgwalters]

First, let's use the term rojigRPM for the special "ostree image RPM". I think we can have this be one command that replaces compose, except note it doesn't take --repo since it doesn't write to an ostree repo (at least not a persistent one):

rpm-ostree compose jigdo --oirpm fedora-atomic-host-27.3-1 fedora-atomic/fedora-atomic.json

But for supportability reasons, I think we need both rpmostree-jigdo and "traditional" ostree repos (can I call that traditional yet?). Also implementation wise, I think it will be easier to reason about things if this is implemented as a magical postprocessing step given an ostree commit. Hence:

$ rpm-ostree compose tree --repo=repo-build fedora-atomic/fedora-atomic.json
fedora/27/x86_64/atomic-host ➭ 54c9c878ffdbf96a6773d920f8b6dc89751c1ad1b8297a6efa3f41ca5ede0142
$ rpm-ostree compose commit2jigdo  --repo=repo-build  --oirpm fedora-atomic-host-27.3-1 54c9c878ffdbf96a6773d920f8b6dc89751c1ad1b8297a6efa3f41ca5ede0142
Wrote: fedora-atomic-host-27.3-1.x86_64.rpm

or so?

And then we have a:

$ rpm-ostree compose jigdo2commit --reposdir /etc/yum.repos.d --releasever 27 --repo=repo-build fedora-atomic-host-27-3.1

Which inverts things back, for ease of testing.

[cgwalters]

First, let's use the term rojigRPM for the special "ostree image RPM". I think we can have this be one command that replaces compose, except note it doesn't take --repo since it doesn't write to an ostree repo (at least not a persistent one):

rpm-ostree compose jigdo --oirpm fedora-atomic-host-27.3-1 fedora-atomic/fedora-atomic.json

But for supportability reasons, I think we need both rpmostree-jigdo and "traditional" ostree repos (can I call that traditional yet?). Also implementation wise, I think it will be easier to reason about things if this is implemented as a magical postprocessing step given an ostree commit. Hence:

$ rpm-ostree compose tree --repo=repo-build fedora-atomic/fedora-atomic.json
fedora/27/x86_64/atomic-host ➭ 54c9c878ffdbf96a6773d920f8b6dc89751c1ad1b8297a6efa3f41ca5ede0142
$ rpm-ostree compose commit2jigdo  --repo=repo-build  --oirpm fedora-atomic-host-27.3-1 54c9c878ffdbf96a6773d920f8b6dc89751c1ad1b8297a6efa3f41ca5ede0142
Wrote: fedora-atomic-host-27.3-1.x86_64.rpm

or so?

And then we have a:

$ rpm-ostree compose jigdo2commit --reposdir /etc/yum.repos.d --releasever 27 --repo=repo-build fedora-atomic-host-27-3.1

Which inverts things back, for ease of testing.

[dustymabe]

is the plan for us to support both traditional ostree and OIRPM distribution methods forever? I know at least one thing that doesn't work for us in Fedora is that we don't keep around old rpms so deploying older comits would probably not be possible (or even the 'latest' released commit that was released two weeks ago and a newer version of an rpm within that commit has hit stable). Fedora is not the only use case and it's possible things could change there.

I see OIRPM as an augmentation to traditional repository. i.e. the client could choose which one they want to use. The traditional repo would still be the source of truth, but the rpms could be used to distribute things via RPM channels if some people couldn't mirror around ostrees within a big organization for some reason. How do you see it? Augmenting a traditional ostree server or completely replacing it?

Also, how would we handle deltas? Maybe they are no longer needed if the rpms from the previous jigdo are cached and you only need to download the drpms of the new ones.

[dustymabe]

is the plan for us to support both traditional ostree and OIRPM distribution methods forever? I know at least one thing that doesn't work for us in Fedora is that we don't keep around old rpms so deploying older comits would probably not be possible (or even the 'latest' released commit that was released two weeks ago and a newer version of an rpm within that commit has hit stable). Fedora is not the only use case and it's possible things could change there.

I see OIRPM as an augmentation to traditional repository. i.e. the client could choose which one they want to use. The traditional repo would still be the source of truth, but the rpms could be used to distribute things via RPM channels if some people couldn't mirror around ostrees within a big organization for some reason. How do you see it? Augmenting a traditional ostree server or completely replacing it?

Also, how would we handle deltas? Maybe they are no longer needed if the rpms from the previous jigdo are cached and you only need to download the drpms of the new ones.

[cgwalters]

Doesn't currently keep older RPMs on the mirrors - yes, today, but it certainly seems to me we can fix that, or just use S3/Cloudfront/whatever. Or create a repo subset - the things interesting to pkglayer on AH I suspect is a lot smaller than the total.

[cgwalters]

Doesn't currently keep older RPMs on the mirrors - yes, today, but it certainly seems to me we can fix that, or just use S3/Cloudfront/whatever. Or create a repo subset - the things interesting to pkglayer on AH I suspect is a lot smaller than the total.

[cgwalters]

Just to briefly respond regarding deltas: My thoughts are that we have another OIRPM type which has ostree deltas, and not deltaRPM, because...deltaRPM has a lot of flaws, or conversely ostree deltas are way better than deltaRPM.

IOW we'd have

fedora-atomic-host-27.42-x86_64.rpm and fedora-atomic-host-delta-27.41_27.42.x86_64.rpm or so.

Briefly the big flaws of deltaRPM are:

• It doesn't work with --nodocs as we use for Atomic Host and the container base images
• It always uses bsdiff even when that's not effective or an rsync-style rollsum would work just fine, and bsdiff is very CPU intensive
• Related to the above, it does a bsdiff over the entire RPM rather than per-file, and then is used to reconstruct the final RPM bit-for-bit on disk because that's what's GPG signed...the ostree approach of checksummed objects is a lot more efficient

[cgwalters]

Just to briefly respond regarding deltas: My thoughts are that we have another OIRPM type which has ostree deltas, and not deltaRPM, because...deltaRPM has a lot of flaws, or conversely ostree deltas are way better than deltaRPM.

IOW we'd have

fedora-atomic-host-27.42-x86_64.rpm and fedora-atomic-host-delta-27.41_27.42.x86_64.rpm or so.

Briefly the big flaws of deltaRPM are:

• It doesn't work with --nodocs as we use for Atomic Host and the container base images
• It always uses bsdiff even when that's not effective or an rsync-style rollsum would work just fine, and bsdiff is very CPU intensive
• Related to the above, it does a bsdiff over the entire RPM rather than per-file, and then is used to reconstruct the final RPM bit-for-bit on disk because that's what's GPG signed...the ostree approach of checksummed objects is a lot more efficient

[jlebon]

Having carefully reread over the jigdo docs, I have a somewhat better grasp of the design here and it definitely sounds interesting. How large do you expect the OIRPM to be here?

Also, what are your thoughts about how signature verification will work? We don't want to download all the RPMs ahead of time -- so it seems like we have to download the OIRPM first, unpack it, check the signature, and only then proceed to download new RPMs? Or alternatively use RPM signatures? Either way, it sounds like we'd have to download the whole OIRPM before checking that it has a valid signature, which is unfortunate. Or are you approaching this a different way?

[jlebon]

Having carefully reread over the jigdo docs, I have a somewhat better grasp of the design here and it definitely sounds interesting. How large do you expect the OIRPM to be here?

Also, what are your thoughts about how signature verification will work? We don't want to download all the RPMs ahead of time -- so it seems like we have to download the OIRPM first, unpack it, check the signature, and only then proceed to download new RPMs? Or alternatively use RPM signatures? Either way, it sounds like we'd have to download the whole OIRPM before checking that it has a valid signature, which is unfortunate. Or are you approaching this a different way?

[cgwalters]

The baseline OIRPM for Fedora Atomic Host is currently:
-rw-r--r--. 1 root root 65M Nov 14 21:48 /root/rpmbuild/RPMS/x86_64/fedora-atomic-host-27.42-1.fc26.x86_64.rpm
which is mostly the initramfs, with the rest the rpmdb, and then random "%post" data like all the XML files update-mime-database generates.

As far as signatures ✍ I think it's going to be annoying to people doing "mirror all RPMs with gpgcheck=1" unless we do RPM signatures. But just like static deltas we'd embed the ostree commit meta (hm I need to implement that).

We'd unpack the RPM and check the ostree signature too (if enabled).

[cgwalters]

The baseline OIRPM for Fedora Atomic Host is currently:
-rw-r--r--. 1 root root 65M Nov 14 21:48 /root/rpmbuild/RPMS/x86_64/fedora-atomic-host-27.42-1.fc26.x86_64.rpm
which is mostly the initramfs, with the rest the rpmdb, and then random "%post" data like all the XML files update-mime-database generates.

As far as signatures ✍ I think it's going to be annoying to people doing "mirror all RPMs with gpgcheck=1" unless we do RPM signatures. But just like static deltas we'd embed the ostree commit meta (hm I need to implement that).

We'd unpack the RPM and check the ostree signature too (if enabled).

[jlebon]

Doesn't currently keep older RPMs on the mirrors - yes, today, but it certainly seems to me we can fix that, or just use S3/Cloudfront/whatever.

I'm still trying to wrap my head around this. IIUC, this means that in order to support deploying older commits, the content provider will also have to keep around every RPM referenced in those commits, right? Seems like a pretty big change in RPM management. Or I suppose we could at the very least just keep "delta-OIRPMs" to go from stable to the specific version? But those would still have to carefully managed separately.

Of course, right now on traditional yum/dnf-managed Fedora systems, you only have access to stable and updates snapshots too. Though having access to older versions is definitely one of the nice features of AH.

[jlebon]

Doesn't currently keep older RPMs on the mirrors - yes, today, but it certainly seems to me we can fix that, or just use S3/Cloudfront/whatever.

I'm still trying to wrap my head around this. IIUC, this means that in order to support deploying older commits, the content provider will also have to keep around every RPM referenced in those commits, right? Seems like a pretty big change in RPM management. Or I suppose we could at the very least just keep "delta-OIRPMs" to go from stable to the specific version? But those would still have to carefully managed separately.

Of course, right now on traditional yum/dnf-managed Fedora systems, you only have access to stable and updates snapshots too. Though having access to older versions is definitely one of the nice features of AH.

[cgwalters]

Both CentOS and RHEL keep older RPM versions. It's only Fedora that's a standout. That said clearly we would need to drive versioning of the rpm-md repos more consistently and expose it.

[cgwalters]

Both CentOS and RHEL keep older RPM versions. It's only Fedora that's a standout. That said clearly we would need to drive versioning of the rpm-md repos more consistently and expose it.

[cgwalters]

BTW just from today: https://lwn.net/Articles/740052/

[cgwalters]

BTW just from today: https://lwn.net/Articles/740052/

[cgwalters]

OK so let's start talking about how this looks/feels on the client side. Strawman:

Actually before we dive in here let's make a presupposition: we rework the fedora.repo and fedora-updates.repo to be the same thing, or at least rpm-ostree is aware they're "the same". And we also "repo-bind" the oirpm - in other words, the fedora repo here acts very similar to an ostree remote.

Upgrading

# rpm-ostree status
State: idle, autoexit in 9 seconds
Repos: fedora fedora-cisco-openh264
Transport Security: GPG, https
Booted Image ●: fedora:fedora-atomic-workstation-27.6-1.x86_64 (2017-11-28 23:33:42)
  LayeredPackages: emacs fuse-sshfs git-evtag keepassx krb5-workstation libvirt-client
                   opensc origin-clients pcsc-lite-ccid powerline strace tmux vagrant-libvirt
                   virt-manager xsel ykclient ykpers

# rpm-ostree upgrade
Enabled rpm-md repositories: fedora fedora-cisco-openh264
rpm-md repo 'fedora' (cached); generated: 2017-11-05 05:51:47
Downloading metadata: fedora 18MB...
Updated image: fedora:fedora-atomic-workstation-27.7-1.x86_64 (2017-12-01 10:32:11)
Downloading 5 packages:
  kernel 4.13.15-300.fc27 -> 4.13.16-300.fc27
  kernel-core 4.13.15-300.fc27 -> 4.13.16-300.fc27
  kernel-modules 4.13.15-300.fc27 -> 4.13.16-300.fc27
  kernel-modules-extra 4.13.15-300.fc27 -> 4.13.16-300.fc27
  rpm-ostree 2017.10-2.fc27 -> 2017.10-3.fc27
  rpm-ostree-libs 2017.10-2.fc27 -> 2017.10-3.fc27
...
Success; new update pending: fedora:fedora-atomic-workstation-27.7-1.x86_64
# rpm-ostree status
State: idle, autoexit in 9 seconds
Repos: fedora fedora-cisco-openh264
Transport Security: GPG, https

Pending Image 🆕: fedora:fedora-atomic-workstation-27.7-1.x86_64 (2017-12-01 10:32:11)

Booted Image ●: fedora:fedora-atomic-workstation-27.6-1.x86_64 (2017-11-28 23:33:42)
  LayeredPackages: emacs fuse-sshfs git-evtag keepassx krb5-workstation libvirt-client
                   opensc origin-clients pcsc-lite-ccid powerline strace tmux vagrant-libvirt
                   virt-manager xsel ykclient ykpers
#

Well OK I omitted layered pkg updates but eh.

deploy

# rpm-ostree deploy 27.2
Enabled rpm-md repositories: fedora fedora-cisco-openh264
...
Downloading fedora:fedora-atomic-workstation-27.2-1.x86_64
...

rebase

# rpm-ostree rebase fedora-testing:fedora-atomic-workstation
Enabled rpm-md repositories: fedora fedora-testing fedora-cisco-openh264

Adding the repo here automatically enables it; this is like yum --enablerepo=updates-testing distro-sync.

[cgwalters]

OK so let's start talking about how this looks/feels on the client side. Strawman:

Actually before we dive in here let's make a presupposition: we rework the fedora.repo and fedora-updates.repo to be the same thing, or at least rpm-ostree is aware they're "the same". And we also "repo-bind" the oirpm - in other words, the fedora repo here acts very similar to an ostree remote.

Upgrading

# rpm-ostree status
State: idle, autoexit in 9 seconds
Repos: fedora fedora-cisco-openh264
Transport Security: GPG, https
Booted Image ●: fedora:fedora-atomic-workstation-27.6-1.x86_64 (2017-11-28 23:33:42)
  LayeredPackages: emacs fuse-sshfs git-evtag keepassx krb5-workstation libvirt-client
                   opensc origin-clients pcsc-lite-ccid powerline strace tmux vagrant-libvirt
                   virt-manager xsel ykclient ykpers

# rpm-ostree upgrade
Enabled rpm-md repositories: fedora fedora-cisco-openh264
rpm-md repo 'fedora' (cached); generated: 2017-11-05 05:51:47
Downloading metadata: fedora 18MB...
Updated image: fedora:fedora-atomic-workstation-27.7-1.x86_64 (2017-12-01 10:32:11)
Downloading 5 packages:
  kernel 4.13.15-300.fc27 -> 4.13.16-300.fc27
  kernel-core 4.13.15-300.fc27 -> 4.13.16-300.fc27
  kernel-modules 4.13.15-300.fc27 -> 4.13.16-300.fc27
  kernel-modules-extra 4.13.15-300.fc27 -> 4.13.16-300.fc27
  rpm-ostree 2017.10-2.fc27 -> 2017.10-3.fc27
  rpm-ostree-libs 2017.10-2.fc27 -> 2017.10-3.fc27
...
Success; new update pending: fedora:fedora-atomic-workstation-27.7-1.x86_64
# rpm-ostree status
State: idle, autoexit in 9 seconds
Repos: fedora fedora-cisco-openh264
Transport Security: GPG, https

Pending Image 🆕: fedora:fedora-atomic-workstation-27.7-1.x86_64 (2017-12-01 10:32:11)

Booted Image ●: fedora:fedora-atomic-workstation-27.6-1.x86_64 (2017-11-28 23:33:42)
  LayeredPackages: emacs fuse-sshfs git-evtag keepassx krb5-workstation libvirt-client
                   opensc origin-clients pcsc-lite-ccid powerline strace tmux vagrant-libvirt
                   virt-manager xsel ykclient ykpers
#

Well OK I omitted layered pkg updates but eh.

deploy

# rpm-ostree deploy 27.2
Enabled rpm-md repositories: fedora fedora-cisco-openh264
...
Downloading fedora:fedora-atomic-workstation-27.2-1.x86_64
...

rebase

# rpm-ostree rebase fedora-testing:fedora-atomic-workstation
Enabled rpm-md repositories: fedora fedora-testing fedora-cisco-openh264

Adding the repo here automatically enables it; this is like yum --enablerepo=updates-testing distro-sync.

[cgwalters]

Let me try to answer this:

How do you see it? Augmenting a traditional ostree server or completely replacing it?

For Fedora/CentOS/RHEL: my instinct here is that the answer has to be "completely replacing" - otherwise we're not really gaining the intended benefit of reducing releng/management overhead right?. For other organizations...I don't know?

My instinct here is that the people interested in rpm-ostree are going to be happiest with jigdo but I can't say for sure. I would like rpm-ostree to work in "pure ostree" mode (I heard someone compain that rpm-ostree fails on start if there's no rpmdb). And the people who are using libostree today are likely happy enough with it...the audience there is a lot of Debian and OpenEmbedded. It seems not unlikely that some of those OpenEmbedded users will also want jigdo though?

That said I don't see us deleting any of the existing functionality; or stated more positively, we're just adding an alternative in the code for now. The whole test suite continues to pass. But it is possible that some new feature would only be supported in jigdo mode perhaps?

[cgwalters]

Let me try to answer this:

How do you see it? Augmenting a traditional ostree server or completely replacing it?

For Fedora/CentOS/RHEL: my instinct here is that the answer has to be "completely replacing" - otherwise we're not really gaining the intended benefit of reducing releng/management overhead right?. For other organizations...I don't know?

My instinct here is that the people interested in rpm-ostree are going to be happiest with jigdo but I can't say for sure. I would like rpm-ostree to work in "pure ostree" mode (I heard someone compain that rpm-ostree fails on start if there's no rpmdb). And the people who are using libostree today are likely happy enough with it...the audience there is a lot of Debian and OpenEmbedded. It seems not unlikely that some of those OpenEmbedded users will also want jigdo though?

That said I don't see us deleting any of the existing functionality; or stated more positively, we're just adding an alternative in the code for now. The whole test suite continues to pass. But it is possible that some new feature would only be supported in jigdo mode perhaps?

[jlebon]

Actually before we dive in here let's make a presupposition: we rework the fedora.repo and fedora-updates.repo to be the same thing, or at least rpm-ostree is aware they're "the same". And we also "repo-bind" the oirpm - in other words, the fedora repo here acts very similar to an ostree remote.

That seems kinda cool, but feels a bit too heuristicky/mysterious. It seems conceptually simpler to just stick this information in the origin, no? E.g. have a [jigdo] group in the origin under which we keep the list of repos we have enabled. I guess we'd have to keep the fedora-atomic-workstation magic string there too, right? (I say magic string because I don't know what to call it, maybe... the "jigdo origin" ?). Or were you thinking of storing that somewhere else?

In that case, rebasing to a jigdo origin that's not in the currently enabled repos would just use --enablerepo, which is something people should already be familiar with.

# rpm-ostree deploy 27.2
Enabled rpm-md repositories: fedora fedora-cisco-openh264
...
Downloading fedora:fedora-atomic-workstation-27.2-1.x86_64
...

How does this work? Is it just doing a pure pkg search for ${jigdo_origin}-${version} ? Seems like we want to make sure we use some new Provides notation here to be more explicit? If we also have a provides that embeds the commit SHA, we could keep the deploy $sha syntax as well.

# rpm-ostree rebase fedora-testing:fedora-atomic-workstation
Enabled rpm-md repositories: fedora fedora-testing fedora-cisco-openh264

It would be cool if we supported seamlessly moving back and forth between "ostree-based" and "jigdo-based" remotes. If we don't overload what remotes means, then keeping the remote:branch for ostree-based remotes makes that easy and natural. Though we would need a new syntax for moving from ostree to jigdo... maybe just something like:

rpm-ostree rebase --jigdo fedora-atomic-workstation

?

[jlebon]

Actually before we dive in here let's make a presupposition: we rework the fedora.repo and fedora-updates.repo to be the same thing, or at least rpm-ostree is aware they're "the same". And we also "repo-bind" the oirpm - in other words, the fedora repo here acts very similar to an ostree remote.

That seems kinda cool, but feels a bit too heuristicky/mysterious. It seems conceptually simpler to just stick this information in the origin, no? E.g. have a [jigdo] group in the origin under which we keep the list of repos we have enabled. I guess we'd have to keep the fedora-atomic-workstation magic string there too, right? (I say magic string because I don't know what to call it, maybe... the "jigdo origin" ?). Or were you thinking of storing that somewhere else?

In that case, rebasing to a jigdo origin that's not in the currently enabled repos would just use --enablerepo, which is something people should already be familiar with.

# rpm-ostree deploy 27.2
Enabled rpm-md repositories: fedora fedora-cisco-openh264
...
Downloading fedora:fedora-atomic-workstation-27.2-1.x86_64
...

How does this work? Is it just doing a pure pkg search for ${jigdo_origin}-${version} ? Seems like we want to make sure we use some new Provides notation here to be more explicit? If we also have a provides that embeds the commit SHA, we could keep the deploy $sha syntax as well.

# rpm-ostree rebase fedora-testing:fedora-atomic-workstation
Enabled rpm-md repositories: fedora fedora-testing fedora-cisco-openh264

It would be cool if we supported seamlessly moving back and forth between "ostree-based" and "jigdo-based" remotes. If we don't overload what remotes means, then keeping the remote:branch for ostree-based remotes makes that easy and natural. Though we would need a new syntax for moving from ostree to jigdo... maybe just something like:

rpm-ostree rebase --jigdo fedora-atomic-workstation

?

[cgwalters]

(I say magic string because I don't know what to call it, maybe... the "jigdo origin" ?)

How about jigdo:fedora-testing:fedora-atomic-workstation I.e. we tack on the jigdo: prefix for jigdo mode, and the middle value becomes the rpm-md repo?

Basically I really like the idea of "repo pinning" the jigdo origin. (Not implemented in jigdo2commit right now but it'd be trivial to add to the libdnf query right?)

Seems like we want to make sure we use some new Provides notation here to be more explicit? If we also have a provides that embeds the commit SHA, we could keep the deploy $sha syntax as well.

Ooh those are cool ideas 👍. In fact I may just go implement them now!

[cgwalters]

(I say magic string because I don't know what to call it, maybe... the "jigdo origin" ?)

How about jigdo:fedora-testing:fedora-atomic-workstation I.e. we tack on the jigdo: prefix for jigdo mode, and the middle value becomes the rpm-md repo?

Basically I really like the idea of "repo pinning" the jigdo origin. (Not implemented in jigdo2commit right now but it'd be trivial to add to the libdnf query right?)

Seems like we want to make sure we use some new Provides notation here to be more explicit? If we also have a provides that embeds the commit SHA, we could keep the deploy $sha syntax as well.

Ooh those are cool ideas 👍. In fact I may just go implement them now!

[jlebon]

How about jigdo:fedora-testing:fedora-atomic-workstation I.e. we tack on the jigdo: prefix for jigdo mode,

Yeah, that's cool! So we basically reserve the jigdo string as a remote. Having that in the baserefspec will signal the upgrader that we're jigdo-based.

and the middle value becomes the rpm-md repo?

How would we create names for custom repos/multiple repos at once? It seems cleaner to just make it a line (but an important one!) in the status output that lists enabled repos, no?

[jlebon]

How about jigdo:fedora-testing:fedora-atomic-workstation I.e. we tack on the jigdo: prefix for jigdo mode,

Yeah, that's cool! So we basically reserve the jigdo string as a remote. Having that in the baserefspec will signal the upgrader that we're jigdo-based.

and the middle value becomes the rpm-md repo?

How would we create names for custom repos/multiple repos at once? It seems cleaner to just make it a line (but an important one!) in the status output that lists enabled repos, no?

[jlebon]

/me thinks some more 🤔

Basically I really like the idea of "repo pinning" the jigdo origin. (Not implemented in jigdo2commit right now but it'd be trivial to add to the libdnf query right?)

OK, I totally understand what you mean now. Yeah, that makes complete sense. The rpmmd repo in the refspec is the one from which we want to make sure the OIRPM comes from.

[jlebon]

/me thinks some more 🤔

Basically I really like the idea of "repo pinning" the jigdo origin. (Not implemented in jigdo2commit right now but it'd be trivial to add to the libdnf query right?)

OK, I totally understand what you mean now. Yeah, that makes complete sense. The rpmmd repo in the refspec is the one from which we want to make sure the OIRPM comes from.

[cgwalters]

Yeah, that's cool! So we basically reserve the jigdo string as a remote.

I was thinking of it more like a "type prefix", like what skopeo does with the transports e.g. oci: docker:. So for full symmetry...we could render 👁 current ostree origins in the status command like this:

# rpm-ostree status
State: idle
Deployments:
  ostree:fedora-27:fedora/27/x86_64/workstation

(I wouldn't propose changing current origin files due to the obvious backcompat issue)

[cgwalters]

Yeah, that's cool! So we basically reserve the jigdo string as a remote.

I was thinking of it more like a "type prefix", like what skopeo does with the transports e.g. oci: docker:. So for full symmetry...we could render 👁 current ostree origins in the status command like this:

# rpm-ostree status
State: idle
Deployments:
  ostree:fedora-27:fedora/27/x86_64/workstation

(I wouldn't propose changing current origin files due to the obvious backcompat issue)

[jlebon]

I was thinking of it more like a "type prefix"

Yup, that makes sense. We can follow the same rules as the current rebase syntax if the type is missing. I.e. if already on a jigdo remote, then rebase updates-testing: will know to stay in jigdo mode and just switch to updates-testing. And vice-versa, as usual.

Actually before we dive in here let's make a presupposition: we rework the fedora.repo and fedora-updates.repo to be the same thing, or at least rpm-ostree is aware they're "the same".

In light of the above, I think I understand better where this is coming from now. You want to special case fedora and updates to just show up as fedora? This is just a UI/polish thing though, right? Seems OK I guess, though it seems fine to me if the jigdo refspec we ship is e.g. jigdo:updates:fedora-atomic-host. It makes the remote = repo name concept easier to explain.

[jlebon]

I was thinking of it more like a "type prefix"

Yup, that makes sense. We can follow the same rules as the current rebase syntax if the type is missing. I.e. if already on a jigdo remote, then rebase updates-testing: will know to stay in jigdo mode and just switch to updates-testing. And vice-versa, as usual.

Actually before we dive in here let's make a presupposition: we rework the fedora.repo and fedora-updates.repo to be the same thing, or at least rpm-ostree is aware they're "the same".

In light of the above, I think I understand better where this is coming from now. You want to special case fedora and updates to just show up as fedora? This is just a UI/polish thing though, right? Seems OK I guess, though it seems fine to me if the jigdo refspec we ship is e.g. jigdo:updates:fedora-atomic-host. It makes the remote = repo name concept easier to explain.

[cgwalters]

I just realized there's a subtle issue in the whole current design: because on the server side today we only look for an object once in the whole package set (we build a map of object hash → (pkg, file path) ), and then on the client side skip importing any duplicates, it creates a subtle dependency between the jigdo set on the server side, and the client's imported packages.

Basically if we don't import all content from a package, we can't rely on having a local version being part of a new jigdo set. Consider what would happen if for example we decided to turn on documentation for FAH.

I see two options here:

• Somehow include an invalidation mechanism; the docs case is obviously straightforward, but that doesn't help if e.g. a duplicated object happened to be added in a new package.
• Always import all package content; this would mean today's FAH would gain all the docs on disk in the pkg imports, even if we still don't include them in the final tree

[cgwalters]

I just realized there's a subtle issue in the whole current design: because on the server side today we only look for an object once in the whole package set (we build a map of object hash → (pkg, file path) ), and then on the client side skip importing any duplicates, it creates a subtle dependency between the jigdo set on the server side, and the client's imported packages.

Basically if we don't import all content from a package, we can't rely on having a local version being part of a new jigdo set. Consider what would happen if for example we decided to turn on documentation for FAH.

I see two options here:

• Somehow include an invalidation mechanism; the docs case is obviously straightforward, but that doesn't help if e.g. a duplicated object happened to be added in a new package.
• Always import all package content; this would mean today's FAH would gain all the docs on disk in the pkg imports, even if we still don't include them in the final tree

[cgwalters]

From this effort we also have an opportunity to "retune" the pitch for rpm-ostree client side; this bleeds into the discussions in renaming considerations. One way I've been thinking about this is building an analogy between the "image RPM" (jigdoRPM) and a "metapackage" or comps group.

We can say that in "yum-osimg" your system tracks a (single) "image RPM". Some of the semantics are quite a lot like a metapackage.

An entirely different analogy is with the Dockerfile FROM line. We can describe the semantics there as a lot like:

FROM registry.fedoraproject.org/fedora:27
RUN yum -y install foo

Except that that we actually have a useful definition of "in place upgrade" where we e.g. preserve modified files in /etc plus your data in /var. And updates we don't require redownloading the RPMs, and we also don't allow (by default) replacing components from the base. Also we handle SELinux labeling. Also updates to the "base image" don't require redownloading the whole thing. (So...not really like docker at all but the analogy might be a bit useful still)

[cgwalters]

From this effort we also have an opportunity to "retune" the pitch for rpm-ostree client side; this bleeds into the discussions in renaming considerations. One way I've been thinking about this is building an analogy between the "image RPM" (jigdoRPM) and a "metapackage" or comps group.

We can say that in "yum-osimg" your system tracks a (single) "image RPM". Some of the semantics are quite a lot like a metapackage.

An entirely different analogy is with the Dockerfile FROM line. We can describe the semantics there as a lot like:

FROM registry.fedoraproject.org/fedora:27
RUN yum -y install foo

Except that that we actually have a useful definition of "in place upgrade" where we e.g. preserve modified files in /etc plus your data in /var. And updates we don't require redownloading the RPMs, and we also don't allow (by default) replacing components from the base. Also we handle SELinux labeling. Also updates to the "base image" don't require redownloading the whole thing. (So...not really like docker at all but the analogy might be a bit useful still)

[cgwalters]

OK so the docs issue. We need a way to pass from the server to the client the fact that we don't want to import docs. How about a documentation: [bool] in the commit metadata? We do of course ship /usr/share/rpm-ostree/treefile.json but...I'm not certain about starting to parse that client side.

Actually in order to support parallel install of the jigdoRPM and the jigdoSet maybe we need it to be a special Provides: rpmostree-jigdo(nodocs) or so?

[cgwalters]

OK so the docs issue. We need a way to pass from the server to the client the fact that we don't want to import docs. How about a documentation: [bool] in the commit metadata? We do of course ship /usr/share/rpm-ostree/treefile.json but...I'm not certain about starting to parse that client side.

Actually in order to support parallel install of the jigdoRPM and the jigdoSet maybe we need it to be a special Provides: rpmostree-jigdo(nodocs) or so?

[cgwalters]

I'd like to fix this soon since today the importer will pull in docs into the pkg commits even though they aren't in the final commit; so it's sort of the worst of both worlds: we use the on-disk space of the docs without actually having them in the final tree.

[cgwalters]

I'd like to fix this soon since today the importer will pull in docs into the pkg commits even though they aren't in the final commit; so it's sort of the worst of both worlds: we use the on-disk space of the docs without actually having them in the final tree.

[jlebon]

Actually in order to support parallel install of the jigdoRPM and the jigdoSet maybe we need it to be a special Provides: rpmostree-jigdo(nodocs) or so?

Yeah, seems like it belongs more as a property of the RPM itself rather than in the commit metadata.

Somehow include an invalidation mechanism; the docs case is obviously straightforward, but that doesn't help if e.g. a duplicated object happened to be added in a new package.

I'm not sure I fully understand this. The docs case is clear, though I don't follow this part:

that doesn't help if e.g. a duplicated object happened to be added in a new package

Can you elaborate on this? If the new package is not part of the jigdo set, then we don't need to import it, right?

[jlebon]

Actually in order to support parallel install of the jigdoRPM and the jigdoSet maybe we need it to be a special Provides: rpmostree-jigdo(nodocs) or so?

Yeah, seems like it belongs more as a property of the RPM itself rather than in the commit metadata.

Somehow include an invalidation mechanism; the docs case is obviously straightforward, but that doesn't help if e.g. a duplicated object happened to be added in a new package.

I'm not sure I fully understand this. The docs case is clear, though I don't follow this part:

that doesn't help if e.g. a duplicated object happened to be added in a new package

Can you elaborate on this? If the new package is not part of the jigdo set, then we don't need to import it, right?

[cgwalters]

Can you elaborate on this? If the new package is not part of the jigdo set, then we don't need to import it, right?

I broke that out into #1197

[cgwalters]

Can you elaborate on this? If the new package is not part of the jigdo set, then we don't need to import it, right?

I broke that out into #1197