Support OSTree archive as tarball

[agners]

To decouple the build system from OSTree repository handling a tarball containing the OSTree repository in archive mode with just the single commit would be useful. This tarball can then be copied/moved/uploaded to a server where it then gets moved into a (potentially public) repository (e.g. using pull-local).

This is how the Fedora CoreOS builder currently works, where one of the build output artifacts is a tarball with the OSTree with just the single commit of this particular build.

[pattivacek]

To decouple the build system from OSTree repository handling a tarball containing the OSTree repository in archive mode with just the single commit would be useful.

If the goal is just decoupling, don't we achieve that with garage-push as well as with the tarballs that are already generated? Is the difference in your goal that you want a tarball with the repo/filesystem as OSTree sees it? That sounds reasonable to me and also shouldn't be particularly hard to add as an output or entry in IMAGE_FSTYPES.

[agners]

The tarballs we generate are either pure rootfs or images with OSTree in bare mode. What I am looking for is a tarball which just contains an OSTree in archive mode. Basically whatever ${OSTREE_REPO} tar'ed. This then allows to do the garage-push later in a separate step (e.g. after testing/promoting the release...).

Currently there is no IMAGE_FSTYPES for that, but it is actually fairly easy to do by just adding the following to classes/image_types_ostree.bbclass:

TAR_IMAGE_ROOTFS_task-image-ostreecommit = "${OSTREE_REPO}"

Then adding

IMAGE_FSTYPES += "ostreecommit.tar.xz"

Creates a tarball of that OSTree in archive mode. However, if that OSTREE_REPO is shared, then this of course contains more than the commit of that particular build. Also OSTREE_REPO is not cleaned after a build.

So I see two ways here:

• Add a variable e.g. OSTREE_REPO_VOLATILE, which if set just deletes the OSTREE_REPO after the build. One has to make sure that ${OSTREE_REPO} points to a location which is not shared with other builds. With this approach the user can have either a shared ${OSTREE_REPO} or the tarballs.
• Do something more involved, and build a OSTree archive somewhere in build tmp and then ostree pull-local that commit to ${OSTREE_REPO}. This way the user can have a tarball as well as a repo with all builds...

[eu-smirnov]

I think it would be nice to have this. For our general use-case, garage-push is still better as it uploads only the missing objects. But it might become useful for the potential "offline USB-update" use-case. Having a tarball of the repo as a build artifact is definitely more convenient than doing pull-local on the build machine to the USB drive.

Also OSTREE_REPO is not cleaned after a build.

BTW, I'm not sure if that's intentional. AFAIK older commits are not used anywhere. Normally, ${IMGDEPLOYDIR} will contain only the latest image files, so this behavior may be surprising for users, as after a while it can take a significant amount of disk space.

Also, is there a reason to have ${OSTREE_REPO} in ${IMGDEPLOYDIR} and not in ${WORKDIR}?

[pattivacek]

Also OSTREE_REPO is not cleaned after a build.

BTW, I'm not sure if that's intentional. AFAIK older commits are not used anywhere. Normally, ${IMGDEPLOYDIR} will contain only the latest image files, so this behavior may be surprising for users, as after a while it can take a significant amount of disk space.

Agreed, I don't think we rely on this functionality.

Also, is there a reason to have ${OSTREE_REPO} in ${IMGDEPLOYDIR} and not in ${WORKDIR}?

Good question, but this might be philosophical. It is arguably deployed, and it's definitely an image, so I don't think it's unreasonable to be in $IMGDEPLOYDIR, but $WORKDIR seems fine, too.

Currently there is no IMAGE_FSTYPES for that, but it is actually fairly easy to do by just adding the following to classes/image_types_ostree.bbclass:

TAR_IMAGE_ROOTFS_task-image-ostreecommit = "${OSTREE_REPO}"

Then adding

IMAGE_FSTYPES += "ostreecommit.tar.xz"

Creates a tarball of that OSTree in archive mode.

Yep, makes sense and sounds reasonable to me. I'm open to this route.

So I see two ways here:

* Add a variable e.g. `OSTREE_REPO_VOLATILE`, which if set just deletes the `OSTREE_REPO` after the build. One has to make sure that ${OSTREE_REPO} points to a location which is not shared with other builds. With this approach the user can have either a shared `${OSTREE_REPO}` or the tarballs.

* Do something more involved, and build a OSTree archive somewhere in build tmp and then ostree pull-local that commit to `${OSTREE_REPO}`. This way the user can have a tarball as well as a repo with all builds...

How about this: by default we keep the repo volatile and clean it up preferably before reusing it. (It is sometimes useful to keep it after bitbaking to examine it.) Then we have a separate option that enables persisting a separate repo somewhere that won't get deleted as often (not sure what would be appropriate). Then we'd just pull from the temporary repo into the persistent one if the option is enabled.