Define Elemental Toolkit Scope (obsolete)

[davidcassany]

Elemental Toolkit repository deserves some attention if we ever consider growing a community around this project. For that a clear scope should be defined and the repository should be maintainable for an eventual small upstream community.

What if we focus Elemental Toolkit on providing the very basic essentials that could be added on top a Linux distro to turn it into an immutable OS built and delivered from OCI images? So no actual OS is built as an outcome of this repository, probably just few examples are required to show case how the Elemental Toolkit utilities could be applied on top of the base OCI image of common distros. Those could also be used in CI just for testing purposes.

On that regard these are few draft ideas about how we could re arrange and hopefully simplify the project:

1. Define Elemental core packages: properly state what a release means and what is actually supported and expected to remain stable

2. Try to reduce the maintainability overhead of the repository:

   • Simplify release scheme to public registries, consider whole repository as a block, we publish on elemental-tookit repo tags:
     What if we assume three repository types
     • main repo: builds from main branch (unstable, publish on merge if CI is green)
     • latest repo:builds from latest elemental-toolkit tag (stable, publish on tag, no CI required)
     • tagged repo: repository holding a tagged version of elemental-toolkit, just for the records (publish on tag no CI required)
   • Simplify build artifacts selection
     • Build everything from scratch every time. What if core and toolchain packages are always rebuild in CI, so we prevent issues such as a PR did no actually test the changes.
   • Probably drop some packages. At a glance I could see:
     • system/cos: I doubt this should exist in elemental-tookit, as this is not software package like others, this is actually building and defining a root-tree. I'd use a Dockerfile for example purposes.
     • initrd and kernel: IMHO elemental-toolkit shouldn't have this sort of packages, they are misleading (there is no elemental specific kernel or initrd), this stuff should be provided by the base image and created or included during the root-tree build.
     • cos-wrapper: this is outdated and from stone age
     • live/boot, live/syslinux, live/systemd-boot: they are likely to be unmaintained, plus they are repackaging stuff that every distro already provides in their own repos. I don't think repacking binaries has any value.
     • backports: I'd drop all of them, this old untested and unmaintained stuff.
     • utils/rancherd: deprecated
   • Include a single Dockerfile per flavor to build a full OS using elemental-toolkit core packages. I think this is a nice and easy to read way to show case what Elemental Toolkit offers. I would not release the build out of it, just use it as an example playground and for OS build in CI.
   • Drop Packer templates and builds in favor of a image builder based on elemental-cli:
     • makes disk builds ligther (no VM involved)
     • no need to create install media to later on create disks (one step less in tests)
     • no need to maintain packer templates
     • building a disk is mostly a wrapper around elemental install, it is not a coding effort

3. Rearrange project structure to define a set of core packages and rearrange the repository in way to make obvious these are the relevant and stable bits

   • probably rearrange packages in three folders?
     <root>/packages/core: core packages that must remain stable and likely to be required on any Elemental based OS
     <root>/packages/toolchain: packages only used at build time
     <root>/packages/misc: all other non core and non build time packages, no stability commitment

Last but not least, something I have been thinking about is trying to make core packages really distro agnostic, so no need to build them for different flavors. The same build should work and makes sense for all (green, orange, blue and probably others). Also would it make sense installing everything under /usr/local to be respectful with the underlaying distro? In any case this would be an idea for a next iteration.

[Itxaka]

main repo: builds from main branch (unstable, publish on merge if CI is green)
latest repo:builds from latest elemental-toolkit tag (stable, publish on tag, no CI required)
tagged repo: repository holding a tagged version of elemental-toolkit, just for the records (publish on tag no CI required)

Not sure about this. the current implementation where you have everything on a repo is very nice.
We currently publish on master merge if CI is green (with a tagged release basically but that points to a sha) and on tag if CI is green (tagged release with version number)
The main repository which points to the latest artifacts is done as part of the tagged and its basically free (only uploads a new repository.yaml which points to the existing artifacts)

I would keep the CI on both as getting the CI to run on a tag is very nice and at that point we are not in a hurry to publish.

The good thing about pushing to the repo both on main merge and tag is that:

• Doesnt need to build all packages from scratch, reuses versions that were built on master (cross repo main/tag wont work for this, we will need to rebuild the whole tag packages)
• Packages can independently be pushed without a cos release (so elemental-cli latest version can be in the repos for others to consume, no need to wait for a cos release that uses it). This gives us more freedom to test different packages and to decouple the packages from the COS release.

Build everything from scratch every time. What if core and toolchain packages are always rebuild in CI, so we prevent issues such as a PR did no actually test the changes.

This sounds nice but it was a nightmare. Workers run out of space when building everything and just the build step took over 1 hour, sometimes 1:30. If we vastly reduce the number of packages this would be ok, but the build.go code was done to avoid rebuilding everything which was pretty time-consuming (not counting the running out of space during the full build)

system/cos: I doubt this should exist in elemental-tookit, as this is not software package like others, this is actually building and defining a root-tree. I'd use a Dockerfile for example purposes.
initrd and kernel: IMHO elemental-toolkit shouldn't have this sort of packages, they are misleading (there is no elemental specific kernel or initrd), this stuff should be provided by the base image and created or included during the root-tree build.
cos-wrapper: this is outdated and from stone age
live/boot, live/syslinux, live/systemd-boot: they are likely to be unmaintained, plus they are repackaging stuff that every distro already provides in their own repos. I don't think repacking binaries has any value.
backports: I'd drop all of them, this old untested and unmaintained stuff.
utils/rancherd: deprecated

agree on all

Include a single Dockerfile per flavor to build a full OS using elemental-toolkit core packages. I think this is a nice and easy to read way to show case what Elemental Toolkit offers. I would not release the build out of it, just use it as an example playground and for OS build in CI.

Sounds good but we also had issues with the docker builds and we had to disable them, cant remember why, sorry. But good for me to re-test this.

Drop Packer templates and builds in favor of a image builder based on elemental-cli:

Packer was mostly due to having a "installed" machine ready for testing with the exact artifacts generated during the CI build. As long as we can reproduce the same thing Im ok with this. Do you mean with this to use the build-image from elemental-cli or an extra thing? This will also mean that our test has to change as everything is based on the assumption that we will get a pre-configured VM with the latest artifacts in there. This sounds a bit costly in manpower for the benefits that we would get. You meantion is not costly but I don understand that, how are we gonna test this stuff? The combo of packer+vagrant makes this very easy AND reproducible locally, which is usually a pain point when testing the toolkit issues.

Also dont forget that the packer calling the installation has been instrumental in finding issues with the installs. We kind of use it as an end2end install testing for elemental-cli lol

probably rearrange packages in three folders?
/packages/core: core packages that must remain stable and likely to be required on any Elemental based OS
/packages/toolchain: packages only used at build time
/packages/misc: all other non core and non build time packages, no stability commitment

Nice, agreed.

Last but not least, something I have been thinking about is trying to make core packages really distro agnostic, so no need to build them for different flavors. The same build should work and makes sense for all (green, orange, blue and probably others).

Well...if we drop the mostly-distro-dependent packages like grub and kernel crap maybe. Not sure how difficult would this be

Also would it make sense installing everything under /usr/local to be respectful with the underlaying distro? In any case this would be an idea for a next iteration

Isnt /usr/local the persistent partition?

[davidcassany]

main repo: builds from main branch (unstable, publish on merge if CI is green)
latest repo:builds from latest elemental-toolkit tag (stable, publish on tag, no CI required)
tagged repo: repository holding a tagged version of elemental-toolkit, just for the records (publish on tag no CI required)

Not sure about this. the current implementation where you have everything on a repo is very nice. We currently publish on master merge if CI is green (with a tagged release basically but that points to a sha) and on tag if CI is green (tagged release with version number) The main repository which points to the latest artifacts is done as part of the tagged and its basically free (only uploads a new repository.yaml which points to the existing artifacts)

I would keep the CI on both as getting the CI to run on a tag is very nice and at that point we are not in a hurry to publish.

Yes I am also not completely sure about this. I was just thinking on understanding the release as a whole and how to keep previous releases available, so you can pin no to specific package to a repository including all the consistent packages for a release.

Build everything from scratch every time. What if core and toolchain packages are always rebuild in CI, so we prevent issues such as a PR did no actually test the changes.

This sounds nice but it was a nightmare. Workers run out of space when building everything and just the build step took over 1 hour, sometimes 1:30. If we vastly reduce the number of packages this would be ok, but the build.go code was done to avoid rebuilding everything which was pretty time-consuming (not counting the running out of space during the full build)

Yes, I know it was a nightmare. It is the fast way I could think of ensuring we actually test and release what is in the repository and we are not omitting to build or publish some stuff because we did not bump whatever depends on the changes we aim to commit. Anyway I guess this handled as a aside improvement/discussion. It is just that it already happened few times our master is broken because we did not bump package A which was silently dependent on the PR changes, so it was not tested in PRs and as soon it gets rebuild and published later on we are in a difficult position as we have to hack to force publish or force merge untested PRs... probably we should better define and identify pain cases and find solutions.

Drop Packer templates and builds in favor of a image builder based on elemental-cli:

Packer was mostly due to having a "installed" machine ready for testing with the exact artifacts generated during the CI build. As long as we can reproduce the same thing Im ok with this. Do you mean with this to use the build-image from elemental-cli or an extra thing? This will also mean that our test has to change as everything is based on the assumption that we will get a pre-configured VM with the latest artifacts in there. This sounds a bit costly in manpower for the benefits that we would get. You meantion is not costly but I don understand that, how are we gonna test this stuff? The combo of packer+vagrant makes this very easy AND reproducible locally, which is usually a pain point when testing the toolkit issues.

Yes, my idea is to create a build-image command that wraps this docs basically is simple elemental install call on a loop device and get the full customization possibilities.

Last but not least, something I have been thinking about is trying to make core packages really distro agnostic, so no need to build them for different flavors. The same build should work and makes sense for all (green, orange, blue and probably others).

Well...if we drop the mostly-distro-dependent packages like grub and kernel crap maybe. Not sure how difficult would this be

Yes I am also unsure about if there would be any rock block. I think that, at least, for what we use in Elemental Teal this would be possible.

Also would it make sense installing everything under /usr/local to be respectful with the underlaying distro? In any case this would be an idea for a next iteration

Isnt /usr/local the persistent partition?

Yes, thats why this is something for a next iteration 😅 . The idea of distro-agnostic packages that do not mess with in distro packages is kind a couple with that, we should install under /usr/local and probably set the persistent partition under /usr/local/var or somewhere else. This is just to be a nice citizen and try to follow FHS and common linux practices.

[davidcassany]

@mudler I'd be happy to get you feedback in here too, would be nice if the toolkit is arranged in a way that could eventually be a clean playground for different projects and products.

[mudler]

Elemental Toolkit repository deserves some attention if we ever consider growing a community around this project. For that a clear scope should be defined and the repository should be maintainable for an eventual small upstream community.

What if we focus Elemental Toolkit on providing the very basic essentials that could be added on top a Linux distro to turn it into an immutable OS built and delivered from OCI images? So no actual OS is built as an outcome of this repository, probably just few examples are required to show case how the Elemental Toolkit utilities could be applied on top of the base OCI image of common distros. Those could also be used in CI just for testing purposes.

On that regard these are few draft ideas about how we could re arrange and hopefully simplify the project:

1. **Define Elemental core packages**: properly state what a release means and what is actually supported and expected to remain stable

2. **Try to reduce the maintainability overhead** of the repository:
   
   * Simplify release scheme to public registries, consider whole repository as a block, we publish on elemental-tookit repo tags:
     What if we assume three repository types
     
     * **main repo**: builds from main branch (unstable, publish on merge if CI is green)
     * **latest repo**:builds from latest elemental-toolkit tag (stable, publish on tag, no CI required)
     * **tagged repo**: repository holding a tagged version of elemental-toolkit, just for the records (publish on tag no CI required)
   * Simplify build artifacts selection
     
     * Build everything from scratch every time. What if core and toolchain packages are always rebuild in CI, so we prevent issues such as a PR did no actually test the changes.
   * Probably drop some packages. At a glance I could see:
     
     * **system/cos**: I doubt this should exist in elemental-tookit, as this is not software package like others, this is actually building and defining a root-tree. I'd use a Dockerfile for example purposes.

Agree, system/cos has no purpose rather than testing, it can be simplified a lot to match toolkit consumers

     * **initrd and kernel**: IMHO elemental-toolkit shouldn't have this sort of packages, they are misleading (there is no elemental specific kernel or initrd), this stuff should be provided by the base image and created or included during the root-tree build.
     * **cos-wrapper**: this is outdated and from stone age
     * **live/boot, live/syslinux, live/systemd-boot**: they are likely to be unmaintained, plus they are repackaging stuff that every distro already provides in their own repos. I don't think repacking binaries has any value.
     * **backports**: I'd drop all of them, this old untested and unmaintained stuff.
     * **utils/rancherd**: deprecated
   * Include **a single Dockerfile per flavor** to build a full OS using elemental-toolkit core packages. I think this is a nice and easy to read way to show case what Elemental Toolkit offers. I would not release the build out of it, just use it as an example playground and for OS build in CI.

👍 that sounds good here

   * **Drop Packer** templates and builds in favor of a image builder based on elemental-cli:
     
     * makes disk builds ligther (no VM involved)
     * no need to create install media to later on create disks (one step less in tests)
     * no need to maintain packer templates
     * building a disk is mostly a wrapper around `elemental install`, it is not a coding effort

Argh, mixed feelings here. I understand and it's unnecessary - however as @Itxaka mentioned was very instrumental. I think something similar can be achieved without packer, so maybe it's just about dropping it for a simpler bash script.

3. **Rearrange project structure** to define a set of core packages and rearrange the repository in way to make obvious these are the relevant and stable bits
   
   * probably rearrange packages in three folders?
     `<root>/packages/core`: core packages that must remain stable and likely to be required on any Elemental based OS
     `<root>/packages/toolchain`: packages only used at build time
     `<root>/packages/misc`: all other non core and non build time packages, no stability commitment

Last but not least, something I have been thinking about is trying to make core packages really distro agnostic, so no need to build them for different flavors. The same build should work and makes sense for all (green, orange, blue and probably others). Also would it make sense installing everything under /usr/local to be respectful with the underlaying distro? In any case this would be an idea for a next iteration.

While this sounds good, could clash horribly with our current setup, but there are other ways to achieve that and makes sense indeed.

This is all good - my only concern is that to be consumed outside it's hard to keep up and your requirements are shifting already - breaking changes like splitting bios and efi should have been discussed before-ahead and leaving already consumers like me to pinned versions around.

[Itxaka]

breaking changes like splitting bios and efi should have been discussed before-ahead and leaving already consumers like me to pinned versions around.

Yes, this was a mistake on our part for trying to move too fast. Sorry about that, as you can see we now moved to use discussions so this should not happen again without community feedback :)

What part of the new setup are you not able to rework regarding that? Is kairos still supporting bios? AFAIK we still support both paths no? Could you open a bug about this if its really impacting you so we can work this out?

[davidcassany]

This is all good - my only concern is that to be consumed outside it's hard to keep up and your requirements are shifting already -

Yes, that's also why I think a narrower and focused scope of this repository is relevant so we can better define what is actually provided and how to consume it. Making a clear statement on the core parts will help to make the project more accessible and also help us on preventing breaks.

I'll try to digest this discussion on few actionable items that, ideally, can be sorted in a reasonable time frame.

[davidcassany]

So far from the discussion I believe we kind of agree on few aspects which could be all translated into actionable cards. Here they are:

• Drop system/cos in favor of a simple (and not optimized at all) Dockerfile approach (it should install missing base packages from distro repos, missing luet package from elemental repos and not much more). Used as a common usage example and for CI, nothing else.
• Drop a bunch of unneeded/obsolete packages (backports, rancherd, initrd, kernel, etc.), some of the logic of the dropped packages should be moved to Dockerfile (e.g. initrd creation)
• Organize all packages in one of these three categories (probably there can be more): core, toolchain, misc. Map this somehow to the project structure. Probably also mapped in independent published repos? Using basic criteria, no package in core requires anything from misc, neither anything from core requires at runtime anything from toolchain.

I believe with these three changes we could already make the elemental-toolkit way easier to be consumed for any project based on these bits. Hopefully also easier to maintain.

@Itxaka @mudler @bk201 @rancher/elemental if you agree I'd create these three cards and try to further elaborate one by one

[Itxaka]

@bk201 WE can see that one of the packages we want to deprecate (rancherd) is still being maintained from the harvester and according to the messages on teh readme, plans to be used by rancherd. Do we need to keep it in the toolkit? or we can just remove it directly?

[bk201]

We are fetching the executable from the Github directly: https://github.com/harvester/os2/blob/95fe7e55511f3d78b3b84f2a7854b19a6fe4119b/Dockerfile#L41
So if it's not used in elemental, I guess it's OK to drop it.

Not sure if it's related: before we migrate to latest elemental, we still need the luet packages on this index: https://github.com/harvester/os2/blob/95fe7e55511f3d78b3b84f2a7854b19a6fe4119b/files/etc/luet/luet.yaml#L17

[bk201]

cc @Vicente-Cheng @masteryyh

[Itxaka]

Not sure if it's related: before we migrate to latest elemental, we still need the luet packages on this index: harvester/os2@95fe7e5/files/etc/luet/luet.yaml#L17

luet repos are not updated unless its the latest one (repository.yaml). Once we have released a "named" repo (either DATE-repository.yaml or more recently SNAPSHOT-repository.yaml) those are never touched again, they represent a snapshot of the components at that time. We would never drop a repo or update it unless it has some broken packages. So that repo will keep on working :)