Modules Registry Planning

[gmpinder]

I think we might want to start working towards a module registry for people to start submitting their own modules for others to use so that the maintenance burden doesn't fall on our small team. That way anyone could share their creations.

The idea is something similar to how OpenTofu handles their registry. We have a repo that basically has directories of json files that contain information about modules that are submitted to the repo. These json files contain information like what modules a repo has, the schema, the public key. We then statically host those json files and it's then read by the CLI when it encounters a "namespaced" module.

Say like I have a repo at github.com/gmpinder/some-module, the user could set the type to gmpinder/some-module and the CLI would look for its entry in the registry, find the image reference, validate the module based on the schema, verify the image based on the registered public key, then mount it like how our official modules are mounted on RUN statements.

Ideally we would have a bot that is used to get this information and all the user has to do is supply their repo and the name of the module. The bot creates a PR that contains all of the information the registry needs to operate based on what's in the repo. We, as maintainers, would then review the registry entry and take a look at the module code just to make sure it's not doing anything malicious, approve, and then anyone can make use of the new module.

This will shift the burden of maintenance to the original author of the module and any new updates will be picked up by a scheduled job in the registry repo. We can then create a rather simple registry site that lists all the registered modules that people have submitted.

This would require some work on our part to get the system setup, but it would open up the ecosystem for others to contribute to.

Tasks

• module subcommand
• Registry
  • Create schema for registry entries
  • Repos
    • Registry site
    • Registry
    • Module template
  • Create site for registry
  • Create Github Action workflow for automating registry additions
  • Setup issues template for easy module submission

[xynydev]

Alright, some brainstorming...

Module namespacing / organization

I think it would be a good idea to separate modules into core, extra and community or something.

core would contain modules made by us that are considered fundamental to the build system (or something):

• script
• containerfile
• files
• signing ?
• maybe dnf and other system-level package managers?

extra would contain modules made by us that add extra features to a system:

• chezmoi
• default-flatpaks
• gnome-extensions

community would contain modules not made by us

The separation between core and extra need not necessarily be a technical one, just different categories on the website to make searching for documentation clearer.

community would be an entirely new module repository that would along the lines of what you outlined

community repo technical questions

• Should we build on top of the current module repository concept/functionality or replace it?
  • Background: It is currently possible to use non-official modules by setting source: to a different module repository.
  • Building on top of the current in my view would look like this: community members who want to create modules are instructed to setup module repositories similar to what blue-build/modules is like right now and their modules can be added to the central community registry by adding the details of where the module OCI is published and what its name is
  • Replacing it could look like this: we have a simplified module repo template that can contain only one module at a time, potentially publishing it as an OCI image, community modules are just OCI images
• What happens to source: as a module property?
• Do we run the CI/CD or is that the responsibility of module authors?
  • IMO it would be easiest if we had a custom GH action for module authors to use, like we do with image builders. Burning BlueBuild's CI/CD minutes on this would be disasterous if/when GH starts charging for usage.
  • Then again, only by running our own CI/CD could we add the security benefit of being able to guarantee that say v1.2.3 is always the same image and not changed by the module author
• Speaking of versioning: How should we do versioning to allow for the most security and usability?
  • Module authors effectively have RCE on all of their users machines and CI/CD pipelines. Currently with only official modules it would be pretty easy for us as the maintainers to sneak in malicious code, since most people's image builds always use the latest version of all modules. The risks coming from this need to be mitigated when opening up an official registry of unofficial modules.
  • With GitHub Actions, hash-based versioning is the most secure, because version tags can be updated after they are published and thus only commit hash versions can be audited. We should either use some form of user-facing hash-based versioning, or pin published version numbers to commit hashes. The latter could be accomplished for example by having the json files declaring the modules declare all published versions using commit hashes like this "v1.2.3": "c0a6ef7f6136469ef4a43dde79b7e22f11457b62". This would effectively require us to run some sort of CI to build the distributed module versions and documentation from the correct hashes.
  • Of course we could always allow for unsafe aliased usage such as latest and v1 as long as users are properly warned of the risks
• How would schema validation work?
  • We currently build JSONSchemas from TypeSpec centrally in blue-build/schema. Not sure if this approach would scale? Maybe we would need to build the JSONSchemas alongside the module OCIs? But how would they be integrated into the main schema in that case?
  • And what if a module creator doesn't care for schemas, neglects updating them, etc.? This could break builds if passing schema validation is made mandatory.

Registry site

Building a custom fully functional module registry site would be great. The current mechanism that adds module documentation to the documentation site is not perfect and could be improved by building a custom site. We could totally have official and community modules on the same site, with some neat organization and labeling.

The site would be fully static as is our main website, with content being based on the actual registry metadata in git. The registry site could live in the same repository as the registry to make fetching the content easier. Based on my work with the GitHub API in the BlueBuild Workshop, I'm pretty sure I could make a module submission form on the website that automatically forks the registry repo, creates a branch for the new module, commits the required files with automatically fetched details, and creates a well-formed PR for the main repo. This could also be adapted to do module updates based on semver and commit hashes, to reduce toil and increase trust with module updates.

A cool challenge to tackle: Could I figure out how to add modules to the main search bar on blue-build.org even though they are different websites? Right now I often look up module docs using the search bar. Of course the registry site would have a search bar as well, but this is just an interesting challenge I might tackle.

Broad implementation steps

(IMO)

1. Some amount of consensus on how everything should work (in terms of the actual product, not the website or docs or anything)
2. Make the repository for the community registry, a template for a module repo, and a test module we can add to the registry
3. CLI changes to make the test module run from the registry using a syntax of our choosing, test versioning robustness
4. Implement module schema related stuff?
5. Make MVP of registry website that contains docs for the official modules and the community registry modules
6. Announcement, community feedback, iteration, extra features

Useless open questions, thinking...

How does this effort relate to "competitors" like vib? Does this effort go towards enabling larger usecases like building non-desktop non-bootc OCIs with BlueBuild, even though that is not what the tool is originally intended for? How can we make sure that the tooling is generic enough to enable novel use cases that we have not planned for in a world where multiple (even desktop Linux) projects utilize OCI images as a backend? Is that necessary?

[gmpinder]

I think it would be a good idea to separate modules into core, extra and community or something.

Ideally yes, but we've already got a pattern for first-party modules that would break pretty much everybody's recipe's. It would be easier to support adding base namespaces that denote a user's module repo. Cause from what I'm reading, you're suggesting community/namespace/module@v1, core/dnf@v1, or extra/brew@v1 where we can instead stick with where our modules currently are and just allow namespace/module@v1. It's backwards compatible with all current recipes. Now if you're purely talking about organizing on the site, yeah I'd be fine with labeling them as such.

Building on top of the current in my view would look like this: community members who want to create modules are instructed to setup module repositories similar to what blue-build/modules is like right now and their modules can be added to the central community registry by adding the details of where the module OCI is published and what its name is

I was thinking of doing something close to our modules repo where your one repository can have multiple modules in it with a modules.json. However, I think we should create a release please workflow that creates tags that our system would scan for to add to the community registry. That way we can make use of image attestation to help to ensure the validity of the code.

What happens to source: as a module property?

I think we should keep it functioning as-is and mark it as an unsafe way to run custom module code. That way existing recipes will continue to work without the need for migration. It would also be useful for self-hosted/third-party image registries.

Do we run the CI/CD or is that the responsibility of module authors?

Their repo will build the images for their code.

Then again, only by running our own CI/CD could we add the security benefit of being able to guarantee that say v1.2.3 is always the same image and not changed by the module author
Speaking of versioning: How should we do versioning to allow for the most security and usability?
Module authors effectively have RCE on all of their users machines and CI/CD pipelines. Currently with only official modules it would be pretty easy for us as the maintainers to sneak in malicious code, since most people's image builds always use the latest version of all modules. The risks coming from this need to be mitigated when opening up an official registry of unofficial modules.

This is where my idea for tagging and image attestation comes into play. We would be able to ensure that the image with that version was built from that tag in a Github Action runner. We already do this with our CLI images, so we have a pattern we can follow.

With GitHub Actions, hash-based versioning is the most secure, because version tags can be updated after they are published and thus only commit hash versions can be audited. We should either use some form of user-facing hash-based versioning, or pin published version numbers to commit hashes. The latter could be accomplished for example by having the json files declaring the modules declare all published versions using commit hashes like this "v1.2.3": "c0a6ef7f6136469ef4a43dde79b7e22f11457b62". This would effectively require us to run some sort of CI to build the distributed module versions and documentation from the correct hashes.

So we would have our own CICD on the registry repo that scans each registered repository for all tags. The commit sha would be logged under that tag as well as all the digests for the modules that were published from that commit sha. When bluebuild is determining the image to use, it will check the registry and return the correct digest to use for the image.

How would schema validation work?
We currently build JSONSchemas from TypeSpec centrally in blue-build/schema. Not sure if this approach would scale? Maybe we would need to build the JSONSchemas alongside the module OCIs? But how would they be integrated into the main schema in that case?

Yeah I was thinking of having TypeSpec files be mandatory to be included. Ideally the CLI would have a way to template out a new module directory in the repo to make it easier for the user. We'd have to include a way to build the schema, which would probably require creating a minimal image for us to use that has all the necessary tooling. We could put the resulting schema in an image label/annotation that will get picked up by our registry scanner and then stored in the registry for that tagged version.

And what if a module creator doesn't care for schemas, neglects updating them, etc.? This could break builds if passing schema validation is made mandatory.

Then that's on the author and the users will have to let them know. We can't have control over that.

The site would be fully static as is our main website, with content being based on the actual registry metadata in git. The registry site could live in the same repository as the registry to make fetching the content easier. Based on my work with the GitHub API in the BlueBuild Workshop, I'm pretty sure I could make a module submission form on the website that automatically forks the registry repo, creates a branch for the new module, commits the required files with automatically fetched details, and creates a well-formed PR for the main repo. This could also be adapted to do module updates based on semver and commit hashes, to reduce toil and increase trust with module updates.

This sounds fantastic!

How does this effort relate to "competitors" like vib? Does this effort go towards enabling larger usecases like building non-desktop non-bootc OCIs with BlueBuild, even though that is not what the tool is originally intended for? How can we make sure that the tooling is generic enough to enable novel use cases that we have not planned for in a world where multiple (even desktop Linux) projects utilize OCI images as a backend? Is that necessary?

I think, for now, we just focus on building tooling for Bootc images and only support that use-case. If users want to try to use it for other things, that's on them and we can't and shouldn't stop them as it's their right, but we don't have to help. We're a small team of volunteers who aren't always around, so I'd like to keep that scope narrow.

[xynydev]

Thanks for the detailed response.

I think it would be a good idea to separate modules into core, extra and community or something.

Ideally yes, but we've already got a pattern for first-party modules that would break pretty much everybody's recipe's. It would be easier to support adding base namespaces that denote a user's module repo. Cause from what I'm reading, you're suggesting community/namespace/module@v1, core/dnf@v1, or extra/brew@v1 where we can instead stick with where our modules currently are and just allow namespace/module@v1. It's backwards compatible with all current recipes. Now if you're purely talking about organizing on the site, yeah I'd be fine with labeling them as such.

Yes, I am talking mostly about how the site should be organized. The modules would actually still be published under their current names in the same place. The idea is kinda like arch core and extra repo separation, where the extra repo can container less-essential modules that might have initially come from outside contributors, while we would make sure core modules are all tight, standard, and secure. But you would still use them the same way, just like you can pacman -S packages from both repos.

The community repo would then be more like a clear opt-in thingy like the AUR, where we communicate clearly that the user needs to trust the third-party developer.

---

On CI/CD:

• That attestation thing sounds really great, if it works as you've described it (I'm not that familiar). Does it allow proving that a certain image version was built from a certain commit hash as well? That's a concern since it's possible to modify tags after publishing, I think.
• Currently building TypeSpec requires all module schemas to be pulled in our CI/CD at build-time and inserted into the spec files using a custom script. It works for us right now, but is not pretty. We need to refactor this system somehow. Ideally, we should be able to have TypeSpec schemas built in the third-party maintainers CI/CD and just use that somehow. Need to do some research and thinking on this.

[gmpinder]

That attestation thing sounds really great, if it works as you've described it (I'm not that familiar). Does it allow proving that a certain image version was built from a certain commit hash as well? That's a concern since it's possible to modify tags after publishing, I think.

Attestation is designed to check an image was built from a specific git sha, so if the sha isn't in our registry, then we would be able to fail the validation.

• Currently building TypeSpec requires all module schemas to be pulled in our CI/CD at build-time and inserted into the spec files using a custom script. It works for us right now, but is not pretty. We need to refactor this system somehow. Ideally, we should be able to have TypeSpec schemas built in the third-party maintainers CI/CD and just use that somehow. Need to do some research and thinking on this.

I think for this we'll want to publish our typespec library so that community modules can import the common module type. I'd like to also try to setup a system where the resulting schema for the module is built and added as an annotation on the OCI that the cli can then read and use for properly validating the individual module.

The validation code in the CLI currently deserializes the file into a serde_json::Value type before running the jsonschema code over it. We could update the logic to iterate over each module, retrieve the value from type, determine the image, retrieve the annotation, load the schema for the module, and validate the Value containing the module portion with the schema pulled from the image. It makes it more dynamic in the CLI, but has the downside that we wouldn't be able to allow our base schema to be used by editors to fully validate every module, just validate the basic shape of the recipe. If a user needed to validate it fully, they would need to run it through bluebuild validate.

[xynydev]

Not having custom modules as part of the base schema has the downside of not offering any editor intellisense either, yeah... I wonder if there's a way to indicate to the LSP that a certain object in an array uses a different schema URL? Ehhh, whatever, as long as we have intellisense on our built-in modules and bluebuild validate works for the custom ones I guess it's fine.

Edit: Hmm, I wonder if it would work that the modules array type in Jsonschema is AnyOf a list of external $ref, all of which are paths to Jsonschema files on schema.blue-build.org. In this case all we would have to do is to republish all registry module schemas on schema.blue-build.org while enforcing a specific shape for those schemas.

[gmpinder]

Edit: Hmm, I wonder if it would work that the modules array type in Jsonschema is AnyOf a list of external $ref, all of which are paths to Jsonschema files on schema.blue-build.org. In this case all we would have to do is to republish all registry module schemas on schema.blue-build.org while enforcing a specific shape for those schemas.

Oh yeah, good idea. And we would also be able to rebuild that from the officially registered modules. Since each tag is already going to have its own entry in the registry, then we could also store the schema for the image of that tag.