Add registry support for dynamic stack updates [1/2]

[thepetk]

/kind user-story

Which area this user story is related to?

/area ci
/area registry

User Story

As a stack owner trying to create a new stack version, update an existing one or even introduce a new stack, I want to be able to see this change in the registry (e.g. the community registry) shortly after my PR is merged. This way, I don't have to wait for the new deployment of the devfile services team.

This user story focuses on the support of dynamic updates on stacks, like a creation/update of a stack version, or the creation of a new stack in general. Those updates should be available on the deployed registry shortly after the merge of each PR updating the stacks/ dir.

Current Format

The current flow of the devfile registry:

• Pushes all devfile artifacts inside the oci-registry.
• Uses a generated index.json file (stored inside the devfile-registry container, generated from the index/generator) to parse the indexSchema.
• The registry-viewer supports specific paths predefined during the build process.

As a result, in order to update the contents of an existing stack or create a new one, someone should:

• Update the artifacts inside the oci-registry.
• Regenerate the index.json.
• Regenerate the routes of the registry-viewer.

Suggested workaround

The key concept of the suggested workaround is to create a mechanism that will take care of the update of the OCI registry and the index.json so the index/server can fetch the updated information. In addition to that, the registry-viewer should be able to support dynamic routing so we are able to see every change in the viewer's interface.

Additional context

Some more details about the necessary changes required to implement the suggested workaround are:

Support generating index.json directly from github

The index/generator should be able to generate the index.json directly from github if a github URL is provided as a CLI arg.

Add the monitored registry repo's URL as ENV Var to the registry deployment

A registry deployment will support dynamic updates only if a specific env var is set during the deployment, pointing to the registry repo we would like to monitor.

Include the commit hash of the registry to the index.json

To be able to compare the different versions of the registry used to generate the index.json, the index/generator should include the commit hash used to generate the file, so we can compare it with another version.

Store index.json in oci-registry & use a cached version in the devfile-registry

The index.json should be moved to the OCI registry and get updated after every stack update. To avoid the index/server (runs inside devfile-registry container) fetching the index.json from the oci-registry, we should store a cached version inside the devfile-registry. Additionally to that we should create a mechanism inside the devfile-registry container that will take care to compare periodically the cached index.json version with the one stored in the OCI registry.

Support dynamic routing in the registry-viewer

The registry viewer should be able to support dynamic routing, so every time a new stack is added in the index.json it can serve the new stack's page.

Add functionality for OCI registry updates

After all the above steps are done, we are sure that we can support any updates made to the OCI registry. The next step is to implement the functionality that will take care of the OCI registry artifacts update and re-generate the index.json.

This functionality should only run after a specific request to the index/server made from the registry repository.

Create post-merge job on the registry side

The last step will be to create a post-merge action which will:

• Check if the merged updates affect the stacks/ directory.
• If yes it should send an authenticated request to the deployed registry, so the OCI registry update task runs.

Acceptance Criteria

**NOTE: This is the first of two parts for this EPIC. The rest of the action items required are here

As described in the above sections the acceptance criteria are:

• Add index generator support for registry github URL source. #1506
• Add support for registryRepoURL env var to the registry deployment. #1507
• Add support for commit hash field inside the index.json file. #1508
• Move index.json to oci-registry and use a cached version in devfile-registry #1509