Define process for testing PR builds

[perber]

Description

Our current pipeline only publishes builds from main when they are tagged with the current version. This makes it difficult to test changes from pull requests when a pre-built image is required to verify the behavior.

We should define a process for creating and testing builds from PRs, so changes can be validated before they are merged into main.

The test image creation should not happen automatically for every PR. Instead, it should be triggered explicitly via ChatOps, for example with a command like /create-test-image.

Goals

• Define how PR test images can be created.
• Provide a way to verify PR changes that require a pre-built image.
• Avoid creating test images automatically for every PR.
• Make it clear that these images are intended for testing only and are not guaranteed to be stable.

Open Questions

• Who is allowed to trigger the ChatOps command?
• How should PR test images be tagged?
• Should old PR test images be cleaned up automatically?
• Should the test image be rebuilt when new commits are pushed to the PR?

Proposal

PR test images should be created on demand via ChatOps, for example by commenting /create-test-image on a pull request. (I would say just maintainers?)

The resulting image should be tagged in a way that makes it clear that it belongs to a specific PR and commit. A possible tag format could be:

pr-<pr-number>-<short-sha>

Optionally, an additional human-readable tag based on the branch name could be published as well, but it should not be the only identifier.

[perber]

Hi @Hugo-Galley,

would you like to take a look at this topic.
I think it will help us in the future to test the PRs faster.

[Hugo-Galley]

Hello @perber,

I really like the idea of being able to easily build images on demand and in an automated way.

Regarding your questions, here are my thoughts:

• Who is allowed to trigger the ChatOps command?
  It depends on who should have access to this feature. If it is mainly intended for you, then restricting it to the owner with the possibility of your validation when someone else wants to test it could make sense.
  However, I think the best approach would be to make it available to Contributors and First-time Contributors. This way, only people who actually need it would be able to use it. It would reduce unnecessary usage and CI credit consumption, while still allowing you to approve access for additional users when needed.

• How should PR test images be tagged?
  The approach you suggested looks good to me. It makes the name unique and easy to identify.

• Should old PR test images be cleaned up automatically?
  We could use a GitHub Actions job to automatically delete the associated image when a PR is closed. We could also add a cleanup mechanism for PRs that remain open for too long (e.g., 30 days).

• Should the test image be rebuilt when new commits are pushed to the PR?
  I think the image should only be built when the command is explicitly triggered. This gives users control and allows them to push multiple commits before creating a new image, avoiding unnecessary image builds and CI usage.

Overall, I think the idea is interesting and would be a nice improvement to implement.

[perber]

Nice. Lets see how it will behave.
If you need feedback or if I should take a look at something let me know.