Add github action to run e2e command "on-demand"

[JorTurFer]

Signed-off-by: jorturfer jorge_turrado@hotmail.es

We have been discussing how to run e2e test in PRs before merge them when anyone with writing permissions requests it.
There are several options to achieve this behavior, in this PR I propose to use several actions in a row to allow it.

When someone with writing permissions comment the PR with the message /e2e, automatically this action will be executed reacting the comment with 🚀 to refer that the pipeline is in process, and also with 👍 / 👎 to notify if the e2e tests have passed or not.

Also, the trigger supports setting the e2e test discovery regex using if you want, using this format /e2e REGEX, ex:

• /run-e2e *.test.ts
• /run-e2e cron*

Note: To use this feature, we should build and push this image ghcr.io/kedacore/build-tools:main because a few extra packages are needed like up-to-dated git, or hub pr
Note2: Because of this, a PAT with delete permissions in the packages should be added also in order to be able to drop the e2e-test-tag from ghcr. GITHUB_TOKEN is not enough. The expected secret name is GHCR_AUTH_PAT

Achievements:

• Execute e2e test on demand using a command
• Allow providing the regex to select a subset of the tests
• Execute the process to ensure that it works

Checklist

• Commits are signed with Developer Certificate of Origin (DCO - learn more)
• Tests have been added
• A PR is opened to update our Helm chart (repo) (if applicable, ie. when deployment manifests are modified)
• A PR is opened to update the documentation on (repo) (if applicable)
• Changelog has been updated

Fixes #2224

[zroubalik]

Looking great so far, minor nit, I'd change the trigger command e2e -> /e2e, so we avoid unintentional execution :)

[JorTurFer]

I think that I'm near of the solution :)
https://github.com/JorTurFer/keda2/pull/8#issuecomment-960047587
https://github.com/JorTurFer/keda2/runs/4098101471?check_suite_focus=true

[JorTurFer]

Another sample is this PR
The first comment (/e2e cron*) passed the tests and has 🚀 👍
The second comment (/e2e) didn't pass the tests and has 🚀 👎

[zroubalik]

Looking really great!

[tomkerkhove]

Finally caught up on the idea and it's great 👍

I'd suggest renaming the command to /run e2e-tests though to be open and feel more natural, it's just a suggestion.

[tomkerkhove]

We should also have a follow-up item to refactor our actions and reduce the duplication by using reusable workflows: https://docs.github.com/en/actions/learn-github-actions/reusing-workflows

[JorTurFer]

Finally caught up on the idea and it's great 👍

I'd suggest renaming the command to /run e2e-tests though to be open and feel more natural, it's just a suggestion.

Honestly, I'd like to keep the command shorter than possible to reduce the possibility of errors. But maybe we can rename to /test {regex}. WDYT?

• /test
• /test cron*

[JorTurFer]

I think that now all the goals are achieved:
The PR
The comment (/e2e cron*) passed the tests and has 🚀 👍
The comment (/e2e azure-blob*) didn't pass the tests and has 🚀 👎

The images used during the process are deleted after the e2e tests have finished.
I'm going to update this PR with the latest changes from my working repository and I think that all is done except the setup of the final "triggering comment".

Please, take a look at the PR description, I updated it to add 2 important notes related with the requirements to do this usable

[zroubalik]

@JorTurFer excellent!

ad Note1: the image will be automatically build once we merge this PR

ad Note2: @tomkerkhove might help?

[zroubalik]

agree on making the command short, /e2e is fine for me or /run-e2e too.

[zroubalik]

@JorTurFer so it doens't mean that if any tag has > 5k downloads, the whole package (thus all the other tags) is protected?

[tomkerkhove]

Let's make it very explicit and use dedicated images to fully isolate - Better safe than sorry.

[JorTurFer]

@JorTurFer so it doens't mean that if any tag has > 5k downloads, the whole package (thus all the other tags) is protected?

I don't understand that. I understand that the package itself it's protected but not all the versions. You could delete one version or whole package.
We can do a test easily, you could create a tag manually and try to delete it

[zroubalik]

Let's make it very explicit and use dedicated images to fully isolate - Better safe than sorry.

💯

[JorTurFer]

do you want to use other organization or only other package?
I could do something like this:

# If E2E_IMAGE_TAG is defined, we are on pr e2e test and we have to use the new tag and append -test to the repository
ifeq '${E2E_IMAGE_TAG}' ''
VERSION = main
SUFFIX = ''
endif

ifneq '${E2E_IMAGE_TAG}' ''
VERSION = ${E2E_IMAGE_TAG}
SUFFIX = '-test'
endif

IMAGE_REGISTRY ?= ghcr.io
IMAGE_REPO     ?= kedacore

IMAGE_CONTROLLER = $(IMAGE_REGISTRY)/$(IMAGE_REPO)/keda$(SUFFIX):$(VERSION)
IMAGE_ADAPTER    = $(IMAGE_REGISTRY)/$(IMAGE_REPO)/keda-metrics-apiserver$(SUFFIX):$(VERSION)

This change will append -test to the package name if we set E2E_IMAGE_TAG.

In this second case, we have to create at least 1 tag manual because it's not possible to keep the repository without any version (tag), you will get an error if you try it saying that you should delete the whole package. We can create an empty image manually and push to only to have one tag always.

WDYT?

[zroubalik]

@JorTurFer yeah, that would be probably the best approach.

[JorTurFer]

I'll try it and I hope this afternoon the PR will be ready :)
Testing this changes is a bit tedious because I have to adapt them in other organization (and honestly, I start to having hype about see this running xD)

[zroubalik]

@JorTurFer no worries, if you change the target images we can merge this PR and do the testing direclty in here.

[JorTurFer]

@zroubalik FYI

[zroubalik]

LGTM

[JorTurFer]

could you create the *-test packages with one dummy tag? Have I got enough permissions to do that?

[zroubalik]

@tomkerkhove could you please help here, I am afk

[tomkerkhove]

could you create the *-test packages with one dummy tag? Have I got enough permissions to do that?

Why is that? Can't you just push them through a test run?

[JorTurFer]

In this second case, we have to create at least 1 tag manual because it's not possible to keep the repository without any version (tag), you will get an error if you try it saying that you should delete the whole package. We can create an empty image manually and push to only to have one tag always.

Yes, but we can't delete a tag if it's the only tag that keeps there, so I think that could be better if we create a dummy tag in order to be able to delete others

[tomkerkhove]

Frankly, if it's a test image we don't have to delete the tags perse which allow us to reduce the scope of the PAT anyway.

[JorTurFer]

Frankly, if it's a test image we don't have to delete the tags perse which allow us to reduce the scope of the PAT anyway.

If we don't want to remove the test images, we don't need the PAT, for pushing proposes it's enough the GITHUB_TOKEN available in the runner
It's also an option :)

[tomkerkhove]

Yes, that was what I was suggesting indeed - This feels better.

[JorTurFer]

For me it's totally enough. If it's another package, we could delete them manually from time to time if we want
Do you agree with this @zroubalik (with not delete them during the execution)?

[zroubalik]

For me it's totally enough. If it's another package, we could delete them manually from time to time if we want Do you agree with this @zroubalik (with not delete them during the execution)?

Ack, It's not my storage 😄

[JorTurFer]

I've just deleted the steps of removing both images, I don't have any other change pending

[zroubalik]

[zroubalik]

Let's wait until the build-tools image is being built.

This is awesome @JorTurFer!!! I have been hoping to have something like this for more than a year :)