This repository has been archived by the owner on Aug 19, 2024. It is now read-only.
-
Notifications
You must be signed in to change notification settings - Fork 15
Add E2E tests using our examples against real clusters #204
Merged
openshift-merge-bot
merged 9 commits into
janus-idp:main
from
rm3l:90-ci-end-to-end-testing-against-real-cluster
Feb 26, 2024
Merged
Changes from all commits
Commits
Show all changes
9 commits
Select commit
Hold shift + click to select a range
7fa70f5
Add E2E tests against our examples on real clusters
rm3l fc87e04
Try running E2E tests on PRs by leveraging the already built operator…
rm3l 8e0f4a7
Revert "Try running E2E tests on PRs by leveraging the already built …
rm3l a452881
Check if image exists locally before trying to export an archive
rm3l efbe7c2
Update README.md
rm3l 8b220c3
Ignore gosec warnings in test code
rm3l ff4daa1
Clarify in README that a connection to a cluster in the current kubec…
rm3l f10c8c6
Increase timeout when waiting for controller to be up
rm3l 2b5b821
fixup! Clarify in README that a connection to a cluster in the curren…
rm3l File filter
Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
There are no files selected for viewing
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,82 @@ | ||
| name: Nightly checks | ||
|
|
||
| on: | ||
| # workflow_dispatch so that it can be triggered manually if needed | ||
| workflow_dispatch: | ||
| schedule: | ||
| - cron: "34 23 * * *" | ||
|
|
||
| jobs: | ||
| e2e-tests: | ||
| runs-on: ubuntu-latest | ||
| strategy: | ||
| fail-fast: false | ||
| matrix: | ||
| branch: [ main, 1.1.x ] | ||
| name: E2E Tests - ${{ matrix.branch }} | ||
| concurrency: | ||
| group: ${{ github.workflow }}-${{ matrix.branch }} | ||
| cancel-in-progress: true | ||
| env: | ||
| CONTAINER_ENGINE: podman | ||
| steps: | ||
| - uses: actions/checkout@v4 # default branch will be checked out by default on scheduled workflows | ||
| with: | ||
| fetch-depth: 0 | ||
|
|
||
| - if: ${{ matrix.branch != 'main' }} | ||
| name: Checkout ${{ matrix.branch }} branch | ||
| run: git switch ${{ matrix.branch }} | ||
|
|
||
| # check changes in this commit for regex include and exclude matches; pipe to an env var | ||
| - name: Check for changes to build | ||
| run: | | ||
| # don't fail if nothing returned by grep | ||
| set +e | ||
| CHANGES="$(git diff --name-only HEAD~1 | \ | ||
| grep -E "workflows/.+-container-build.yaml|Makefile|bundle/|config/|go.mod|go.sum|.+\.go|docker/|\.dockerignore" | \ | ||
| grep -v -E ".+_test.go|/.rhdh/")"; | ||
| echo "Changed files for this commit:" | ||
| echo "==============================" | ||
| echo "$CHANGES" | ||
| echo "==============================" | ||
| { | ||
| echo 'CHANGES<<EOF' | ||
| echo $CHANGES | ||
| echo EOF | ||
| } >> "$GITHUB_ENV" | ||
|
|
||
| - name: Determine built operator image | ||
| # run this stage only if there are changes that match the includes and not the excludes | ||
| if: ${{ env.CHANGES != '' }} | ||
| run: | | ||
| SHORT_SHA=$(git rev-parse --short HEAD) | ||
| BASE_VERSION=$(grep -E "^VERSION \?=" Makefile | sed -r -e "s/.+= //") # 0.1.0 | ||
| echo "OPERATOR_IMAGE=quay.io/janus-idp/operator:${BASE_VERSION}-${SHORT_SHA}" >> $GITHUB_ENV | ||
|
|
||
| - name: Wait until image exists in registry or timeout is reached | ||
| # run this stage only if there are changes that match the includes and not the excludes | ||
| if: ${{ env.CHANGES != '' }} | ||
| timeout-minutes: 10 | ||
| run: | | ||
| echo "Waiting until operator image is found or timeout expires: ${{ env.OPERATOR_IMAGE }}..." | ||
| until ${CONTAINER_ENGINE} image pull "${{ env.OPERATOR_IMAGE }}"; do | ||
| sleep 2 | ||
| echo ... | ||
| done | ||
| echo "... operator image found: ${{ env.OPERATOR_IMAGE }}." | ||
|
|
||
| - name: Start Minikube | ||
| # run this stage only if there are changes that match the includes and not the excludes | ||
| if: ${{ env.CHANGES != '' }} | ||
| uses: medyagh/setup-minikube@606b71970c783154fe49b711486c717f5780f485 # v0.0.15 | ||
| with: | ||
| addons: ingress | ||
|
|
||
| - name: Run E2E tests | ||
| # run this stage only if there are changes that match the includes and not the excludes | ||
| if: ${{ env.CHANGES != '' }} | ||
| env: | ||
| BACKSTAGE_OPERATOR_TESTS_PLATFORM: minikube | ||
| IMG: ${{ env.OPERATOR_IMAGE }} | ||
| run: make test-e2e | ||
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,146 @@ | ||
| ## End-to-end tests | ||
|
|
||
| The end-to-end tests use the [Ginkgo framework](https://onsi.github.io/ginkgo/) and allow to test the operator against a real cluster in the following scenarios: | ||
| - building and deploying the operator image off of the current code | ||
| - using a specific image or a specific downstream build | ||
|
|
||
| Deployment of the operator itself can be done by: | ||
| - deploying with or without OLM, | ||
| - or deploying the downstream bundle in both online and air-gapped scenarios | ||
|
|
||
| To run the end-to-end tests, make sure you have an active connection to a cluster in your current [kubeconfig](https://kubernetes.io/docs/concepts/configuration/organize-cluster-access-kubeconfig/) and run: | ||
| ```shell | ||
| # Check your current context | ||
| $ kubectl config current-context | ||
| $ make test-e2e | ||
| ``` | ||
|
|
||
| ### Configuration | ||
|
|
||
| The behavior is configurable using the following environment variables: | ||
|
|
||
| | Name | Type | Description | Default value | Example | | ||
| |------------------------------------------------------------------------------------------------|--------|-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|---------------------------------------------------|---------------------------------------------------------| | ||
| | `BACKSTAGE_OPERATOR_TEST_MODE` | string | The test mode:<br>- if not set, it will call `make deploy`<br>- `olm`: it will call `make deploy-olm`<br>- `rhdh-latest` or `rhdh-next`: it will install the operator using the [`install-rhdh-catalog-source.sh`](../../.rhdh/scripts/install-rhdh-catalog-source.sh) script<br>- `rhdh-airgap`: it will install the operator using the [`prepare-restricted-environment.sh`](../../.rhdh/scripts/prepare-restricted-environment.sh) script. | | `rhdh-latest` | | ||
| | `IMG` (or any variables from the Makefile that are used by `make deploy` or `make deploy-olm`) | string | The image to use. Relevant if `BACKSTAGE_OPERATOR_TEST_MODE` is not set or set to `olm` | `VERSION` defined in [`Makefile`](../../Makefile) | `quay.io/janus-idp/operator:0.0.1-latest` | | ||
| | `BACKSTAGE_OPERATOR_TESTS_BUILD_IMAGES` | bool | If set to `true`, it will build the operator image with `make image-build`.<br>Relevant if `BACKSTAGE_OPERATOR_TEST_MODE` is not set or set to `olm`. | | `false` | | ||
| | `BACKSTAGE_OPERATOR_TESTS_PUSH_IMAGES` | bool | If set to `true`, it will push the operator image with `make image-push`.<br>Relevant if `BACKSTAGE_OPERATOR_TEST_MODE` is not set or set to `olm`. | | `false` | | ||
| | `BACKSTAGE_OPERATOR_TESTS_PLATFORM` | string | The platform type, to directly load the operator image if supported instead of pushing it.<br>Relevant if `BACKSTAGE_OPERATOR_TEST_MODE` is not set or set to `olm`.br>Supported values: [`kind`](#building-and-testing-local-changes-on-kind), [`k3d`](#building-and-testing-local-changes-on-k3d), [`minikube`](#building-and-testing-local-changes-on-minikube) | | `kind` | | ||
| | `BACKSTAGE_OPERATOR_TESTS_KIND_CLUSTER` | string | Name of the local KinD cluster to use. Relevant only if `BACKSTAGE_OPERATOR_TESTS_PLATFORM` is `kind`. | `kind` | `kind-local-k8s-cluster` | | ||
| | `BACKSTAGE_OPERATOR_TESTS_K3D_CLUSTER` | string | Name of the local k3d cluster to use. Relevant only if `BACKSTAGE_OPERATOR_TESTS_PLATFORM` is `k3d`. | `k3s-default` | `k3d-local-k8s-cluster` | | ||
| | `BACKSTAGE_OPERATOR_TESTS_AIRGAP_INDEX_IMAGE` | string | Index image to use in the airgap scenario.<br>Relevant if `BACKSTAGE_OPERATOR_TEST_MODE` is `rhdh-airgap`. | `quay.io/rhdh/iib:latest-v4.14-x86_64` | `registry.redhat.io/redhat/redhat-operator-index:v4.14` | | ||
| | `BACKSTAGE_OPERATOR_TESTS_AIRGAP_OPERATOR_VERSION` | string | Operator version to use in the airgap scenario.<br>Relevant if `BACKSTAGE_OPERATOR_TEST_MODE` is `rhdh-airgap`. | `v1.1.0` | `v1.1.0` | | ||
| | `BACKSTAGE_OPERATOR_TESTS_AIRGAP_MIRROR_REGISTRY` | string | Existing mirror registry to use in the airgap scenario.<br>Relevant if `BACKSTAGE_OPERATOR_TEST_MODE` is `rhdh-airgap`<br>. | | `my-registry.example.com` | | ||
|
|
||
| ### Examples | ||
|
|
||
| #### Testing the operator available for the VERSION (default) | ||
|
|
||
| In this scenario, you want to run the E2E test suite against the operator image corresponding to the `VERSION` declared in the project [`Makefile`](../../Makefile), which should be publicly available at `quay.io/janus-idp/operator:<VERSION>`. | ||
|
|
||
| This is the default behavior. | ||
|
|
||
| This should work on any Kubernetes or OpenShift cluster: | ||
|
|
||
| ```shell | ||
| $ make test-e2e | ||
| ``` | ||
|
|
||
| #### Testing a specific image (e.g. PR image) | ||
|
|
||
| In this scenario, you want to run the E2E test suite against an existing operator image. | ||
|
|
||
| This should work on any Kubernetes or OpenShift cluster: | ||
|
|
||
| ```shell | ||
| # if the tag is already published and available at the default location: quay.io/janus-idp/operator | ||
| $ make test-e2e VERSION=0.2.0-3d1c1e0 | ||
|
|
||
| # or you can override the full image repo name | ||
| $ make test-e2e IMG=my.registry.example.com/operator:0.2.0-3d1c1e0 | ||
| ``` | ||
|
|
||
| Note that `VERSION` and `IMG` override the respective variables declared in the project [`Makefile`](../../Makefile). | ||
|
|
||
| #### Building and testing local changes on supported local clusters | ||
|
|
||
| In this scenario, you are iterating locally, and want to run the E2E test suite against your local changes. You are already using a local cluster like [`kind`](https://kind.sigs.k8s.io/), [`k3d`](https://k3d.io/) or [`minikube`](https://minikube.sigs.k8s.io/docs/), which provide the ability to import images into the cluster nodes. | ||
|
|
||
| To do so, you can: | ||
| 1. set `BACKSTAGE_OPERATOR_TESTS_BUILD_IMAGES` to `true`, which will result in building the operator image from the local changes, | ||
| 2. and set `BACKSTAGE_OPERATOR_TESTS_PLATFORM` to a supported local cluster, which will result in loading the image built directly in that cluster (without having to push to a separate registry). | ||
|
|
||
| ##### `kind` | ||
|
|
||
| ```shell | ||
| $ kind create cluster | ||
| $ make test-e2e \ | ||
| BACKSTAGE_OPERATOR_TESTS_BUILD_IMAGES=true \ | ||
| BACKSTAGE_OPERATOR_TESTS_PLATFORM=kind | ||
| ``` | ||
|
|
||
| ##### `k3d` | ||
|
|
||
| ```shell | ||
| $ k3d cluster create | ||
| $ make test-e2e \ | ||
| BACKSTAGE_OPERATOR_TESTS_BUILD_IMAGES=true \ | ||
| BACKSTAGE_OPERATOR_TESTS_PLATFORM=k3d | ||
| ``` | ||
|
|
||
| ##### `minikube` | ||
|
|
||
| ```shell | ||
| $ minikube start | ||
| $ make test-e2e \ | ||
| BACKSTAGE_OPERATOR_TESTS_BUILD_IMAGES=true \ | ||
| BACKSTAGE_OPERATOR_TESTS_PLATFORM=minikube | ||
| ``` | ||
|
|
||
| #### Testing a specific version using OLM | ||
|
|
||
| In this scenario, you want to leverage the [Operator Lifecycle Manager (OLM)](https://olm.operatorframework.io/) to deploy the Operator. | ||
|
|
||
| This requires OLM to be installed in the cluster. | ||
|
|
||
| ```shell | ||
| $ make test-e2e BACKSTAGE_OPERATOR_TEST_MODE=olm | ||
| ``` | ||
|
|
||
| #### Testing a downstream build of Red Hat Developer Hub (RHDH) | ||
|
|
||
| In this scenario, you want to run the E2E tests against a downstream build of RHDH. | ||
|
|
||
| This works only against OpenShift clusters. So make sure you are logged in to the OpenShift cluster using the `oc` command. See [Logging in to the OpenShift CLI](https://docs.openshift.com/container-platform/4.14/cli_reference/openshift_cli/getting-started-cli.html#cli-logging-in_cli-developer-commands) for more details. | ||
|
|
||
| You can check your current context by running `oc config current-context` or `kubectl config current-context`. | ||
|
|
||
| If testing a CI build, please follow the instructions in [Installing CI builds of Red Hat Developer Hub](../../.rhdh/docs/installing-ci-builds.adoc) to add your Quay token to the cluster. | ||
|
|
||
| ```shell | ||
| # latest | ||
| $ make test-e2e BACKSTAGE_OPERATOR_TEST_MODE=rhdh-latest | ||
|
|
||
| # or next | ||
| $ make test-e2e BACKSTAGE_OPERATOR_TEST_MODE=rhdh-next | ||
| ``` | ||
|
|
||
| #### Airgap testing of Red Hat Developer Hub (RHDH) | ||
|
|
||
| In this scenario, you want to run the E2E tests against an OpenShift cluster running in a restricted network. For this, the command below will make sure to prepare it by copying all the necessary images to a mirror registry, then deploy the operator. | ||
|
|
||
| Make sure you are logged in to the OpenShift cluster using the `oc` command. See [Logging in to the OpenShift CLI](https://docs.openshift.com/container-platform/4.14/cli_reference/openshift_cli/getting-started-cli.html#cli-logging-in_cli-developer-commands) for more details. | ||
|
|
||
| You can check your current context by running `oc config current-context` or `kubectl config current-context`. | ||
|
|
||
| Also make sure to read the prerequisites in [Installing Red Hat Developer Hub (RHDH) in restricted environments](../../.rhdh/docs/airgap.adoc). | ||
|
|
||
| ```shell | ||
| # if you want to have a mirror registry to be created for you as part of the airgap environment setup | ||
| $ make test-e2e BACKSTAGE_OPERATOR_TEST_MODE=rhdh-airgap | ||
|
|
||
| # or if you already have a mirror registry available and reachable from within your cluster | ||
| $ make test-e2e \ | ||
| BACKSTAGE_OPERATOR_TEST_MODE=rhdh-airgap \ | ||
| BACKSTAGE_OPERATOR_TESTS_AIRGAP_MIRROR_REGISTRY=my-registry.example.com | ||
| ``` |
Oops, something went wrong.
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Here is an example of a similar run (slightly changed to run on this PR, but the idea should be the same): https://github.com/janus-idp/operator/actions/runs/7960039726/job/21728192711?pr=204#step:11:1