OADP-2745: Testing instructions for AWS/IBM Cloud #1153

mateusoliveira43 · 2023-09-12T18:48:05Z

Update docs regarding how to run E2E (end to end) tests locally.

How to test

Prerequisites

AWS or IBM Cloud with OCP cluster

Run

Checkout PR branch locally

git clone -b fix/testing-instructions --single-branch --depth 1 https://github.com/mateusoliveira43/oadp-operator

Read instructions in https://github.com/mateusoliveira43/oadp-operator/blob/fix/testing-instructions/docs/developer/testing/TESTING.md about prerequisites and how to set up environment variables to run tests in AWS or IBM Cloud.

Either

run all tests with make test-e2e, or

run only MySQL application RESTIC test, following these steps

Run E2E tests setup (creates DPA spec that will be used for tests run) (NORMALLY THIS STEP IS NOT NECESSARY, if running make test-e2e)

make test-e2e-setup

Install Ginkgo CLI (NORMALLY THIS STEP IS NOT NECESSARY, if running make test-e2e)

make install-ginkgo

Run MySQL application RESTIC test (this is basically make test-e2e with --ginkgo.focus)

export CLUSTER_TYPE=$(oc get infrastructures cluster -o jsonpath='{.status.platform}' | tr A-Z a-z)
ginkgo run -mod=mod tests/e2e/ -- \
-settings="/tmp/test-settings/oadpcreds" \
-oc_cli="$(which oc)" \
-provider="$CLUSTER_TYPE" \
-creds_secret_ref="cloud-credentials" \
-credentials="$OADP_CRED_FILE" \
-ci_cred_file="$CI_CRED_FILE" \
-velero_namespace="$OADP_TEST_NAMESPACE" \
-velero_instance_name="velero-test" \
-timeout_multiplier=1 \
-artifact_dir="/tmp" \
--ginkgo.label-filter="$(echo "($(echo '! aws && ! gcp && ! azure && ! ibmcloud' | sed -r "s/[&]* [\!] $CLUSTER_TYPE|[\!] $CLUSTER_TYPE [&]*//")) || $CLUSTER_TYPE")" \
--ginkgo.timeout=2h \
--ginkgo.focus="MySQL application RESTIC"

Signed-off-by: Mateus Oliveira <msouzaol@redhat.com>

openshift-ci · 2023-09-12T18:48:09Z

Skipping CI for Draft Pull Request.
If you want CI signal for your change, please convert it to an actual PR.
You can still manually trigger a test run with /test all

Makefile

docs/developer/testing/TESTING.md

Makefile

Signed-off-by: Mateus Oliveira <msouzaol@redhat.com>

docs/developer/testing/TESTING.md

Signed-off-by: Mateus Oliveira <msouzaol@redhat.com>

docs/developer/testing/TESTING.md

go.mod

mateusoliveira43 · 2023-09-18T14:50:41Z

/hold
Want everyone with AWS/IBM Cloud to run steps from PR description before merging, and checking if it works 😄

weshayutin · 2023-09-19T01:27:53Z

nice! Thank you @mateusoliveira43 this got me past my initial error. FYI.. I had to export bucket_file vs.. bucket
e.g.

#export OADP_BUCKET=/home/whayutin/OPENSHIFT/git/OADP/oadp-operator/KEYS/bucket
export OADP_BUCKET_FILE=/home/whayutin/OPENSHIFT/git/OADP/oadp-operator/KEYS/bucket

Thank you! Still hitting something but I need to dig into it. The test is running through :)

weshayutin · 2023-09-19T01:34:21Z

ah nice.. cleaned up MI BUCKET laddi and I'm good to go! Thank you!!!

------------------------------
2023/09/18 19:33:14 Deleting Velero CR
I0918 19:33:15.577040  154123 request.go:690] Waited for 1.000056678s due to client-side throttling, not priority and fairness, request: GET:https://api.cluster-wdhfips0906.wdhfips0906.mg.dog8code.com:6443/apis/operator.openshift.io/v1?timeout=32s

Ran 1 of 25 Specs in 165.641 seconds
SUCCESS! -- 1 Passed | 0 Failed | 1 Pending | 23 Skipped
PASS

Ginkgo ran 1 suite in 2m48.83036395s
Test Suite Passed
[whayutin@thinkdoe oadp-operator]$

mateusoliveira43 · 2023-09-19T11:46:15Z

@weshayutin we can debug together, but from your snippet, I suspect OADP_BUCKET was in the wrong format. It should be the bucket name, not a file path (but yes, you can use OADP_BUCKET_FILE, but I feel the other variable is easier to set up)

mateusoliveira43 · 2023-09-20T14:38:36Z

Makefile

@@ -434,7 +428,11 @@ catalog-build-replaces: opm ## Build a catalog image using replace mode
 catalog-push: ## Push a catalog image.
 	$(MAKE) docker-push IMG=$(CATALOG_IMG)

-OADP_BUCKET = $(shell cat $(OADP_BUCKET_FILE))
+.PHONY: install-ginkgo
+install-ginkgo: # Make sure ginkgo is in $GOPATH/bin


@kaovilai since we are discussing it

Do you know how to install this in ProjectRoot/bin/ instead of $GOPATH/bin?
(we can address this later in another PR as well)

Yes. The code for this is already in the makefile under go-install-tool.

Here's an example: https://github.com/kaovilai/oadp-operator/blob/b09ce0461bc113e3dbb409d52e38b46555e808dc/Makefile#L177-L178

Do you wanna do go-install-tool for ginkgo here? or in a later PR?

will try on this later. If its simple, will do in this one; otherwise, another

It's very simple like you couldn't believe

I didn't like we have to hardcode version

Like we have to pick a version rather than using latest?

You could @latest as well but we wanna minimize external changes to CI. We should bump versions via explicit PRs IMO.

You could get CLI based on go mod version. like here
https://github.com/mateusoliveira43/oadp-operator/blob/fix/testing-instructions/build/ci-Dockerfile#L8C5-L8C5

RUN go install -mod=mod github.com/onsi/ginkgo/v2/ginkgo

This would install the same CLI version as specified in go.mod

Might need to update go-install-tool to accept -mod=mod

we could use latest (already outdated, we are on 2.11.0 and latest is 2.12.1) and update when needed (do not like this approach because the update necessity will be when CI start breaking and not when we want)

Yeah, was using this approach, but then it install computer wide ($GOPATH/bin instead of PROJECT_ROOT/bin)

going to try to modify go-install-tool , if I can not, going to add a TODO

alright, got it working

# go-install-tool-mod will 'go install' any package $2 and install it to $1 with same version as from Project's go.mod. PROJECT_DIR := $(shell dirname $(abspath $(lastword $(MAKEFILE_LIST)))) define go-install-tool-mod @[ -f $(1) ] || { \ set -e ;\ TMP_DIR=$$(mktemp -d) ;\ cd $$TMP_DIR ;\ cp $(PROJECT_DIR)/go.mod . ;\ echo "Downloading $(2)" ;\ GOBIN=$(PROJECT_DIR)/bin go install -mod=mod $(2) ;\ rm -rf $$TMP_DIR ;\ } endef

But then, this https://github.com/openshift/oadp-operator/blob/master/build/ci-Dockerfile#L8 would be unnecessary, right?

If so, I propose we do this in a separate PR to also change Dockerfile

mrnold

I tried these instructions on a newly installed system with a newly installed AWS cluster, and got reasonable test results (just two failures from unrelated temporary VolumeSnapshot errors). Looks good to me.

Signed-off-by: Mateus Oliveira <msouzaol@redhat.com>

mateusoliveira43 · 2023-09-22T15:12:57Z

tests/e2e/dpa_deployment_suite_test.go

@@ -22,7 +22,7 @@ import (
 )

 var _ = Describe("Configuration testing for DPA Custom Resource", func() {
-	provider := Dpa.Spec.BackupLocations[0].Velero.Provider
+	providerFromDPA := Dpa.Spec.BackupLocations[0].Velero.Provider


This change was needed for IBM use case

weshayutin · 2023-09-22T16:17:03Z

Upon further review... I cleaned up my check out.. it's working
+1
Please add the following for a more detailed execution path ( step by step ) in addition to what you already have here :)
as discussed...

make install-ginkgo

make test-e2e-setup

export CLUSTER_TYPE=$(oc get infrastructures cluster -o jsonpath='{.status.platform}' | tr A-Z a-z)
ginkgo run -mod=mod tests/e2e/ -- \
-settings="/tmp/test-settings/oadpcreds" \
-oc_cli="$(which oc)" \
-provider="$CLUSTER_TYPE" \
-creds_secret_ref="cloud-credentials" \
-credentials="$OADP_CRED_FILE" \
-ci_cred_file="$CI_CRED_FILE" \
-velero_namespace="$OADP_TEST_NAMESPACE" \
-velero_instance_name="velero-test" \
-timeout_multiplier=1 \
-artifact_dir="/tmp" \
--ginkgo.label-filter="$(echo "($(echo '! aws && ! gcp && ! azure && ! ibmcloud' | sed -r "s/[&]* [\!] $CLUSTER_TYPE|[\!] $CLUSTER_TYPE [&]*//")) || $CLUSTER_TYPE")" \
--ginkgo.timeout=2h \
--ginkgo.focus="MySQL application RESTIC"

Signed-off-by: Mateus Oliveira <msouzaol@redhat.com>

mateusoliveira43 · 2023-09-23T00:38:00Z

/unhold

openshift-ci-robot · 2023-09-23T00:39:09Z

@mateusoliveira43: This pull request references OADP-2745 which is a valid jira issue.

Warning: The referenced jira issue has an invalid target version for the target branch this PR targets: expected the task to target the "4.15.0" version, but it targets "OADP 1.3.0" instead.

In response to this:

Update docs regarding how to run E2E (end to end) tests locally.

How to test

Prerequisites

AWS or IBM Cloud with OCP cluster

Run

Checkout PR branch locally
git clone -b fix/testing-instructions --single-branch --depth 1 https://github.com/mateusoliveira43/oadp-operator
Read instructions in https://github.com/mateusoliveira43/oadp-operator/blob/fix/testing-instructions/docs/developer/testing/TESTING.md about prerequisites and how to set up environment variables to run tests in AWS or IBM Cloud.

Either

run all tests with make test-e2e, or

run only MySQL application RESTIC test, following these steps

Run E2E tests setup (creates DPA spec that will be used for tests run) (NORMALLY THIS STEP IS NOT NECESSARY, if running make test-e2e)
make test-e2e-setup
Install Ginkgo CLI (NORMALLY THIS STEP IS NOT NECESSARY, if running make test-e2e)
make install-ginkgo
Run MySQL application RESTIC test (this is basically make test-e2e with --ginkgo.focus)
export CLUSTER_TYPE=$(oc get infrastructures cluster -o jsonpath='{.status.platform}' | tr A-Z a-z)
ginkgo run -mod=mod tests/e2e/ -- \
-settings="/tmp/test-settings/oadpcreds" \
-oc_cli="$(which oc)" \
-provider="$CLUSTER_TYPE" \
-creds_secret_ref="cloud-credentials" \
-credentials="$OADP_CRED_FILE" \
-ci_cred_file="$CI_CRED_FILE" \
-velero_namespace="$OADP_TEST_NAMESPACE" \
-velero_instance_name="velero-test" \
-timeout_multiplier=1 \
-artifact_dir="/tmp" \
--ginkgo.label-filter="$(echo "($(echo '! aws && ! gcp && ! azure && ! ibmcloud' | sed -r "s/[&]* [\!] $CLUSTER_TYPE|[\!] $CLUSTER_TYPE [&]*//")) || $CLUSTER_TYPE")" \
--ginkgo.timeout=2h \
--ginkgo.focus="MySQL application RESTIC"

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

Signed-off-by: Mateus Oliveira <msouzaol@redhat.com>

mateusoliveira43 · 2023-09-23T23:21:23Z

/retest

mpryc · 2023-09-27T10:14:52Z

I think this can be improved a bit, question if within this PR or separate ones?

There is no OC CLI in the PATH

bin/setup-envtest missing
oc missing.

$ make install-ginkgo
which: no oc in (/home/migi/.local/bin:/home/migi/bin:/usr/lib64/qt-3.3/bin:/usr/local/bin:/usr/local/sbin:/usr/bin:/usr/sbin)
/bin/sh: line 1: get: command not found
$CLUSTER_TYPE is []
bash: line 1: /home/migi/Development/Upstream/OADP/mpryc-oadp-operator/bin/setup-envtest: No such file or directory
go install -v -mod=mod github.com/onsi/ginkgo/v2/ginkgo
go: downloading golang.org/x/tools v0.11.0

There is no OC CLI in the PATH

tests/e2e/scripts/_settings.sh missing
oc missing

$ make test-e2e-setup
which: no oc in (/home/migi/.local/bin:/home/migi/bin:/usr/lib64/qt-3.3/bin:/usr/local/bin:/usr/local/sbin:/usr/bin:/usr/sbin)
/bin/sh: line 1: get: command not found
$CLUSTER_TYPE is []
bash: line 1: /home/migi/Development/Upstream/OADP/mpryc-oadp-operator/bin/setup-envtest: No such file or directory
cat: /var/run/oadp-credentials/new-velero-bucket-name: No such file or directory
mkdir -p /tmp/test-settings
TMP_DIR=/tmp/test-settings \
OPENSHIFT_CI="true" \
PROVIDER="" \
SECRET="cloud-credentials" \
AZURE_RESOURCE_FILE="/var/run/secrets/ci.openshift.io/multi-stage/metadata.json" \
CI_JSON_CRED_FILE="/Users/drajds/.aws/osServicePrincipal.json" \
OADP_JSON_CRED_FILE="/var/run/oadp-credentials/azure-credentials" \
OADP_CRED_FILE="/var/run/oadp-credentials/new-aws-credentials" \
BUCKET="" \
TARGET_CI_CRED_FILE="/Users/drajds/.aws/.awscred" \
VSL_REGION="" \
BSL_REGION="us-east-1" \
BSL_AWS_PROFILE="default" \
/bin/bash "tests/e2e/scripts/_settings.sh"
/bin/bash: tests/e2e/scripts/_settings.sh: No such file or directory

Not logged to the custer:

$ make install-ginkgo
E0927 11:09:56.226559 1999614 memcache.go:238] couldn't get current server API group list: the server has asked for the client to provide credentials
E0927 11:09:56.280106 1999614 memcache.go:238] couldn't get current server API group list: the server has asked for the client to provide credentials
E0927 11:09:56.334531 1999614 memcache.go:238] couldn't get current server API group list: the server has asked for the client to provide credentials
E0927 11:09:56.391443 1999614 memcache.go:238] couldn't get current server API group list: the server has asked for the client to provide credentials
E0927 11:09:56.453311 1999614 memcache.go:238] couldn't get current server API group list: the server has asked for the client to provide credentials
error: You must be logged in to the server (the server has asked for the client to provide credentials)
$CLUSTER_TYPE is []
bash: line 1: /home/migi/Development/Upstream/OADP/mpryc-oadp-operator/bin/setup-envtest: No such file or directory
go install -v -mod=mod github.com/onsi/ginkgo/v2/ginkgo

After being logged to the cluster:

#### After oc login to the cluseter:
$ make install-ginkgo
$CLUSTER_TYPE is [aws]
bash: line 1: /home/migi/Development/Upstream/OADP/mpryc-oadp-operator/bin/setup-envtest: No such file or directory
go install -v -mod=mod github.com/onsi/ginkgo/v2/ginkgo

mateusoliveira43 · 2023-09-27T14:02:57Z

@mpryc we currently do not install oc, like in Pelorus. Would let this improvement to another PR

bin/setup-envtest missing comes from here https://github.com/openshift/oadp-operator/blob/master/Makefile#L172-L178 It is not necessary for E2E, but it runs every time. Would let this improvement to another PR

tests/e2e/scripts/_settings.sh missing was because you were not logged in, but I consider that a logic mistake, will try to fix now

openshift-ci · 2023-09-27T16:28:59Z

[APPROVALNOTIFIER] This PR is APPROVED

Approval requirements bypassed by manually added approval.

This pull-request has been approved by: mateusoliveira43, mrnold

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

OWNERS

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

openshift-ci-robot · 2023-09-27T16:33:38Z

/retest-required

Remaining retests: 0 against base HEAD 37e7e26 and 2 for PR HEAD 820a124 in total

kaovilai · 2023-09-27T16:38:04Z

/override ci/prow/4.12-operator-e2e-gcp

openshift-ci · 2023-09-27T16:39:14Z

@kaovilai: Overrode contexts on behalf of kaovilai: ci/prow/4.12-operator-e2e-gcp

In response to this:

/override ci/prow/4.12-operator-e2e-gcp

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

mateusoliveira43 · 2023-09-27T18:52:55Z

/retest

kaovilai · 2023-09-27T18:53:44Z

/override ci/prow/4.12-operator-e2e-gcp

openshift-ci · 2023-09-27T18:54:21Z

@kaovilai: Overrode contexts on behalf of kaovilai: ci/prow/4.12-operator-e2e-gcp

In response to this:

/override ci/prow/4.12-operator-e2e-gcp

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

mateusoliveira43 · 2023-09-27T21:05:05Z

/override ci/prow/4.12-operator-e2e-gcp

openshift-ci · 2023-09-27T21:09:00Z

@mateusoliveira43: mateusoliveira43 unauthorized: /override is restricted to Repo administrators, approvers in top level OWNERS file, and the following github teams:openshift: openshift-release-oversight.

In response to this:

/override ci/prow/4.12-operator-e2e-gcp

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

kaovilai · 2023-09-27T22:42:25Z

/override ci/prow/4.12-operator-e2e-gcp

openshift-ci · 2023-09-27T22:47:26Z

@kaovilai: Overrode contexts on behalf of kaovilai: ci/prow/4.12-operator-e2e-gcp

In response to this:

/override ci/prow/4.12-operator-e2e-gcp

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

openshift-ci · 2023-09-27T22:47:33Z

@mateusoliveira43: all tests passed!

Full PR test history. Your PR dashboard.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository. I understand the commands that are listed here.

fix: Testing instructions

eb2d7a9

Signed-off-by: Mateus Oliveira <msouzaol@redhat.com>

openshift-ci bot added the do-not-merge/work-in-progress Indicates that a PR should not merge because it is a work in progress. label Sep 12, 2023

mateusoliveira43 commented Sep 12, 2023

View reviewed changes

Makefile Outdated Show resolved Hide resolved

mateusoliveira43 commented Sep 12, 2023

View reviewed changes

docs/developer/testing/TESTING.md Outdated Show resolved Hide resolved

kaovilai reviewed Sep 12, 2023

View reviewed changes

Makefile Show resolved Hide resolved

fixup! fix: Testing instructions

69bb59a

Signed-off-by: Mateus Oliveira <msouzaol@redhat.com>

kaovilai requested changes Sep 15, 2023

View reviewed changes

docs/developer/testing/TESTING.md Outdated Show resolved Hide resolved

docs/developer/testing/TESTING.md Outdated Show resolved Hide resolved

docs/developer/testing/TESTING.md Show resolved Hide resolved

docs/developer/testing/TESTING.md Show resolved Hide resolved

fixup! fix: Testing instructions

a59a63e

Signed-off-by: Mateus Oliveira <msouzaol@redhat.com>

mateusoliveira43 changed the title ~~fix: Testing instructions~~ fix: Testing instructions for AWS/IBM Cloud Sep 18, 2023

mateusoliveira43 commented Sep 18, 2023

View reviewed changes

docs/developer/testing/TESTING.md Outdated Show resolved Hide resolved

mateusoliveira43 commented Sep 18, 2023

View reviewed changes

go.mod Outdated Show resolved Hide resolved

mateusoliveira43 marked this pull request as ready for review September 18, 2023 14:49

openshift-ci bot removed the do-not-merge/work-in-progress Indicates that a PR should not merge because it is a work in progress. label Sep 18, 2023

openshift-ci bot requested review from savitharaghunathan and shubham-pampattiwar September 18, 2023 14:50

openshift-ci bot added the do-not-merge/hold Indicates that a PR should not merge because someone has issued a /hold command. label Sep 18, 2023

mateusoliveira43 commented Sep 20, 2023

View reviewed changes

mrnold approved these changes Sep 22, 2023

View reviewed changes

fixup! fix: Testing instructions

bfaa440

Signed-off-by: Mateus Oliveira <msouzaol@redhat.com>

mateusoliveira43 commented Sep 22, 2023

View reviewed changes

fixup! fix: Testing instructions

bd12778

Signed-off-by: Mateus Oliveira <msouzaol@redhat.com>

openshift-ci bot removed the do-not-merge/hold Indicates that a PR should not merge because someone has issued a /hold command. label Sep 23, 2023

mateusoliveira43 changed the title ~~fix: Testing instructions for AWS/IBM Cloud~~ OADP-2745: Testing instructions for AWS/IBM Cloud Sep 23, 2023

openshift-ci-robot added the jira/valid-reference Indicates that this PR references a valid Jira ticket of any type. label Sep 23, 2023

fixup! fix: Testing instructions

820a124

Signed-off-by: Mateus Oliveira <msouzaol@redhat.com>

weshayutin added approved Indicates a PR has been approved by an approver from all required OWNERS files. lgtm Indicates that a PR is ready to be merged. labels Sep 27, 2023

openshift-merge-robot merged commit a76c5dc into openshift:master Sep 27, 2023
17 checks passed

OADP-2745: Testing instructions for AWS/IBM Cloud #1153

OADP-2745: Testing instructions for AWS/IBM Cloud #1153

Conversation

mateusoliveira43 commented Sep 12, 2023 • edited

How to test

Prerequisites

Run

openshift-ci bot commented Sep 12, 2023

mateusoliveira43 commented Sep 18, 2023

weshayutin commented Sep 19, 2023

weshayutin commented Sep 19, 2023

mateusoliveira43 commented Sep 19, 2023

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

kaovilai Sep 22, 2023 • edited

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

mrnold left a comment • edited

Choose a reason for hiding this comment

Choose a reason for hiding this comment

weshayutin commented Sep 22, 2023 • edited

mateusoliveira43 commented Sep 23, 2023

openshift-ci-robot commented Sep 23, 2023 • edited by openshift-ci bot

How to test

Prerequisites

Run

mateusoliveira43 commented Sep 23, 2023

mpryc commented Sep 27, 2023

mateusoliveira43 commented Sep 27, 2023

openshift-ci bot commented Sep 27, 2023

openshift-ci-robot commented Sep 27, 2023

kaovilai commented Sep 27, 2023

openshift-ci bot commented Sep 27, 2023

mateusoliveira43 commented Sep 27, 2023

kaovilai commented Sep 27, 2023

openshift-ci bot commented Sep 27, 2023

mateusoliveira43 commented Sep 27, 2023

openshift-ci bot commented Sep 27, 2023

kaovilai commented Sep 27, 2023

openshift-ci bot commented Sep 27, 2023

openshift-ci bot commented Sep 27, 2023

mateusoliveira43 commented Sep 12, 2023 •

edited

kaovilai Sep 22, 2023 •

edited

mrnold left a comment •

edited

weshayutin commented Sep 22, 2023 •

edited

openshift-ci-robot commented Sep 23, 2023 •

edited by openshift-ci bot