Add an E2E test framework to test Velero across cloud platforms #3060

dsu-igeek · 2020-11-09T23:16:05Z

Uses distributed data generator

Signed-off-by: Dave Smith-Uchida dsmithuchida@vmware.com

Uses distributed data generator Signed-off-by: Dave Smith-Uchida <dsmithuchida@vmware.com>

…restore Signed-off-by: Dave Smith-Uchida <dsmithuchida@vmware.com>

Signed-off-by: Ashish Amarnath <ashisham@vmware.com>

Signed-off-by: Dave Smith-Uchida <dsmithuchida@vmware.com>

Signed-off-by: Ashish Amarnath <ashisham@vmware.com>

…elero into e2e-tests-10-05-2020 # Conflicts: # test/e2e/backup_test.go

Signed-off-by: Dave Smith-Uchida <dsmithuchida@vmware.com>

Signed-off-by: Ashish Amarnath <ashisham@vmware.com>

test/e2e/backup_test.go

test/e2e/e2e_suite_test.go

ashish-amarnath · 2020-11-13T02:13:39Z

Marking this as ready for review.

ashish-amarnath

Approving my own changes.

go.mod

test/e2e/Makefile

test/e2e/README.md

…elero into e2e-tests-10-05-2020 # Conflicts: # .gitignore # test/e2e/Makefile # test/e2e/README.md # test/e2e/backup_test.go # test/e2e/common.go

…e GOPATH/bin Updated to ginkgo 1.14.2 Put cobra back to v0.0.7 Signed-off-by: Dave Smith-Uchida <dsmithuchida@vmware.com>

ashish-amarnath

I added a couple of minor and non-blocking comments. But overall LGTM.

.gitignore

test/e2e/Makefile

…d ginkgo from .gitignore Signed-off-by: Dave Smith-Uchida <dsmithuchida@vmware.com>

Signed-off-by: Ashish Amarnath <ashisham@vmware.com>

ashish-amarnath

Though there is still work to be done here we should get this merged as it adds a foundation to add more tests and making this slightly more robust.

I am already working on a PR to build on top of this and improving things.

@nrb @carlisia @zubron Let's get this merged.

zubron

Thanks for your efforts on this, @dsu-igeek and @ashish-amarnath! I've left some comments but they are all mostly cosmetic things that I think would just make the code a bit easier to read and the results easier to interpret. They aren't blocking, just things that would be good to address in further changes :)

I agree with @ashish-amarnath, I think it's fine to merge and iterate on 👍

zubron · 2020-11-24T17:02:11Z

test/e2e/kibishii_utils.go

+	stdoutReader := bufio.NewReader(stdoutPipe)
+	var readErr error
+	for true {
+		buf, isPrefix, err := stdoutReader.ReadLine()


The docs for ReadLine recommend most users to use ReadString or use a Scanner. I think we could use a Scanner here and it would simplify this loop (we wouldn't need to check the isPrefix value for example). Same applies for the loop in VerifyData below.

zubron · 2020-11-24T18:30:14Z

test/e2e/common.go

+}
+
+func CreateNamespace(ctx context.Context, namespace string) error {
+	// TODO - should we talk directly to the API server?


That would be my preference. It can be a bit of a pain setting up a client, but I think it's clearer than setting up and running external commands when we don't need to. It might make the error handling easier to manage. For example, I tried running this locally and the test failed the first time. On subsequent runs, the kibishii namespace already existed so this returned an error, however there wasn't any useful information along with it, it only printed an *exec.ExitError.

zubron · 2020-11-24T18:49:27Z

test/e2e/velero_utils.go

+		return err
+	}
+
+	jsonBuf := make([]byte, 16*1024) // If the YAML is bigger than 16K, there's probably something bad happening


Rather than creating the slice with a size, we could use a bytes.Buffer here. You can also set the buffer to be Stdout writer on the command directly, rather than needing to get the StdoutPipe and then read into the created slice. Here's an example from the exec.Command docs: https://play.golang.org/p/eGFQtL39Qve

zubron · 2020-11-24T18:52:46Z

test/e2e/backup_test.go

+				Expect(err).NotTo(HaveOccurred())
+				backupName = "backup-" + backupUUID.String()
+				restoreName = "restore-" + backupUUID.String()
+				println("backupName = " + backupName)


I'm sure this will be changed in later commits, but just a general comment that we should use the fmt package rather than builtins.

zubron · 2020-11-24T19:07:49Z

test/e2e/backup_test.go

+				println("running kibishii generate")
+				timeoutCTX, _ = context.WithTimeout(context.Background(), time.Minute*60)
+
+				err = GenerateData(timeoutCTX, kibishiNamespace, 2, 10, 10, 1024, 1024, 0, 2)


It might be nice to wrap the arguments that are needed for kibishii into a struct so that it's a bit clearer what each of these integers represent.

…re-tanzu#3060) * Basic end-to-end tests, generate data/backup/remove/restore/verify Uses distributed data generator Signed-off-by: Dave Smith-Uchida <dsmithuchida@vmware.com> * Moved backup/restore into velero_utils, started using a name for the restore Signed-off-by: Dave Smith-Uchida <dsmithuchida@vmware.com> * remove checked in binary and update test/e2e Makefile Signed-off-by: Ashish Amarnath <ashisham@vmware.com> * Ran make update Signed-off-by: Dave Smith-Uchida <dsmithuchida@vmware.com> * Save Signed-off-by: Ashish Amarnath <ashisham@vmware.com> * Ran make update Signed-off-by: Dave Smith-Uchida <dsmithuchida@vmware.com> * Basic end-to-end test, generate data/backup/remove/restore/verify Uses distributed data generator Signed-off-by: Dave Smith-Uchida <dsmithuchida@vmware.com> * Changed tests/e2e Makefile to just use go get to install ginkgo in the GOPATH/bin Updated to ginkgo 1.14.2 Put cobra back to v0.0.7 Signed-off-by: Dave Smith-Uchida <dsmithuchida@vmware.com> * Added CLOUD_PLATFORM env variable to Makefile, updated README, removed ginkgo from .gitignore Signed-off-by: Dave Smith-Uchida <dsmithuchida@vmware.com> * choose velero CLI binary based on local env Signed-off-by: Ashish Amarnath <ashisham@vmware.com> Co-authored-by: Ashish Amarnath <ashisham@vmware.com>

github-actions bot requested review from ashish-amarnath, carlisia, nrb and zubron November 9, 2020 23:16

github-actions bot assigned dsu-igeek Nov 9, 2020

dsu-igeek force-pushed the e2e-tests-10-05-2020 branch 2 times, most recently from f3898a1 to 18f5b6e Compare November 9, 2020 23:32

Basic end-to-end tests, generate data/backup/remove/restore/verify

a85f968

Uses distributed data generator Signed-off-by: Dave Smith-Uchida <dsmithuchida@vmware.com>

dsu-igeek force-pushed the e2e-tests-10-05-2020 branch from 18f5b6e to a85f968 Compare November 9, 2020 23:39

dsu-igeek changed the title ~~Basic end-to-end tests, generate data/backup/remove/restore/verify~~ WIP - Basic end-to-end tests, generate data/backup/remove/restore/verify Nov 9, 2020

carlisia marked this pull request as draft November 10, 2020 00:30

dsu-igeek and others added 7 commits November 9, 2020 21:35

Moved backup/restore into velero_utils, started using a name for the …

086321e

…restore Signed-off-by: Dave Smith-Uchida <dsmithuchida@vmware.com>

remove checked in binary and update test/e2e Makefile

4859cc0

Signed-off-by: Ashish Amarnath <ashisham@vmware.com>

Ran make update

bd30cd4

Signed-off-by: Dave Smith-Uchida <dsmithuchida@vmware.com>

Save

b2468dd

Signed-off-by: Ashish Amarnath <ashisham@vmware.com>

Merge branch 'e2e-tests-10-05-2020' of https://github.com/dsu-igeek/v…

513176b

…elero into e2e-tests-10-05-2020 # Conflicts: # test/e2e/backup_test.go

Ran make update

5f9c55b

Signed-off-by: Dave Smith-Uchida <dsmithuchida@vmware.com>

add framework to run platform based velero tests

f229d96

Signed-off-by: Ashish Amarnath <ashisham@vmware.com>

ashish-amarnath force-pushed the e2e-tests-10-05-2020 branch from 6a2300f to f229d96 Compare November 13, 2020 02:06

ashish-amarnath changed the title ~~WIP - Basic end-to-end tests, generate data/backup/remove/restore/verify~~ Add an E2E test framework to test Velero across cloud platforms Nov 13, 2020

ashish-amarnath reviewed Nov 13, 2020

View reviewed changes

test/e2e/backup_test.go Outdated Show resolved Hide resolved

ashish-amarnath reviewed Nov 13, 2020

View reviewed changes

test/e2e/e2e_suite_test.go Show resolved Hide resolved

ashish-amarnath marked this pull request as ready for review November 13, 2020 02:13

github-actions bot requested a review from ashish-amarnath November 13, 2020 02:13

ashish-amarnath previously approved these changes Nov 13, 2020

View reviewed changes

ashish-amarnath added this to the v1.6.0 milestone Nov 13, 2020

ashish-amarnath added this to In progress in v1.6.0 via automation Nov 13, 2020

ashish-amarnath mentioned this pull request Nov 13, 2020

Create basic end-to-end test with KinD #2131

Closed

ashish-amarnath moved this from In progress to Review in progress in v1.6.0 Nov 13, 2020

nrb reviewed Nov 19, 2020

View reviewed changes

go.mod Outdated Show resolved Hide resolved

nrb reviewed Nov 19, 2020

View reviewed changes

test/e2e/Makefile Outdated Show resolved Hide resolved

ashish-amarnath reviewed Nov 19, 2020

View reviewed changes

test/e2e/README.md Outdated Show resolved Hide resolved

ashish-amarnath force-pushed the e2e-tests-10-05-2020 branch from e6eccc3 to f229d96 Compare November 19, 2020 23:32

github-actions bot requested review from ashish-amarnath, carlisia and nrb November 19, 2020 23:33

dsu-igeek force-pushed the e2e-tests-10-05-2020 branch from f229d96 to e6eccc3 Compare November 19, 2020 23:46

ashish-amarnath and others added 2 commits November 19, 2020 16:02

Merge branch 'e2e-tests-10-05-2020' of https://github.com/dsu-igeek/v…

b5dd17c

…elero into e2e-tests-10-05-2020 # Conflicts: # .gitignore # test/e2e/Makefile # test/e2e/README.md # test/e2e/backup_test.go # test/e2e/common.go

Changed tests/e2e Makefile to just use go get to install ginkgo in th…

6644b9a

…e GOPATH/bin Updated to ginkgo 1.14.2 Put cobra back to v0.0.7 Signed-off-by: Dave Smith-Uchida <dsmithuchida@vmware.com>

dsu-igeek force-pushed the e2e-tests-10-05-2020 branch from 63ef7f9 to 6644b9a Compare November 23, 2020 22:39

ashish-amarnath previously approved these changes Nov 23, 2020

View reviewed changes

.gitignore Outdated Show resolved Hide resolved

test/e2e/Makefile Show resolved Hide resolved

Added CLOUD_PLATFORM env variable to Makefile, updated README, remove…

e25a6f3

…d ginkgo from .gitignore Signed-off-by: Dave Smith-Uchida <dsmithuchida@vmware.com>

dsu-igeek dismissed ashish-amarnath’s stale review via e25a6f3 November 24, 2020 01:05

github-actions bot requested a review from ashish-amarnath November 24, 2020 01:05

choose velero CLI binary based on local env

24f82cc

Signed-off-by: Ashish Amarnath <ashisham@vmware.com>

ashish-amarnath approved these changes Nov 24, 2020

View reviewed changes

zubron approved these changes Nov 24, 2020

View reviewed changes

v1.6.0 automation moved this from Review in progress to Reviewer approved Nov 24, 2020

zubron merged commit aa47309 into vmware-tanzu:main Nov 24, 2020

v1.6.0 automation moved this from Reviewer approved to Done Nov 24, 2020

ashish-amarnath mentioned this pull request Dec 3, 2020

✨ Add more E2E tests and improvement #3111

Merged

georgettica mentioned this pull request Dec 23, 2020

georgettica/bump external snapshotter version #3201

Closed

dsu-igeek deleted the e2e-tests-10-05-2020 branch March 18, 2022 22:04

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Add an E2E test framework to test Velero across cloud platforms #3060

Add an E2E test framework to test Velero across cloud platforms #3060

dsu-igeek commented Nov 9, 2020

ashish-amarnath commented Nov 13, 2020

ashish-amarnath left a comment

ashish-amarnath left a comment

ashish-amarnath left a comment

zubron left a comment •

edited

zubron Nov 24, 2020

zubron Nov 24, 2020

zubron Nov 24, 2020

zubron Nov 24, 2020

zubron Nov 24, 2020

Add an E2E test framework to test Velero across cloud platforms #3060

Add an E2E test framework to test Velero across cloud platforms #3060

Conversation

dsu-igeek commented Nov 9, 2020

ashish-amarnath commented Nov 13, 2020

ashish-amarnath left a comment

Choose a reason for hiding this comment

ashish-amarnath left a comment

Choose a reason for hiding this comment

ashish-amarnath left a comment

Choose a reason for hiding this comment

zubron left a comment • edited

Choose a reason for hiding this comment

zubron Nov 24, 2020

Choose a reason for hiding this comment

zubron Nov 24, 2020

Choose a reason for hiding this comment

zubron Nov 24, 2020

Choose a reason for hiding this comment

zubron Nov 24, 2020

Choose a reason for hiding this comment

zubron Nov 24, 2020

Choose a reason for hiding this comment

zubron left a comment •

edited