test: minor version upgrade e2e testing #2797

cmwaters · 2023-10-31T16:18:04Z

Overview

Closes: #313

This PR introduces a test for checking compatibility and the running of upgrades of minor versions.

It reads all minor versions tagged in git. Each node begins on a random version.

Each node then individually performs upgrades, going down and spinning back up on the new randomly chosen version (can be downgrades as well)

Currently we only check that their is no app version mismatch and the network is able to continually build blocks

Checklist

New and updated code has appropriate documentation
New and updated code has new and/or updated testing
Required CI checks are passing
Visual proof for any user facing features like CLI or documentation updates
Linked issues closed with keywords

rootulp

Great work! When I run this locally, I encounter:

$ make test-e2e
--> Running end to end tests
=== RUN   TestE2ESimple
    simple_test.go:34: Running simple e2e test version latest
    simple_test.go:37:
        	Error Trace:	/Users/rootulp/git/rootulp/celestia/celestia-app/test/e2e/simple_test.go:37
        	Error:      	Received unexpected error:
        	            	retrieving the Kubernetes config: stat /Users/rootulp/.kube/config: no such file or directory
        	Test:       	TestE2ESimple
--- FAIL: TestE2ESimple (0.00s)
=== RUN   TestMinorVersionCompatibility
    upgrade_test.go:27: skipping e2e test: E2E_VERSION not set
--- SKIP: TestMinorVersionCompatibility (0.00s)

What prerequisites do contributors need to perform prior to running make test-e2e locally? It looks like I need some kube config and I'm not sure why the the e2e test is getting skipped

Makefile

test/e2e/simple_test.go

test/e2e/upgrade_test.go

test/e2e/versions_test.go

cmwaters · 2023-11-01T13:29:00Z

What prerequisites do contributors need to perform prior to running make test-e2e locally? It looks like I need some kube config and I'm not sure why the the e2e test is getting skipped

Yes you need a .kube/config setup. You can ask samuel for it. Also you need the E2E environment variable to run the tests (I purposefully didn't want go test ./... to run these tests). It's best to just run make test-e2e

rootulp · 2023-11-01T13:35:18Z

Can we add the .kube/config to this repo and document where maintainers should place it on their local machines? Doesn't strictly have to be in this PR.
hmm I tried make test-e2e locally. I can retry after the env variable merging

cmwaters · 2023-11-01T15:57:02Z

Can we add the .kube/config to this repo and document where maintainers should place it on their local machines? Doesn't strictly have to be in this PR.

no because this shouldn't be public. Ideally we work out how to support locally running the tests but we don't have that at the moment

codecov-commenter · 2023-11-01T17:27:13Z

Codecov Report

Merging #2797 (215a562) into main (8e5faa7) will increase coverage by 0.22%.
The diff coverage is 71.05%.

@@            Coverage Diff             @@
##             main    #2797      +/-   ##
==========================================
+ Coverage   19.44%   19.67%   +0.22%     
==========================================
  Files         143      144       +1     
  Lines       17320    17395      +75     
==========================================
+ Hits         3368     3422      +54     
- Misses      13633    13653      +20     
- Partials      319      320       +1

Files	Coverage Δ
test/e2e/node.go	`0.00% <0.00%> (ø)`
test/e2e/versions.go	`76.05% <76.05%> (ø)`

rootulp · 2023-11-02T08:38:42Z

no because this shouldn't be public.

[blocking] in this PR can we please indicate that make test-e2e isn't expected to work for external contributors.

evan-forbes

had some questions, but its super dope we're getting to use knuu for compatibility tests

test/e2e/versions.go

evan-forbes · 2023-11-02T15:04:05Z

test/e2e/upgrade_test.go

+		}
+		newVersion := versions.Random(r).String()
+		t.Log("Upgrading node", "node", i%numNodes, "version", newVersion)
+		err := testnet.Node(i % numNodes).Upgrade(newVersion)


does this function block until the node is back up?

I actually don't know. It calls something in knuu (cc @smuu)

Yes the current implementation blocks until the node is back up

test/e2e/upgrade_test.go

evan-forbes · 2023-11-02T15:13:52Z

test/e2e/upgrade_test.go

+		// FIXME: skip the first node because we need them available to
+		// submit txs
+		if i%numNodes == 0 {
+			continue
+		}


is skipping the first node also the mechanism that is testing that the versions are compatible?

I don't quite understand. We need the first node to always be up because we're constantly submitting transactions. If txsim can't submit a transaction it exits.

I see now that the test has been updated for each upgraded node to wait a few heights. I was curious what the mechanism was to ensure that different versions were running concurrently

test/e2e/upgrade_test.go

Co-authored-by: Evan Forbes <42654277+evan-forbes@users.noreply.github.com>

…lestia-app into cal/minor-upgrade-testing

rootulp

When I run this locally, I observe:

[not blocking] verbose logging from test e2e simple. Can we please redirect that logging so that the test results are easier to read?
[blocking] TestMinorVersionCompatibility gets skipped. TBH not sure why b/c the Makefile looks correct but my shell doesn't ever have these values populated.

$ make test-e2e
--> Running end to end tests
=== RUN   TestE2ESimple
    simple_test.go:34: Running simple e2e test version latest
< verbose logging from test e2e simple> 
--- PASS: TestE2ESimple (318.32s)
=== RUN   TestMinorVersionCompatibility
    upgrade_test.go:29: skipping e2e test: E2E_VERSION not set
--- SKIP: TestMinorVersionCompatibility (0.00s)
=== RUN   TestVersionParsing
--- PASS: TestVersionParsing (0.00s)
=== RUN   TestFilterMajorVersions
--- PASS: TestFilterMajorVersions (0.00s)
=== RUN   TestOrder
--- PASS: TestOrder (0.00s)
=== RUN   TestOrderOfReleaseCandidates
--- PASS: TestOrderOfReleaseCandidates (0.00s)
PASS
ok  	github.com/celestiaorg/celestia-app/test/e2e	318.931s

$ echo $E2E_VERSIONS

$ echo $E2E_VERSION

Makefile

test/e2e/upgrade_test.go

cmwaters · 2023-11-06T13:28:40Z

[not blocking] verbose logging from test e2e simple. Can we please redirect that logging so that the test results are easier to read?

Yeah I can remove that. That was initially just to let me know it was working but should be fine to silence them now

[blocking] TestMinorVersionCompatibility gets skipped. TBH not sure why b/c the Makefile looks correct but my shell doesn't ever have these values populated.

Let me look into it

Co-authored-by: Rootul P <rootulp@gmail.com>

evan-forbes

lgtm 👍

cmwaters · 2023-11-06T14:21:07Z

@rootulp do you want to try it now

rootulp

testing upgrade test now and it appears to be running 👍

Update: I quit after 10 mins b/c it didn't seem to be making progress

$ make test-e2e
--> Running end to end tests
=== RUN   TestE2ESimple
    simple_test.go:25:
--- SKIP: TestE2ESimple (0.00s)
=== RUN   TestMinorVersionCompatibility
versions v1.0.0	v1.1.0	v1.2.0	v1.3.0    upgrade_test.go:40: Running minor version compatibility test versions v1.0.0	v1.1.0	v1.2.0	v1.3.0
    upgrade_test.go:58: Starting node node 0 version v1.1.0
    upgrade_test.go:58: Starting node node 1 version v1.3.0
    upgrade_test.go:58: Starting node node 2 version v1.0.0
    upgrade_test.go:58: Starting node node 3 version v1.2.0








^Csignal: interrupt
FAIL	github.com/celestiaorg/celestia-app/test/e2e	585.399s
make: *** [test-e2e] Error 1

test/e2e/simple_test.go

test/e2e/versions_test.go

Co-authored-by: Rootul P <rootulp@gmail.com>

smuu

This looks great!

cmwaters added 2 commits October 31, 2023 17:11

test: add minor version upgrade testing

65b954c

require version to be set

2477ef4

cmwaters requested review from evan-forbes and rootulp as code owners October 31, 2023 16:18

celestia-bot requested a review from a team October 31, 2023 16:18

remove print statement

1c7ad45

cmwaters self-assigned this Oct 31, 2023

rootulp reviewed Nov 1, 2023

View reviewed changes

Makefile Outdated Show resolved Hide resolved

test/e2e/simple_test.go Outdated Show resolved Hide resolved

test/e2e/upgrade_test.go Outdated Show resolved Hide resolved

test/e2e/upgrade_test.go Show resolved Hide resolved

test/e2e/versions_test.go Show resolved Hide resolved

incorporate suggestions

4ac4bdb

celestia-bot requested a review from a team November 1, 2023 17:22

evan-forbes reviewed Nov 2, 2023

View reviewed changes

Simplify IsGreater

93a42f9

Co-authored-by: Evan Forbes <42654277+evan-forbes@users.noreply.github.com>

celestia-bot requested a review from a team November 2, 2023 16:08

cmwaters added 3 commits November 6, 2023 12:13

add assertions that the network progresses

f36db3a

Merge branch 'cal/minor-upgrade-testing' of github.com:celestiaorg/ce…

4285736

…lestia-app into cal/minor-upgrade-testing

Merge branch 'main' into cal/minor-upgrade-testing

c13fa61

rootulp self-requested a review November 6, 2023 12:02

rootulp reviewed Nov 6, 2023

View reviewed changes

Makefile Outdated Show resolved Hide resolved

test/e2e/upgrade_test.go Outdated Show resolved Hide resolved

Apply suggestions from code review

a4eb85d

Co-authored-by: Rootul P <rootulp@gmail.com>

celestia-bot requested a review from a team November 6, 2023 13:36

evan-forbes previously approved these changes Nov 6, 2023

View reviewed changes

fix parsing e2e_version env

59c3b5d

cmwaters dismissed evan-forbes’s stale review via 59c3b5d November 6, 2023 14:14

celestia-bot requested a review from a team November 6, 2023 14:14

evan-forbes previously approved these changes Nov 6, 2023

View reviewed changes

set global knuu timeout

cfcfa5c

cmwaters dismissed evan-forbes’s stale review via cfcfa5c November 6, 2023 14:20

celestia-bot requested a review from a team November 6, 2023 14:20

Merge branch 'main' into cal/minor-upgrade-testing

215a562

rootulp reviewed Nov 6, 2023

View reviewed changes

test/e2e/simple_test.go Outdated Show resolved Hide resolved

test/e2e/versions_test.go Outdated Show resolved Hide resolved

cmwaters and others added 2 commits November 6, 2023 15:35

Update test/e2e/versions_test.go

4aa82ea

Co-authored-by: Rootul P <rootulp@gmail.com>

Merge branch 'main' into cal/minor-upgrade-testing

ebe2022

celestia-bot requested a review from a team November 6, 2023 14:36

smuu approved these changes Nov 8, 2023

View reviewed changes

evan-forbes approved these changes Nov 8, 2023

View reviewed changes

rootulp approved these changes Nov 8, 2023

View reviewed changes

Merge branch 'main' into cal/minor-upgrade-testing

eae9549

celestia-bot requested a review from a team November 8, 2023 15:06

cmwaters merged commit 75f9393 into main Nov 8, 2023
29 checks passed

cmwaters deleted the cal/minor-upgrade-testing branch November 8, 2023 15:58

evan-forbes mentioned this pull request Nov 15, 2023

E2E Fuzzer Tool for celestia-node tests #1535

Closed

6 tasks

rootulp mentioned this pull request Jan 8, 2024

Add E2E tests for upgrades #2633

Closed

4 tasks

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

test: minor version upgrade e2e testing #2797

test: minor version upgrade e2e testing #2797

cmwaters commented Oct 31, 2023 •

edited

Loading

rootulp left a comment •

edited

Loading

cmwaters commented Nov 1, 2023

rootulp commented Nov 1, 2023

cmwaters commented Nov 1, 2023

codecov-commenter commented Nov 1, 2023 •

edited

Loading

rootulp commented Nov 2, 2023

evan-forbes left a comment

evan-forbes Nov 2, 2023

cmwaters Nov 2, 2023

smuu Nov 8, 2023

evan-forbes Nov 2, 2023

cmwaters Nov 2, 2023

evan-forbes Nov 6, 2023

rootulp left a comment

cmwaters commented Nov 6, 2023

evan-forbes left a comment

cmwaters commented Nov 6, 2023

rootulp left a comment •

edited

Loading

smuu left a comment

test: minor version upgrade e2e testing #2797

test: minor version upgrade e2e testing #2797

Conversation

cmwaters commented Oct 31, 2023 • edited Loading

Overview

Checklist

rootulp left a comment • edited Loading

Choose a reason for hiding this comment

cmwaters commented Nov 1, 2023

rootulp commented Nov 1, 2023

cmwaters commented Nov 1, 2023

codecov-commenter commented Nov 1, 2023 • edited Loading

Codecov Report

rootulp commented Nov 2, 2023

evan-forbes left a comment

Choose a reason for hiding this comment

evan-forbes Nov 2, 2023

Choose a reason for hiding this comment

cmwaters Nov 2, 2023

Choose a reason for hiding this comment

smuu Nov 8, 2023

Choose a reason for hiding this comment

evan-forbes Nov 2, 2023

Choose a reason for hiding this comment

cmwaters Nov 2, 2023

Choose a reason for hiding this comment

evan-forbes Nov 6, 2023

Choose a reason for hiding this comment

rootulp left a comment

Choose a reason for hiding this comment

cmwaters commented Nov 6, 2023

evan-forbes left a comment

Choose a reason for hiding this comment

cmwaters commented Nov 6, 2023

rootulp left a comment • edited Loading

Choose a reason for hiding this comment

smuu left a comment

Choose a reason for hiding this comment

cmwaters commented Oct 31, 2023 •

edited

Loading

rootulp left a comment •

edited

Loading

codecov-commenter commented Nov 1, 2023 •

edited

Loading

rootulp left a comment •

edited

Loading