feat (tests): enable multiple upgrades for automated upgrade tests

CHANGELOG.md

@@ -59,6 +59,7 @@ Ref: https://keepachangelog.com/en/1.0.0/
 
 ### Improvements
 
+- (tests) [#1283](https://github.com/evmos/evmos/pull/1283) Enable multiple upgrades for automated upgrade tests
 - (deps) [#1279](https://github.com/evmos/evmos/pull/1279) Bump Cosmos SDK version to [`v0.46.8-ledger`](https://github.com/evmos/cosmos-sdk/releases/tag/v0.46.8-ledger)
 - (inflation) [#1258](https://github.com/evmos/evmos/pull/1258) Remove unnecessary `Coin` validation and store calls for `Params`
 - (deps) [#1248](https://github.com/evmos/evmos/pull/1248) Use the Informal Systems Tendermint Core fork

Makefile

@@ -330,7 +330,9 @@ build-docs-versioned:
 
 test: test-unit
 test-all: test-unit test-race
-PACKAGES_UNIT=$(shell go list ./... | grep -v '/tests/e2e')
+# For unit tests we don't want to execute the upgrade tests in tests/e2e but
+# we want to include all unit tests in the subfolders (tests/e2e/*)
+PACKAGES_UNIT=$(shell go list ./... | grep -v '/tests/e2e$$')
 TEST_PACKAGES=./...
 TEST_TARGETS := test-unit test-unit-cover test-race
 
@@ -352,7 +354,7 @@ test-upgrade:
 	rm -rf build/.evmosd
 	INITIAL_VERSION=$(INITIAL_VERSION) TARGET_VERSION=$(TARGET_VERSION) \
 	E2E_SKIP_CLEANUP=$(E2E_SKIP_CLEANUP) MOUNT_PATH=$(MOUNT_PATH) CHAIN_ID=$(CHAIN_ID) \
-	go test -v ./tests/e2e/...
+	go test -v ./tests/e2e
 
 run-tests:
 ifneq (,$(shell which tparse 2>/dev/null))

tests/e2e/README.md

@@ -1,6 +1,18 @@
 # End-to-End Testing Suite
 
-The End-to-End (E2E) testing suite provides an environment for running end-to-end tests on Evmos. It is used for testing chain upgrades, as it allows for initializing multiple Evmos chains with different versions.
+The End-to-End (E2E) testing suite provides an environment
+for running end-to-end tests on Evmos.
+It is used for testing chain upgrades,
+as it allows for initializing multiple Evmos chains with different versions.
+
+- [Quick Start](#quick-start)
+- [Upgrade Process](#upgrade-process)
+- [Test Suite Structure](#test-suite-structure)
+    - [`e2e` Package](#e2e-package)
+    - [`upgrade` Package](#upgrade-package)
+    - [Version retrieve](#version-retrieve)
+    - [Testing Results](#testing-results)
+- [Running multiple upgrades](#running-multiple-upgrades)
 
 ### Quick Start
 
@@ -14,125 +26,170 @@ This logic utilizes parameters that can be set manually(if necessary):
 
 ```shell
 # flag to skip containers cleanup after upgrade
-# should be set true with make test-e2e command if you need access to the node after upgrade
+# should be set true with make test-e2e command if you need access to the node
+# after upgrading
 E2E_SKIP_CLEANUP := false
 
-# version of initial evmos node that will be upgraded, tag e.g. 'v9.1.0'
+# version(s) of initial evmos node(s) that will be upgraded, tag e.g. 'v9.1.0'
+# to use multiple upgrades separate the versions with a forward slash, e.g.
+# 'v10.0.1/v11.0.0-rc1'
 INITIAL_VERSION
 
-# version of upgraded evmos node that will replace the initial node, tag e.g. 'v10.0.0'
+# version of upgraded evmos node that will replace the initial node, tag e.g.
+# 'v10.0.0'
 TARGET_VERSION
 
-# mount point for upgraded node container, to mount new node version to previous node state folder
-# by defaullt './build/.evmosd:/root/.evmosd'
-# more info https://docs.docker.com/engine/reference/builder/#volume
+# mount point for the upgraded node container, to mount new node version to
+# previous node state folder. By default this is './build/.evmosd:/root/.evmosd'
+# More info at https://docs.docker.com/engine/reference/builder/#volume
 MOUNT_PATH
 
-# '--chain-id' evmos cli parameter, used to start nodes with specific chain-id and submit proposals
-# by default 'evmos_9000-1'
+# '--chain-id' evmos cli parameter, used to start nodes with a specific
+# chain-id and submit proposals
+# By default this is 'evmos_9000-1'
 CHAIN_ID
 ```
 
-To test an upgrade to explicit target version and continue to run the upgraded node, run:
+To test an upgrade to explicit target version
+and continue to run the upgraded node, use:
 
 ```shell
-make test-e2e E2E_SKIP_CLEANUP=true INITIAL_VERSION=<tag> TARGET_VERSION=<tag>
+make test-upgrade E2E_SKIP_CLEANUP=true INITIAL_VERSION=<tag> TARGET_VERSION=<tag>
 ```
 
 ### Upgrade Process
 
 Testing a chain upgrade is a multi-step process:
 
-1. Build a docker image for the evmos target version (local repo by default, if no explicit `TARGET_VERSION` provided as argument) (e.g. `v10.0.0`)
+1. Build a docker image for the evmos target version
+(local repo by default, if no explicit `TARGET_VERSION` provided as argument)
+(e.g. `v10.0.0`)
 2. Run tests
 3. The e2e test will first run an `INITIAL_VERSION` node container.
-4. The node will submit, deposit and vote for an upgrade proposal for upgrading to the `TARGET_VERSION`.
-5. After block `50` is reached, the test suite exports `/.evmosd` folder from docker container to local `build/` and than purge the container.
-6. Suite will mount `TARGET_VERSION` node to local `build/` dir and start the node. Node will get upgrade information from `upgrade-info.json` and will execute the upgrade.
-
-## Structure
+4. The node will submit, deposit and vote for an upgrade proposal
+for upgrading to the `TARGET_VERSION`.
+5. After block `50` is reached,
+the test suite exports `/.evmosd` folder from the docker container
+to the local `build/` folder and then purges the container.
+6. Suite will mount the node with `TARGET_VERSION`
+to the local `build/` dir and start the node.
+The node will get upgrade information from `upgrade-info.json`
+and will execute the upgrade.
+
+## Test Suite Structure
 
 ### `e2e` Package
 
-The `e2e` package defines an integration testing suite used for full end-to-end testing functionality. This package is decoupled from depending on the Evmos codebase. It initializes the chains for testing via Docker.
-As a result, the test suite may provide the desired Evmos version to Docker containers during the initialization. This design allows for the opportunity of testing chain upgrades by providing an older Evmos version to the container, performing the chain upgrade, and running the latest test suite. Here's an overview of the files:
-
-* `e2e_suite_test.go`: defines the testing suite and contains the core bootstrapping logic that creates a testing environment via Docker containers. A testing network is created dynamically with 2 test validators.
-
-* `e2e_test.go`: contains the actual end-to-end integration tests that utilize the testing suite.
+The `e2e` package defines an integration testing suite
+used for full end-to-end testing functionality.
+This package is decoupled from depending on the Evmos codebase.
+It initializes the chains for testing via Docker.  
+As a result, the test suite may provide the
+desired Evmos version to Docker containers during the initialization.
+This design allows for the opportunity of testing chain upgrades
+by providing an older Evmos version to the container,
+performing the chain upgrade,
+and running the latest test suite.  
+Here's an overview of the files:
+
+* `e2e_suite_test.go`: defines the testing suite
+and contains the core bootstrapping logic
+that creates a testing environment via Docker containers.
+A testing network is created dynamically with 2 test validators.
+
+* `e2e_test.go`: contains the actual end-to-end integration tests
+that utilize the testing suite.
 
 * `e2e_utils_test.go`: contains suite upgrade params loading logic.
 
 ### `upgrade` Package
 
-The `e2e` package defines an upgrade `Manager` abstraction. Suite will utilize `Manager`'s functions to run different versions of evmos containers, propose, vote, delegate and query nodes.
+The `e2e` package defines an upgrade `Manager` abstraction.
+Suite will utilize `Manager`'s functions
+to run different versions of evmos containers,
+propose, vote, delegate and query nodes.
 
-* `manager.go`: defines core manager logic for running containers, export state and create networks.
+* `manager.go`: defines core manager logic for running containers,
+export state and create networks.
 
-* `govexec.go`: defines `gov-specific` exec commands to submit/delegate/vote through nodes `gov` module.
+* `govexec.go`: defines `gov-specific` exec commands to submit/delegate/vote
+through nodes `gov` module.
 
-* `node.go`: defines `Node` strcuture responsible for setting node container parameters before run.
+* `node.go`: defines `Node` structure
+responsible for setting node container parameters before run.
 
 ### Version retrieve
 
-`TARGET_VERSION` by default retieved latest upgrade version from local codebase `evmos/app/upgrades` folder according to sevmver scheme.
-If explicit `TARGET_VERSION` provided as argument, corresponding node container will be pulled from [dockerhub](https://hub.docker.com/r/tharsishq/evmos/tags).
+If `INITIAL_VERSION` is provided as an argument,
+node container(s) with the corresponding version(s)
+will be pulled from [DockerHub](https://hub.docker.com/r/tharsishq/evmos/tags).
+If it is not specified,
+the test suite retrieves the second-to-last upgrade version
+from the local codebase (in the `evmos/app/upgrades` folder)
+according to [Semantic Versioning](https://semver.org/).
 
-`INITIAL_VERSION` retrieved as one version before the latest upgrade in `evmos/app/upgrades` correspondingly.
+If `TARGET_VERSION` is specified,
+the corresponding container will also be pulled from DockerHub.
+When not specified, the test suite will retrieve the latest upgrade version
+from `evmos/app/upgrades`.
 
 ### Testing Results
 
-The `make test-upgrade` script will output the test results for each testing file. In case of a successful upgrade, the script will print the following output (example):
+The `make test-upgrade` script will output the test results
+for each testing file.
+In case of a successful upgrade,
+the script will print the following output (example):
 
 ```log
 ok  	github.com/evmos/evmos/v9/tests/e2e	174.137s.
 ```
 
-if target node version failed to start, caintainers `[error stream]` and `[output stream]` will be printed from container to local terminal:
+If the target node version fails to start,
+the logs from the docker container will be printed:
 
 ```log
-            	Error:      	Received unexpected error:
-            	            	can't start evmos node, container exit code: 2
-
-            	            	[error stream]:
-
-            	            	7:03AM INF Unlocking keyring
-            	            	7:03AM INF starting ABCI with Tendermint
-            	            	panic: invalid minimum gas prices: invalid decimal coin expression: 0aevmos
-
-            	            	goroutine 1 [running]:
-            	            	github.com/cosmos/cosmos-sdk/baseapp.SetMinGasPrices({0xc0013563e7?, 0xc00163a3c0?})
-            	            		github.com/cosmos/cosmos-sdk@v0.46.5/baseapp/options.go:29 +0xd9
-            	            	main.appCreator.newApp({{{0x3399b40, 0xc000ec1db8}, {0x33ac0f8, 0xc0011314e0}, {0x33a2920, 0xc000ed2b80}, 0xc0000155f0}}, {0x3394520, 0xc001633bc0}, {0x33a5cc0, ...}, ...)
-            	            		github.com/evmos/evmos/v10/cmd/evmosd/root.go:243 +0x2ca
-            	            	github.com/evmos/ethermint/server.startInProcess(_, {{0x0, 0x0, 0x0}, {0x33b7490, 0xc001784c30}, 0x0, {0x7fff50b37f3d, 0xc}, {0x33ac0f8, ...}, ...}, ...)
-            	            		github.com/evmos/ethermint@v0.20.0-rc2/server/start.go:304 +0x9c5
-            	            	github.com/evmos/ethermint/server.StartCmd.func2(0xc001620600?, {0xc001745bd0?, 0x0?, 0x1?})
-            	            		github.com/evmos/ethermint@v0.20.0-rc2/server/start.go:123 +0x1ec
-            	            	github.com/spf13/cobra.(*Command).execute(0xc001620600, {0xc001745bb0, 0x1, 0x1})
-            	            		github.com/spf13/cobra@v1.6.1/command.go:916 +0x862
-            	            	github.com/spf13/cobra.(*Command).ExecuteC(0xc00160e000)
-            	            		github.com/spf13/cobra@v1.6.1/command.go:1044 +0x3bd
-            	            	github.com/spf13/cobra.(*Command).Execute(...)
-            	            		github.com/spf13/cobra@v1.6.1/command.go:968
-            	            	github.com/spf13/cobra.(*Command).ExecuteContext(...)
-            	            		github.com/spf13/cobra@v1.6.1/command.go:961
-            	            	github.com/cosmos/cosmos-sdk/server/cmd.Execute(0x2170d50?, {0x26d961f, 0x6}, {0xc00112c490, 0xd})
-            	            		github.com/cosmos/cosmos-sdk@v0.46.5/server/cmd/execute.go:36 +0x20f
-            	            	main.main()
-            	            		github.com/evmos/evmos/v10/cmd/evmosd/main.go:20 +0x45
-
-
-            	            	[output stream]:
-
-            	Test:       	TestIntegrationTestSuite/TestUpgrade
-            	Messages:   	can't mount and run upgraded node container
+Error:  Received unexpected error:
+        can't start evmos node, container exit code: 2
+
+        [error stream]:
+
+        7:03AM INF Unlocking keyring
+        7:03AM INF starting ABCI with Tendermint
+        panic: invalid minimum gas prices: invalid decimal coin expression: 0...
+
+        goroutine 1 [running]:
+        github.com/cosmos/cosmos-sdk/baseapp.SetMinGasPrices({0xc0013563e7?, ...
+            github.com/cosmos/cosmos-sdk@v0.46.5/baseapp/options.go:29 +0xd9
+        main.appCreator.newApp({{{0x3399b40, 0xc000ec1db8}, {0x33ac0f8, 0xc00...
+            github.com/evmos/evmos/v10/cmd/evmosd/root.go:243 +0x2ca
+        github.com/evmos/ethermint/server.startInProcess(_, {{0x0, 0x0, 0x0},...
+            github.com/evmos/ethermint@v0.20.0-rc2/server/start.go:304 +0x9c5
+        github.com/evmos/ethermint/server.StartCmd.func2(0xc001620600?, {0xc0...
+            github.com/evmos/ethermint@v0.20.0-rc2/server/start.go:123 +0x1ec
+        github.com/spf13/cobra.(*Command).execute(0xc001620600, {0xc001745bb0...
+            github.com/spf13/cobra@v1.6.1/command.go:916 +0x862
+        github.com/spf13/cobra.(*Command).ExecuteC(0xc00160e000)
+            github.com/spf13/cobra@v1.6.1/command.go:1044 +0x3bd
+        github.com/spf13/cobra.(*Command).Execute(...)
+            github.com/spf13/cobra@v1.6.1/command.go:968
+        github.com/spf13/cobra.(*Command).ExecuteContext(...)
+            github.com/spf13/cobra@v1.6.1/command.go:961
+        github.com/cosmos/cosmos-sdk/server/cmd.Execute(0x2170d50?, {0x26d961...
+            github.com/cosmos/cosmos-sdk@v0.46.5/server/cmd/execute.go:36 +0x...
+        main.main()
+            github.com/evmos/evmos/v10/cmd/evmosd/main.go:20 +0x45
+
+
+        [output stream]:
+
+Test:     TestIntegrationTestSuite/TestUpgrade
+Messages: can't mount and run upgraded node container
 ```
 
 To get all containers run:
 
 ```shell
-# check containters
+# list containers
 docker ps -a
 ```
 
@@ -150,15 +207,29 @@ To access containers logs directly, run:
 docker logs <container-id>
 ```
 
-For interaction with upgraded set `SKIP_CLEANUP=true` flag on make command agruments and enter the container after upgrade finished:
+To interact with the upgraded node
+pass `SKIP_CLEANUP=true` to the make command
+and enter the container after the upgrade has finished:
 
 ```shell
 docker exec -it <container-id> bash
 ```
 
-If cleanup was skipped upgraded node container should be removed manually:
+If the cleanup was skipped
+the upgraded node container should be removed manually:
 
 ```shell
 docker kill <container-id>
 docker rm <container-id>
 ```
+
+## Running multiple upgrades
+
+In order to run multiple upgrades,
+just combine the versions leading up to the last upgrade
+with a forward slash
+and pass them as the `INITIAL_VERSION`.
+
+```bash
+make test-upgrade INITIAL_VERSION=v10.0.1/v11.0.0-rc1 TARGET_VERSION=v11.0.0-rc3
+```