[forge] Add a test for validation set changes #4279

zcchahaha · 2022-09-16T20:20:41Z

Description

Add a test where 1/3 validators leave the validator set and then rejoin. More complex tests can be added in separate PRs.
Add a feature to make test framework add_move_file function compilation conditional due to some move dependencies issues when introducing CliTestFramework to forge. https://aptos-org.slack.com/archives/C036SHTFHGR/p1663364319592519

This change is

sitalkedia · 2022-09-19T18:51:51Z

@zcchahaha - Is this ready for review?

zcchahaha · 2022-09-19T19:13:14Z

@sitalkedia Not yet. Still try to figure out some dependencies issues. https://aptos-org.slack.com/archives/C036SHTFHGR/p1663364319592519

igor-aptos

Some comments inline, but overall looks good

igor-aptos · 2022-09-29T22:11:44Z

testsuite/testcases/src/validator_join_leave_test.rs

+
+        let mut cli = runtime.block_on(async {
+            CliTestFramework::new(
+                swarm.validators().next().unwrap().rest_api_endpoint(),


use rest_client.clone() here?

The parameter is a URL, not a rest client.

igor-aptos · 2022-09-29T22:13:12Z

testsuite/testcases/src/validator_join_leave_test.rs

+                    .await
+                    .unwrap();
+
+                reconfig(


oh, you should probably do reconfig once at the end, this will create bunch of epochs.

we also want to test that they can all ask to leave simultaneously

in the follow-up randomized test you are mentioning below, you can randomly select how many validators to leave simulatenously, before reconfig.

Added a reconfig after the loop.

igor-aptos · 2022-09-29T22:13:37Z

testsuite/testcases/src/validator_join_leave_test.rs

+            for operator_index in validator_cli_indices.iter().rev().take(num_validators / 3) {
+                cli.join_validator_set(*operator_index, None).await.unwrap();
+
+                reconfig(


same, to reconfig once , after the loop

github-actions · 2022-09-30T00:00:29Z

✅ Forge suite `compat` success on `843b204dce971d98449b82624f4f684c7a18b991` ==> `dbf36e3875ea6240d8f3722800242823eb022b40`

Compatibility test results for 843b204dce971d98449b82624f4f684c7a18b991 ==> dbf36e3875ea6240d8f3722800242823eb022b40 (PR)
1. Check liveness of validators at old version: 843b204dce971d98449b82624f4f684c7a18b991
compatibility::simple-validator-upgrade::liveness-check : 7700 TPS, 4911 ms latency, 11100 ms p99 latency,no expired txns
2. Upgrading first Validator to new version: dbf36e3875ea6240d8f3722800242823eb022b40
compatibility::simple-validator-upgrade::single-validator-upgrade : 4967 TPS, 7633 ms latency, 10100 ms p99 latency,no expired txns
3. Upgrading rest of first batch to new version: dbf36e3875ea6240d8f3722800242823eb022b40
compatibility::simple-validator-upgrade::half-validator-upgrade : 5448 TPS, 6950 ms latency, 9000 ms p99 latency,no expired txns
4. upgrading second batch to new version: dbf36e3875ea6240d8f3722800242823eb022b40
compatibility::simple-validator-upgrade::rest-validator-upgrade : 7368 TPS, 4930 ms latency, 9300 ms p99 latency,no expired txns
5. check swarm health
Compatibility test for 843b204dce971d98449b82624f4f684c7a18b991 ==> dbf36e3875ea6240d8f3722800242823eb022b40 passed
Test Ok

Grafana dashboard
Humio Logs
Test runner output
Test run is land-blocking

github-actions · 2022-09-30T00:00:54Z

✅ Forge suite `land_blocking` success on `dbf36e3875ea6240d8f3722800242823eb022b40`

performance benchmark with full nodes : 7188 TPS, 5532 ms latency, 7900 ms p99 latency,no expired txns
Test Ok

Grafana dashboard
Humio Logs
Test runner output
Test run is land-blocking

zcchahaha added the CICD:run-e2e-tests when this label is present github actions will run all land-blocking e2e tests from the PR label Sep 16, 2022

zcchahaha force-pushed the validator_set_change branch 2 times, most recently from b1a072b to c6cd22f Compare September 16, 2022 21:57

rustielin force-pushed the validator_set_change branch from c6cd22f to 80ab611 Compare September 16, 2022 22:54

zcchahaha force-pushed the validator_set_change branch 3 times, most recently from a3b2932 to 92a97c5 Compare September 19, 2022 00:23

zcchahaha force-pushed the validator_set_change branch 3 times, most recently from 226409d to 7a4cb71 Compare September 19, 2022 21:18

zcchahaha requested a review from gregnazario as a code owner September 19, 2022 21:18

zcchahaha marked this pull request as draft September 19, 2022 21:21