[forge] refactory Forge CLI interface

[zihaoccc]

Forge Cli is increasingly complicate since it supports more and more operations. Refactory CLI to different operation categories for better user experience.

For cluster Opertions:

• cargo run -p forge-cli -- operator clean-up
• cargo run -p forge-cli -- operator resize
• cargo run -p forge-cli -- operator set-validator

for Testing:

• cargo run -p forge-cli -- test local-swarm
• ./scripts/fgi/run --local-swarm
• ./script/fgi/run --pr xxx
• ./scripts/fgi/run -T dev_czh_pull_9061 --suite land_blocking_compat

Motivation

(Write your motivation for proposed changes here.)

Have you read the Contributing Guidelines on pull requests?

(Write your answer here.)

Test Plan

(Share your test plan here. If you changed code, please provide us with clear instructions for verifying that your changes work.)

Related PRs

(If this PR adds or changes functionality, please take some time to update or suggest changes to the docs at https://developers.diem.com, and link to your PR here.)

If targeting a release branch, please fill the below out as well

• Justification and breaking nature (who does it affect? validators, full nodes, tooling, operators, AOS, etc.)
• Comprehensive test results that demonstrate the fix working and not breaking existing workflows.
• Why we must have it for V1 launch.
• What workarounds and alternative we have if we do not push the PR.

[rustielin]

much nicer! LGTM

Have you tried passing in these new args in fgi? One side effect of having this many subcommands is that it becomes a bit more complicated how we pass args from fgi to forge. I don't anticipate this causing too much issue though, since we'll likely add more functionality to the forge command itself, and keep the k8s-swarm subcommand simple (not needing any further args).

[zihaoccc]

much nicer! LGTM

Have you tried passing in these new args in fgi? One side effect of having this many subcommands is that it becomes a bit more complicated how we pass args from fgi to forge. I don't anticipate this causing too much issue though, since we'll likely add more functionality to the forge command itself, and keep the k8s-swarm subcommand simple (not needing any further args).

yea, I did mention above in test plan for those routines thourgh fgi instead of cargo, such as:
./scripts/fgi/run --local-swarm
./script/fgi/run --pr xxx
./scripts/fgi/run -T dev_czh_pull_9061 --suite land_blocking_compat

I did make corresponding change in fgi:
ret = subprocess.call(["cargo", "run", "-p", "forge-cli", "--", "test", "local-swarm"]) and
f"timeout {timeout_secs} forge {' '.join(forge_args)} test k8s-swarm --cluster-name {cluster_name} --image-tag {tag}".strip()

[zihaoccc]

/land

[rustielin]

Do you anticipate it needing to have something like this though? I'm not sure how much more we're intending to add to the swarm functionality

f"timeout {timeout_secs} forge {' '.join(forge_args)} test k8s-swarm  {' '.join(k8s_swarm_args)} --cluster-name {cluster_name} --image-tag {tag}".strip()

[github-actions]

Cluster Test Result

Test runner setup time spent 261 secs
Compatibility test results for land_c90be61f ==> land_3e984edd (PR)
1. All instances running land_c90be61f, generating some traffic on network
2. First full node land_c90be61f ==> land_3e984edd, to validate new full node to old validator node traffic
3. First Validator node land_c90be61f ==> land_3e984edd, to validate storage compatibility
4. First batch validators (14) land_c90be61f ==> land_3e984edd, to test consensus and traffic between old full nodes and new validator node
5. First batch full nodes (14) land_c90be61f ==> land_3e984edd
6. Second batch validators (15) land_c90be61f ==> land_3e984edd, to upgrade rest of the validators
7. Second batch of full nodes (15) land_c90be61f ==> land_3e984edd, to finish the network upgrade, time spent 713 secs
all up : 1245 TPS, 3637 ms latency, 4100 ms p99 latency, no expired txns, time spent 249 secs
Logs: http://kibana.ct-0-k8s-testnet.aws.hlw3truzy4ls.com/app/kibana#/discover?_g=(time:(from:'2021-09-01T19:23:32Z',to:'2021-09-01T19:46:40Z'))
Dashboard: http://grafana.ct-0-k8s-testnet.aws.hlw3truzy4ls.com/d/performance/performance?from=1630524212000&to=1630525600000
Validator 1 logs: http://kibana.ct-0-k8s-testnet.aws.hlw3truzy4ls.com/app/kibana#/discover?_g=(time:(from:'2021-09-01T19:23:32Z',to:'2021-09-01T19:46:40Z'))&_a=(columns:!(log),query:(language:kuery,query:'kubernetes.pod_name:"val-1"'),sort:!(!('@timestamp',desc)))

Repro cmd:

./scripts/cti --tag land_c90be61f --cluster-test-tag land_3e984edd -E BATCH_SIZE=15 -E UPDATE_TO_TAG=land_3e984edd --report report.json --suite land_blocking_compat

🎉 Land-blocking cluster test passed! 👌