Run faucet integration tests in CI

[banool]

Description

This PR adds CI for running the faucet integration tests. Because we want to test the faucet against both devnet and testnet I use an action. Currently I use the action once in two separate jobs to make it faster, but this is also wasteful because I build the faucet for tests twice with no shared cache (I think). I'll ask the CI experts the best option.

Given the faucet changes rarely these tests might be sort of wasteful, but I figure that's already the case for many tests and the real fix is a target determinator. I'm using ubuntu-latest so it's not our runners. Of course, this means I have to build everything from scratch so this takes like 45 minutes total. Though later runs should be faster because we use rust-cache in our rust-setup action. The alternative would be to either use high-perf-docker, or even more invasively, add these tests to rust-smoke-test, but then I'd have to spin up a local testnet and Redis in that CI, which is not ideal. Hence why I've chosen this route.

Note: I need #7126 to land into devnet / testnet first. Update: Done.

Test Plan

CI.

[perryjrandall]

Can this all be run locally?

Or to the point, how much of this can we extract from the workflow and into a useful local tool?

[banool]

I addressed Perry's feedback, as much as possible into a script. As for the performance, when the Rust cache is hot the whole workflow takes about 9 minutes, so I think having it on the GH runners is fine.

[github-actions]

✅ Forge suite compat success on testnet_2d8b1b57553d869190f61df1aaf7f31a8fc19a7b ==> 6c36d861c00c22c1dd8bc779f74c2fcac2885499

Compatibility test results for testnet_2d8b1b57553d869190f61df1aaf7f31a8fc19a7b ==> 6c36d861c00c22c1dd8bc779f74c2fcac2885499 (PR)
1. Check liveness of validators at old version: testnet_2d8b1b57553d869190f61df1aaf7f31a8fc19a7b
compatibility::simple-validator-upgrade::liveness-check : 7969 TPS, 4762 ms latency, 7500 ms p99 latency,no expired txns
2. Upgrading first Validator to new version: 6c36d861c00c22c1dd8bc779f74c2fcac2885499
compatibility::simple-validator-upgrade::single-validator-upgrade : 4894 TPS, 7938 ms latency, 11200 ms p99 latency,no expired txns
3. Upgrading rest of first batch to new version: 6c36d861c00c22c1dd8bc779f74c2fcac2885499
compatibility::simple-validator-upgrade::half-validator-upgrade : 4638 TPS, 8348 ms latency, 11200 ms p99 latency,no expired txns
4. upgrading second batch to new version: 6c36d861c00c22c1dd8bc779f74c2fcac2885499
compatibility::simple-validator-upgrade::rest-validator-upgrade : 6968 TPS, 5501 ms latency, 9400 ms p99 latency,no expired txns
5. check swarm health
Compatibility test for testnet_2d8b1b57553d869190f61df1aaf7f31a8fc19a7b ==> 6c36d861c00c22c1dd8bc779f74c2fcac2885499 passed
Test Ok

• Grafana dashboard
• Humio Logs
• Pyroscope Profiling
• Test runner output
• Test run is land-blocking

[github-actions]

✅ Forge suite land_blocking success on 6c36d861c00c22c1dd8bc779f74c2fcac2885499

performance benchmark with full nodes : 6026 TPS, 6594 ms latency, 8700 ms p99 latency,no expired txns
Test Ok

• Grafana dashboard
• Humio Logs
• Pyroscope Profiling
• Test runner output
• Test run is land-blocking

[github-actions]

✅ Forge suite framework_upgrade success on cb4ba0a57c998c60cbab65af31a64875d2588ca5 ==> 6c36d861c00c22c1dd8bc779f74c2fcac2885499

Compatibility test results for cb4ba0a57c998c60cbab65af31a64875d2588ca5 ==> 6c36d861c00c22c1dd8bc779f74c2fcac2885499 (PR)
Upgrade the nodes to version: 6c36d861c00c22c1dd8bc779f74c2fcac2885499
framework_upgrade::framework-upgrade::full-framework-upgrade : 6789 TPS, 5659 ms latency, 8200 ms p99 latency,no expired txns
5. check swarm health
Compatibility test for cb4ba0a57c998c60cbab65af31a64875d2588ca5 ==> 6c36d861c00c22c1dd8bc779f74c2fcac2885499 passed
Test Ok

• Grafana dashboard
• Humio Logs
• Pyroscope Profiling
• Test runner output
• Test run is land-blocking

[Cho202]

Thanks you.