Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Implement nightly performance tests #6140

Merged
merged 43 commits into from Oct 10, 2019

Conversation

@danpaul000
Copy link
Contributor

danpaul000 commented Sep 27, 2019

This is tied to the nightly schedule in this pipeline: https://buildkite.com/solana-labs/testnet-nightly-performance

Fixes #6122

@danpaul000 danpaul000 added this to In progress in SLP1 QA via automation Sep 27, 2019
@danpaul000 danpaul000 requested review from mvines, pgarg66 and sakridge Sep 28, 2019
steps:
- command: "tests/testnet-performance/testnet-automation.sh"
label: "Run performance testnets on EDGE channel CPU ONLY"
# branches: 'master'

This comment has been minimized.

Copy link
@danpaul000

danpaul000 Sep 28, 2019

Author Contributor

branches are commented out while I test this in PR-land, otherwise the buildkite job would filter out everything here until this code actually lives on said branch.

@danpaul000

This comment has been minimized.

Copy link
Contributor Author

danpaul000 commented Sep 28, 2019

So this code takes the same structure from ci/testnet-automation.sh for the release TPS tests (actually replacing that file, as the TPS testing is not part of CI), and version controls the test parameters in the nightly.yml rather than in pipeline settings which are not version controlled. I pulled just about all the values out of the script so that an entire test case (or multiple cases) can be represented by a .yml file.

The nightly.yml here will be scheduled to be run by https://buildkite.com/solana-labs/testnet-nightly-performance . Pretty straight forward to add additional nightly test configurations or other pipelines to take advantage of the structure.

The 4 test cases in nightly.yml are not finalized, I'd like some feedback ( @pgarg66 ? @sakridge ?) on what settings you'd like to see run on the daily.

@@ -0,0 +1,77 @@
steps:

This comment has been minimized.

Copy link
@mvines

mvines Sep 30, 2019

Member

Can we run all the nightly testing on colo for now? I think we'll get a ton of benefit out of just using those machines, they're idling overnight and "free". Once we aren't finding any more issues with colo, we can consider adding a less-regular GCP expensive run or build colo2

@sakridge

This comment has been minimized.

Copy link
Member

sakridge commented Sep 30, 2019

The 4 test cases in nightly.yml are not finalized, I'd like some feedback ( @pgarg66 ? @sakridge ?) on what settings you'd like to see run on the daily.

I would say running a medium size network with a bench-tps client at the highest settings that are stable for at least 30 minutes, but longer would be nice. It's probably somewhere around 30-40k tps.

@danpaul000 danpaul000 force-pushed the danpaul000:nightly branch 3 times, most recently from 70f610b to 8d6acf7 Oct 1, 2019
@danpaul000 danpaul000 removed request for sakridge and pgarg66 Oct 2, 2019
@danpaul000 danpaul000 force-pushed the danpaul000:nightly branch from d9f813f to dcec01a Oct 3, 2019
@mvines

This comment has been minimized.

Copy link
Member

mvines commented Oct 3, 2019

I'm not a super fan of a top-level directory called tests/, too generic. Some alternative ideas:

  • product-test/
  • nightly-test/
  • automation-test/
@danpaul000

This comment has been minimized.

Copy link
Contributor Author

danpaul000 commented Oct 3, 2019

I'm not a super fan of a top-level directory called tests/, too generic. Some alternative ideas:

* `product-test/`

* `nightly-test/`

* `automation-test/`

Sure, no problem. Without going too far into the weeds, I've been trying to keep the automation script and structure generic enough to handle more than just nightly, hence the generic top-level dir (that admittedly doesn't have much in the way of generic tooling yet)

@mvines

This comment has been minimized.

Copy link
Member

mvines commented Oct 3, 2019

system-test/ or product-test/ then? qa/ maybe too generic?

@danpaul000 danpaul000 requested a review from t-nelson Oct 7, 2019
@danpaul000 danpaul000 force-pushed the danpaul000:nightly branch from 27d2a5d to 7fdf6b9 Oct 9, 2019
@danpaul000

This comment has been minimized.

Copy link
Contributor Author

danpaul000 commented Oct 10, 2019

Been soaking the test configuration and job runner every hour on colo, seems stable and passing: https://buildkite.com/solana-labs/testnet-nightly-performance/builds?branch=pull%2F6140%2Fhead

Copy link
Contributor

t-nelson left a comment

LGTM! Just a few minors

danpaul000 added 2 commits Oct 10, 2019
@mergify mergify bot dismissed t-nelson’s stale review Oct 10, 2019

Pull request has been modified.

@danpaul000 danpaul000 merged commit 60e8cf5 into solana-labs:master Oct 10, 2019
12 checks passed
12 checks passed
Rule: v0.19 backport (backport) Backports have been created
Details
Summary 2 rules match and 6 potential rules
Details
buildkite/solana Build #12918 passed (26 minutes, 26 seconds)
Details
buildkite/solana/bench Passed (2 seconds)
Details
buildkite/solana/checks Passed (1 minute, 54 seconds)
Details
buildkite/solana/coverage Passed (36 seconds)
Details
buildkite/solana/local-cluster Passed (13 minutes, 23 seconds)
Details
buildkite/solana/pipeline-upload Passed (8 seconds)
Details
buildkite/solana/shellcheck Passed (30 seconds)
Details
buildkite/solana/stable Passed (24 minutes, 14 seconds)
Details
buildkite/solana/stable-perf Passed (9 seconds)
Details
ci-gate Pull Request accepted for CI pipeline
SLP1 QA automation moved this from In progress to Done Oct 10, 2019
@danpaul000 danpaul000 deleted the danpaul000:nightly branch Oct 10, 2019
mergify bot pushed a commit that referenced this pull request Oct 10, 2019
* Implement nightly performance tests on colo

(cherry picked from commit 60e8cf5)
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
SLP1 QA
  
Done
4 participants
You can’t perform that action at this time.