feat: Add lotus-bench cli option to stress test any binary #11270
Conversation
There was a problem hiding this comment.
This seems generally useful but an odd addition to lotus-bench given that it's general-purpose.
It looks like you'd be better off with something like https://github.com/sharkdp/hyperfine (available in pretty much every package manager)
|
|
||
| Here are some real examples: | ||
| lotus-bench cli --cmd='lotus-shed mpool miner-select-messages' // runs the command with default concurrency and qps | ||
| lotus-bench cli --cmd='lotus-shed mpool miner-select-messages:3' // override concurrency to 3 |
There was a problem hiding this comment.
Using : could cause issues down the road (for any command including a literal :. Why not just rely on the flags?
Also, for the command, I'd suggest using positional parameters. I.e., "the rest of the args".
Yes, this could be an independent general purpose tool but I put it here for a lack of a better place to put it as it already shares many of the same code when stress testing our RPC endpoints, and this would enable us to do the same by stress testing lotus backend through our different cli commands. I am not familiar with hyperfine but this does seem different enough as its based on generating specific workloads for a certain duration where you can tune the concurrency/qps for each independently and then get simple histogram/latency distribution. |
|
|
||
| Here are some real examples: | ||
| lotus-bench cli --cmd='lotus-shed mpool miner-select-messages' // runs the command with default concurrency and qps | ||
| lotus-bench cli --cmd='lotus-shed mpool miner-select-messages:3' // override concurrency to 3 |
Didn't review, but I'm fine with this landing |
Related Issues
Written to help with solving #11251
Extension to the RPC benchmark introduced in #10761
Proposed Changes
This PR extends the RPC benchmark to support running any binary and analyze their latency and histogram distribution, track most common errors, perform stress testing under different concurrency levels and see how it works under different QPS.
Test plan
Lets say you want to understand how
lotus-shed mpool miner-select-messagesbehaves over time.You can use this new
lotus-bench clitool to achieve this, for example the following command runs the command using a single thread (concurrency=1) at 1 qps and provides a realtime updated latency/histogram UI and it runs for 24 hours.lotus-bench cli --cmd='lotus-shed mpool miner-select-messages' --concurrency=1 --qps=1 --watch=5s --duration=24hIt will produce something like:
Additional Info
This works exactly the same as the rpc benchmark, which has a detailed summary (see PR)
Checklist
Before you mark the PR ready for review, please make sure that:
<PR type>: <area>: <change being made>fix: mempool: Introduce a cache for valid signaturesPR type: fix, feat, build, chore, ci, docs, perf, refactor, revert, style, testarea, e.g. api, chain, state, market, mempool, multisig, networking, paych, proving, sealing, wallet, deps