Wrk stats script ex

[OvermindDL1]

The simple elixir-script I'd been using to automate wrk benchmark testing. It is not at all prettied up and the code is pretty organic as it 'grew' over time rather than being designed for this purpose.

Basic usage:

╰─➤  tools/stats.exs

Usage:  tools/stats.exs <options> <matching-servers>

Options are as listed below, matching-servers will run any server that even partially matches, thus `fastht` will match `go_fasthttprouter`.  For all built servers you can pass in just `_` as all servers contain that character.

Examples:

  * tools/stats.exs cpp router_cr nim rust fasthttp
  * tools/stats.exs -w 1 -d 3 cpp crystal nim rust go

Standard Options:

  --help, -h, -?                        : Show this help menu
  --threads <pos_int>, -t <pos_int>     : Number of threads to use, defaults: CPU Core count * 2/3 truncated
  --connections <pos_int>, -c <pos_int> : Max number of simultaneous streams to use, default: 1000
  --duration <pos_int>, -d <pos_int>    : Number of seconds per test, default: 5
  --warmup <pos_int>, -w <pos_int>      : Number of seconds to perform warmup, matters on GC languages, default: 5
  --verbose, -v                         : Turn on verbose logging


SSH Options:

Do note that currently the system should be built remotely and same-named (though empty is fine) server files should be on the client side as well.  Generally if the remote system type is the same as the local system then just build locally and upload the entire directory over.
(TODO: Get server list from remote connection instead of getting the list locally)

  --server <string>, -s <string>        : The IP or DNS name of the remote server to test against and SSH to.
  --sshpath <string>, -p <string>       : The full path to the root location of the repo directory

An example session:

╰─➤ tools/stats.exs -w 1 -d 3 evhtp mofuw router_cr
Total Cores: 16
Concurrent Connections: 1000
Threads: 10
Warmup: 1 seconds
Duration: 3 seconds

Processing servers:

• bin/server_cpp_evhtp
• bin/server_nim_mofuw
• bin/server_crystal_router_cr

Processing: bin/server_cpp_evhtp
Processing: bin/server_nim_mofuw
Processing: bin/server_crystal_router_cr

Path	URL	Errors	Total Requests Count	Total Requests/s	Total Requests Throughput	Total Throughput/s	Req/s Avg	Req/s Stdev	Req/s Max	Req/s +/-	Latency Avg	Latency Stdev	Latency Max	Latency +/-	50%	75%	90%	99%
bin/server_cpp_evhtp	http://127.0.0.1:3000/	0	1970134	637262.09	120.25MB	38.90MB	65.37k	18.21k	120.85k	75.67%	2.35ms	5.06ms	225.44ms	91.11%	655.00us	2.64ms	6.92ms	15.37ms
bin/server_cpp_evhtp	http://127.0.0.1:3000/user	0	1856089	603652.51	113.29MB	36.84MB	61.69k	14.84k	105.39k	80.00%	2.40ms	3.29ms	40.00ms	86.65%	839.00us	3.02ms	6.97ms	14.84ms
bin/server_cpp_evhtp	http://127.0.0.1:3000/user/0	0	1730182	567894.68	155.10MB	50.91MB	57.66k	12.76k	107.25k	78.00%	2.50ms	6.74ms	219.62ms	95.53%	817.00us	2.87ms	6.28ms	14.70ms
bin/server_crystal_router_cr	http://127.0.0.1:3000/	0	218060	71996.15	12.89MB	4.26MB	7.30k	650.07	8.47k	82.00%	13.43ms	2.47ms	24.10ms	81.12%	12.40ms	12.92ms	18.25ms	19.34ms
bin/server_crystal_router_cr	http://127.0.0.1:3000/user	0	184744	60990.38	10.92MB	3.61MB	6.19k	694.09	7.86k	77.33%	15.52ms	3.56ms	27.98ms	53.47%	13.50ms	19.54ms	19.90ms	25.82ms
bin/server_crystal_router_cr	http://127.0.0.1:3000/user/0	0	190301	61911.69	16.70MB	5.43MB	6.37k	374.15	7.04k	83.67%	15.73ms	3.36ms	36.94ms	69.35%	13.77ms	19.74ms	20.23ms	23.38ms
bin/server_nim_mofuw	http://127.0.0.1:3000/	0	1565248	505667.1	204.50MB	66.07MB	51.99k	16.32k	92.63k	71.33%	2.65ms	3.37ms	36.18ms	86.15%	0.87ms	3.76ms	7.23ms	15.07ms
bin/server_nim_mofuw	http://127.0.0.1:3000/user	0	1530053	497595.53	199.91MB	65.01MB	51.01k	15.90k	94.17k	73.58%	2.74ms	3.47ms	35.28ms	85.75%	0.90ms	3.93ms	7.57ms	15.44ms
bin/server_nim_mofuw	http://127.0.0.1:3000/user/0	0	1386912	447526.81	220.88MB	71.27MB	46.89k	14.51k	88.33k	77.29%	2.70ms	3.26ms	36.73ms	86.08%	1.02ms	3.91ms	7.12ms	14.56ms

Rankings

Ranking by Average Requests per second:

1. 602936 req/sec : bin/server_cpp_evhtp
2. 483596 req/sec : bin/server_nim_mofuw
3. 64966 req/sec : bin/server_crystal_router_cr

 

To replicate the current single-core benchmarker, you can just set threads to 1, but why you would do that when every server everywhere has many cores makes no sense and will not saturate really any of the servers and will not test their locking and communication methodologies or anything of the sort, but if you really really wanted to for whatever reason:

╰─➤ tools/stats.exs -w 1 -d 3 -t 1 evhtp mofuw router_cr fasthttp
Total Cores: 16
Concurrent Connections: 1000
Threads: 1
Warmup: 1 seconds
Duration: 3 seconds

Processing servers:

• bin/server_cpp_evhtp
• bin/server_nim_mofuw
• bin/server_crystal_router_cr
• bin/server_go_fasthttprouter

Processing: bin/server_cpp_evhtp
Processing: bin/server_nim_mofuw
Processing: bin/server_crystal_router_cr
Processing: bin/server_go_fasthttprouter

Path	URL	Errors	Total Requests Count	Total Requests/s	Total Requests Throughput	Total Throughput/s	Req/s Avg	Req/s Stdev	Req/s Max	Req/s +/-	Latency Avg	Latency Stdev	Latency Max	Latency +/-	50%	75%	90%	99%
bin/server_cpp_evhtp	http://127.0.0.1:3000/	0	322000	105832.7	19.65MB	6.46MB	110.97k	5.59k	120.00k	64.29%	4.66ms	1.60ms	9.03ms	59.01%	4.66ms	6.00ms	6.82ms	7.71ms
bin/server_cpp_evhtp	http://127.0.0.1:3000/user	0	316000	105164.12	19.29MB	6.42MB	110.04k	7.28k	130.00k	71.43%	4.69ms	1.63ms	11.27ms	59.32%	4.68ms	6.05ms	6.89ms	7.83ms
bin/server_cpp_evhtp	http://127.0.0.1:3000/user/0	0	300000	99536.0	26.89MB	8.92MB	105.19k	6.68k	111.11k	75.00%	4.91ms	1.68ms	8.70ms	58.37%	4.92ms	6.35ms	7.20ms	7.94ms
bin/server_crystal_router_cr	http://127.0.0.1:3000/	0	225491	74850.27	13.33MB	4.43MB	78.25k	3.99k	84.68k	65.52%	12.73ms	3.19ms	24.30ms	65.26%	11.13ms	15.87ms	16.58ms	21.86ms
bin/server_crystal_router_cr	http://127.0.0.1:3000/user	0	200284	66455.73	11.84MB	3.93MB	69.52k	2.58k	72.54k	62.07%	13.91ms	3.64ms	26.74ms	65.17%	12.01ms	17.49ms	18.31ms	23.77ms
bin/server_crystal_router_cr	http://127.0.0.1:3000/user/0	0	210971	70301.24	18.31MB	6.10MB	73.11k	3.18k	78.15k	62.07%	13.64ms	3.30ms	25.98ms	69.21%	12.07ms	17.24ms	18.25ms	24.15ms
bin/server_go_fasthttprouter	http://127.0.0.1:3000/	0	350000	114412.62	31.04MB	10.15MB	118.71k	5.37k	130.00k	75.86%	4.38ms	1.40ms	8.73ms	60.22%	4.35ms	5.51ms	6.26ms	7.21ms
bin/server_go_fasthttprouter	http://127.0.0.1:3000/user	0	346000	111502.7	30.69MB	9.89MB	117.06k	5.25k	131.31k	79.31%	4.44ms	1.46ms	10.34ms	61.55%	4.40ms	5.58ms	6.44ms	7.48ms
bin/server_go_fasthttprouter	http://127.0.0.1:3000/user/0	0	322000	107213.9	50.36MB	16.77MB	112.64k	4.74k	120.00k	57.14%	4.63ms	1.47ms	11.26ms	61.74%	4.60ms	5.77ms	6.60ms	7.64ms
bin/server_nim_mofuw	http://127.0.0.1:3000/	0	306873	100360.3	40.09MB	13.11MB	106.83k	2.47k	108.91k	96.43%	4.88ms	1.52ms	12.66ms	58.30%	4.91ms	6.20ms	6.93ms	7.47ms
bin/server_nim_mofuw	http://127.0.0.1:3000/user	0	306831	101265.99	40.09MB	13.23MB	107.79k	2.47k	111.00k	89.29%	4.83ms	1.51ms	9.98ms	58.39%	4.85ms	6.12ms	6.86ms	7.48ms
bin/server_nim_mofuw	http://127.0.0.1:3000/user/0	0	305910	99513.51	48.72MB	15.85MB	105.07k	2.93k	111.05k	89.29%	4.96ms	1.53ms	10.45ms	57.84%	4.99ms	6.28ms	7.03ms	7.55ms

Rankings

Ranking by Average Requests per second:

1. 111043 req/sec : bin/server_go_fasthttprouter
2. 103510 req/sec : bin/server_cpp_evhtp
3. 100379 req/sec : bin/server_nim_mofuw
4. 70535 req/sec : bin/server_crystal_router_cr

(Just as a cool note, go is using libevent just like evhtp is, except Go's is not performing a full regex path test where evhtp is now doing as implied by tbrand that it should to slow it down for whatever reasons, even though no one would realistically use a regex path splitter when not necessary, as it is not in this case. I could easily make a pure libevent-based HTTP server that has no framework or hooks or anything of the sort that would blow everything else away as tbrand implied that evhtp was already doing without actually checking the library code to see that it was not...)

@waghanza

[OvermindDL1]

And for note, this is not necessarily for inclusion, just an example that people can play with if a proper wrk script wrapper is to be made (otherwise sure, feel free to include it, but it was very much a play script for me, I'd never normally write something like this in Elixir, rather just a normal shell script).

[waghanza]

@OvermindDL1 what do you think of https://github.com/giltene/wrk2 ?

[OvermindDL1]

Hmm, never heard of wrk2, but based on wrk2 is currently in experimental/development mode, and may well be merged into wrk in the future if others see fit to adopt it's changes. I'm unsure if it is safe depending on it (or has it already been merged in to wrk?).

[waghanza]

@OvermindDL1 the fact is that there is a lot of different functionnalities on the two projects, I'll ask on wrk2 gitter chanel.

Also, do you heard of vegeta (go tool) or bombardier ?

[OvermindDL1]

Also, do you heard of vegeta (go tool) or bombardier ?

Not heard of vegeta, and I tried bombardier once as I recall but I remembered it not being able to send enough throughput out and wasn't able to saturate the server at the time compared to wrk.

[waghanza]

@OvermindDL1 perhaps, a little comparison of these tools should be decided 😎

@tbrand What do you think ?

[OvermindDL1]

A Benchmark Benchmark! ^.^

[waghanza]

@OvermindDL1 seems that results are varying depending on wrk options (benchmark duration, threads ....)
How have you choose yours ?

[OvermindDL1]

How have you choose yours ?

Mostly to minimize the time that I spent on it. I found that a short duration had no difference to a long duration in non-GC languages (C++/Rust/Etc..) so I kept it to 3 seconds. GC languages and JIT languages needed more but even then I saw almost no variance regardless. The thread count was just based on experimentation of me testing from 1 to core-count threads and picking the highest overall performing value, which was the default calculation I put in my script.

I.E. just testing, not special in any way.

[waghanza]

@OvermindDL1 @tbrand I had open a specific issue #238 to debate about what to display