Skip to content

userver: HttpArena benchmark implementation (userver-create-service)#811

Merged
MDA2AV merged 1 commit into
MDA2AV:mainfrom
botanegg:userver-benchmarks
Jun 6, 2026
Merged

userver: HttpArena benchmark implementation (userver-create-service)#811
MDA2AV merged 1 commit into
MDA2AV:mainfrom
botanegg:userver-benchmarks

Conversation

@botanegg
Copy link
Copy Markdown
Contributor

@botanegg botanegg commented Jun 5, 2026

10 benchmark profiles (baseline, pipelined, limited-conn, json, json-comp, upload, api-4, api-16, static, async-db)

  • 6 HTTP handlers (baseline11, baseline2, json, plaintext, upload, async_db)
  • gzip compression middleware
  • Dockerfile based on ubuntu-24.04-userver (prebuilt framework version)

Description

^^^
HTTP 2 benchmarks and TLS are preconfigured, but due to Userver issues with ALPN, they are not included in meta.json.


PR Commands — comment on this PR to trigger (requires collaborator approval):

Command Description
/benchmark -f <framework> Run all benchmark tests
/benchmark -f <framework> -t <test> Run a specific test
/benchmark -f <framework> --save Run and save results (updates leaderboard on merge)

Always specify -f <framework>. Results are automatically compared against the current leaderboard.


Run benchmarks locally

You can validate and benchmark your framework locally with the lite script — no CPU pinning, fixed connection counts, all load generators run in Docker.

./scripts/validate.sh <framework>
./scripts/benchmark-lite.sh <framework> baseline
./scripts/benchmark-lite.sh --load-threads 4 <framework>

Requirements: Docker Engine on Linux. Load generators (gcannon, h2load, h2load-h3, wrk, ghz) are built as self-contained Docker images on first run.

@botanegg botanegg force-pushed the userver-benchmarks branch 2 times, most recently from c5aea18 to 39669be Compare June 5, 2026 21:54
@botanegg
Copy link
Copy Markdown
Contributor Author

botanegg commented Jun 5, 2026

/benchmark -f userver

@github-actions
Copy link
Copy Markdown
Contributor

github-actions Bot commented Jun 5, 2026

👋 /benchmark request received. A collaborator will review and approve the run.

@github-actions
Copy link
Copy Markdown
Contributor

github-actions Bot commented Jun 6, 2026

Benchmark Results

Framework: userver | Test: all tests

Test Conn RPS CPU Mem Δ RPS Δ Mem
baseline 512 573,833 5842.0% 212MiB -4.8% +6.5%
baseline 4096 1,053,509 6508.7% 432MiB +7.7% -2.9%
pipelined 512 1,448,187 6691.4% 256MiB -0.1% -1.9%
pipelined 4096 1,328,604 6689.3% 896MiB -0.9% -0.2%
limited-conn 512 807,499 6119.3% 243MiB +15.1% +1.2%
limited-conn 4096 861,718 6302.8% 572MiB +1.4% -1.0%
json 4096 823,463 6492.9% 507MiB NEW NEW
json-comp 512 510,494 6534.9% 246MiB NEW NEW
json-comp 4096 493,346 6309.3% 556MiB NEW NEW
json-comp 16384 468,665 6466.6% 1.5GiB NEW NEW
upload 32 699 3216.7% 956MiB NEW NEW
upload 256 613 6670.8% 3.5GiB NEW NEW
api-4 256 58,442 403.5% 194MiB NEW NEW
api-16 1024 135,085 1792.8% 382MiB NEW NEW
static 1024 32,623 6488.9% 349MiB -80.3% -20.3%
static 4096 32,315 6501.1% 607MiB -82.9% -34.8%
static 6800 32,165 6486.4% 848MiB -83.4% -36.3%
async-db 1024 242,827 5823.6% 316MiB NEW NEW
Full log
    Latency   125.33ms   60.14ms 414.61ms   69.45%
    Req/Sec   509.45    184.02     7.64k    92.09%
  164814 requests in 5.10s, 2.84GB read
Requests/sec:  32315.88
Transfer/sec:    569.62MB
[info] CPU 6501.1% | Mem 607MiB

[run 3/3]
Running 5s test @ http://localhost:8080
  64 threads and 4096 connections
  Thread Stats   Avg      Stdev     Max   +/- Stdev
    Latency   126.25ms   61.99ms 430.70ms   68.10%
    Req/Sec   506.32    134.97     2.77k    84.57%
  164029 requests in 5.10s, 2.82GB read
Requests/sec:  32157.57
Transfer/sec:    566.99MB
[info] CPU 6467.2% | Mem 609MiB

=== Best: 32315 req/s (CPU: 6501.1%, Mem: 607MiB) ===
[info] saved results/static/4096/userver.json
httparena-bench-userver
httparena-bench-userver

==============================================
=== userver / static / 6800c (tool=wrk) ===
==============================================
[info] waiting for server...
[info] server ready

[run 1/3]
Running 5s test @ http://localhost:8080
  64 threads and 6800 connections
  Thread Stats   Avg      Stdev     Max   +/- Stdev
    Latency   206.89ms  107.66ms 732.96ms   68.88%
    Req/Sec   506.24    221.86     8.91k    86.26%
  164004 requests in 5.10s, 2.82GB read
Requests/sec:  32151.77
Transfer/sec:    566.66MB
[info] CPU 6440.4% | Mem 757MiB

[run 2/3]
Running 5s test @ http://localhost:8080
  64 threads and 6800 connections
  Thread Stats   Avg      Stdev     Max   +/- Stdev
    Latency   206.81ms   97.90ms 656.85ms   69.64%
    Req/Sec   502.81    244.93     7.55k    78.67%
  162893 requests in 5.10s, 2.80GB read
Requests/sec:  31937.91
Transfer/sec:    563.03MB
[info] CPU 6499.2% | Mem 834MiB

[run 3/3]
Running 5s test @ http://localhost:8080
  64 threads and 6800 connections
  Thread Stats   Avg      Stdev     Max   +/- Stdev
    Latency   206.90ms   99.87ms 713.78ms   67.80%
    Req/Sec   509.08    236.94     6.14k    81.00%
  164038 requests in 5.10s, 2.83GB read
Requests/sec:  32165.32
Transfer/sec:    567.25MB
[info] CPU 6486.4% | Mem 848MiB

=== Best: 32165 req/s (CPU: 6486.4%, Mem: 848MiB) ===
[info] saved results/static/6800/userver.json
httparena-bench-userver
httparena-bench-userver

==============================================
=== userver / async-db / 1024c (tool=gcannon) ===
==============================================
[info] waiting for server...
[info] server ready

[run 1/3]
gcannon v0.5.3
  Target:    localhost:8080/
  Threads:   64
  Conns:     1024 (16/thread)
  Pipeline:  1
  Req/conn:  25
  Templates: 5
  Expected:  200
  Duration:  10s


  Thread Stats   Avg      p50      p90      p99    p99.9
    Latency   2.21ms    836us   2.81ms   35.60ms   81.00ms

  4334149 requests in 10.00s, 4334148 responses
  Throughput: 433.33K req/s
  Bandwidth:  527.44MB/s
  Status codes: 2xx=1233596, 3xx=0, 4xx=0, 5xx=3100552
  Latency samples: 4334135 / 4334148 responses (100.0%)
  Reconnects: 173241
  Per-template: 866224,866367,867496,867572,866476
  Per-template-ok: 245638,246651,247800,246861,246644

  WARNING: 3100552/4334148 responses (71.5%) had unexpected status (expected 2xx)
[info] CPU 6265.6% | Mem 352MiB

[run 2/3]
gcannon v0.5.3
  Target:    localhost:8080/
  Threads:   64
  Conns:     1024 (16/thread)
  Pipeline:  1
  Req/conn:  25
  Templates: 5
  Expected:  200
  Duration:  10s


  Thread Stats   Avg      p50      p90      p99    p99.9
    Latency   3.80ms   3.20ms   6.63ms   12.10ms   18.00ms

  2327987 requests in 10.00s, 2327990 responses
  Throughput: 232.75K req/s
  Bandwidth:  897.57MB/s
  Status codes: 2xx=2327910, 3xx=0, 4xx=0, 5xx=80
  Latency samples: 2327986 / 2327990 responses (100.0%)
  Reconnects: 92941
  Per-template: 463796,465420,467042,467048,464680
  Per-template-ok: 463780,465403,467033,467030,464659

  WARNING: 80/2327990 responses (0.0%) had unexpected status (expected 2xx)
[info] CPU 5392.7% | Mem 318MiB

[run 3/3]
gcannon v0.5.3
  Target:    localhost:8080/
  Threads:   64
  Conns:     1024 (16/thread)
  Pipeline:  1
  Req/conn:  25
  Templates: 5
  Expected:  200
  Duration:  10s


  Thread Stats   Avg      p50      p90      p99    p99.9
    Latency   3.66ms   3.11ms   6.28ms   11.40ms   16.90ms

  2428624 requests in 10.00s, 2428629 responses
  Throughput: 242.80K req/s
  Bandwidth:  936.05MB/s
  Status codes: 2xx=2428276, 3xx=0, 4xx=0, 5xx=353
  Latency samples: 2428624 / 2428629 responses (100.0%)
  Reconnects: 97127
  Per-template: 484387,485636,486897,486838,484866
  Per-template-ok: 484336,485558,486827,486776,484774

  WARNING: 353/2428629 responses (0.0%) had unexpected status (expected 2xx)
[info] CPU 5823.6% | Mem 316MiB

=== Best: 242827 req/s (CPU: 5823.6%, Mem: 316MiB) ===
[info] input BW: 16.21MB/s (avg template: 70 bytes)
[info] saved results/async-db/1024/userver.json
httparena-bench-userver
httparena-bench-userver
[info] skip: userver does not subscribe to crud
[info] skip: userver does not subscribe to fortunes
[info] skip: userver does not subscribe to baseline-h2
[info] skip: userver does not subscribe to static-h2
[info] skip: userver does not subscribe to baseline-h2c
[info] skip: userver does not subscribe to json-h2c
[info] skip: userver does not subscribe to baseline-h3
[info] skip: userver does not subscribe to static-h3
[info] skip: userver does not subscribe to gateway-64
[info] skip: userver does not subscribe to gateway-h3
[info] skip: userver does not subscribe to production-stack
[info] skip: userver does not subscribe to unary-grpc
[info] skip: userver does not subscribe to unary-grpc-tls
[info] skip: userver does not subscribe to stream-grpc
[info] skip: userver does not subscribe to stream-grpc-tls
[info] skip: userver does not subscribe to echo-ws
[info] skip: userver does not subscribe to echo-ws-pipeline
[info] rebuilding site/data/*.json
[updated] /home/diogo/actions-runner/_work/HttpArena/HttpArena/site/data/frameworks.json
[updated] /home/diogo/actions-runner/_work/HttpArena/HttpArena/site/data/api-16-1024.json
[updated] /home/diogo/actions-runner/_work/HttpArena/HttpArena/site/data/api-4-256.json
[updated] /home/diogo/actions-runner/_work/HttpArena/HttpArena/site/data/async-db-1024.json
[updated] /home/diogo/actions-runner/_work/HttpArena/HttpArena/site/data/baseline-4096.json
[updated] /home/diogo/actions-runner/_work/HttpArena/HttpArena/site/data/baseline-512.json
[updated] /home/diogo/actions-runner/_work/HttpArena/HttpArena/site/data/json-4096.json
[updated] /home/diogo/actions-runner/_work/HttpArena/HttpArena/site/data/json-comp-16384.json
[updated] /home/diogo/actions-runner/_work/HttpArena/HttpArena/site/data/json-comp-4096.json
[updated] /home/diogo/actions-runner/_work/HttpArena/HttpArena/site/data/json-comp-512.json
[updated] /home/diogo/actions-runner/_work/HttpArena/HttpArena/site/data/limited-conn-4096.json
[updated] /home/diogo/actions-runner/_work/HttpArena/HttpArena/site/data/limited-conn-512.json
[updated] /home/diogo/actions-runner/_work/HttpArena/HttpArena/site/data/pipelined-4096.json
[updated] /home/diogo/actions-runner/_work/HttpArena/HttpArena/site/data/pipelined-512.json
[updated] /home/diogo/actions-runner/_work/HttpArena/HttpArena/site/data/static-1024.json
[updated] /home/diogo/actions-runner/_work/HttpArena/HttpArena/site/data/static-4096.json
[updated] /home/diogo/actions-runner/_work/HttpArena/HttpArena/site/data/static-6800.json
[updated] /home/diogo/actions-runner/_work/HttpArena/HttpArena/site/data/upload-256.json
[updated] /home/diogo/actions-runner/_work/HttpArena/HttpArena/site/data/upload-32.json
[updated] /home/diogo/actions-runner/_work/HttpArena/HttpArena/site/data/current.json
[info] done
httparena-postgres
[info] restoring loopback MTU to 65536

@botanegg botanegg force-pushed the userver-benchmarks branch 2 times, most recently from c8bc1fe to 3c65965 Compare June 6, 2026 10:40
10 benchmark profiles (baseline, pipelined, limited-conn, json,
json-comp, upload, api-4, api-16, static, async-db)

- 6 HTTP handlers (baseline11, baseline2, json, plaintext, upload, async_db)
- gzip compression middleware
- Dockerfile based on ubuntu-24.04-userver
@botanegg botanegg force-pushed the userver-benchmarks branch from 3c65965 to c5d9547 Compare June 6, 2026 10:48
@botanegg
Copy link
Copy Markdown
Contributor Author

botanegg commented Jun 6, 2026

/benchmark -f userver --save

@github-actions
Copy link
Copy Markdown
Contributor

github-actions Bot commented Jun 6, 2026

👋 /benchmark request received. A collaborator will review and approve the run.

@botanegg
Copy link
Copy Markdown
Contributor Author

botanegg commented Jun 6, 2026

@MDA2AV ready to merge after benchmarks run

@github-actions
Copy link
Copy Markdown
Contributor

github-actions Bot commented Jun 6, 2026

Benchmark Results

Framework: userver | Test: all tests

Test Conn RPS CPU Mem Δ RPS Δ Mem
baseline 512 938,499 6352.8% 213MiB +55.8% +7.0%
baseline 4096 1,040,628 6556.3% 431MiB +6.4% -3.1%
pipelined 512 1,444,364 6593.4% 257MiB -0.4% -1.5%
pipelined 4096 1,323,265 6703.7% 905MiB -1.3% +0.8%
limited-conn 512 661,612 6010.5% 237MiB -5.7% -1.2%
limited-conn 4096 878,475 6391.3% 570MiB +3.3% -1.4%
json 4096 818,115 6323.7% 522MiB NEW NEW
json-comp 512 502,618 6502.4% 243MiB NEW NEW
json-comp 4096 494,592 6572.5% 552MiB NEW NEW
json-comp 16384 486,545 6158.5% 1.5GiB NEW NEW
upload 32 708 3220.2% 1.0GiB NEW NEW
upload 256 619 6478.7% 4.3GiB NEW NEW
api-4 256 58,657 402.5% 194MiB NEW NEW
api-16 1024 134,831 1773.7% 380MiB NEW NEW
static 1024 164,982 6585.0% 460MiB -0.2% +5.0%
static 4096 186,711 6454.8% 928MiB -0.9% -0.3%
static 6800 195,082 6401.7% 1.3GiB +0.4% ~0%
async-db 1024 232,577 5621.6% 312MiB NEW NEW
Full log
  64 threads and 4096 connections
  Thread Stats   Avg      Stdev     Max   +/- Stdev
    Latency    22.16ms   10.14ms 128.33ms   68.45%
    Req/Sec     2.91k   409.70    15.84k    97.61%
  944549 requests in 5.10s, 56.11GB read
Requests/sec: 185210.73
Transfer/sec:     11.00GB
[info] CPU 6543.4% | Mem 955MiB

[run 3/3]
Running 5s test @ http://localhost:8080
  64 threads and 4096 connections
  Thread Stats   Avg      Stdev     Max   +/- Stdev
    Latency    22.20ms   10.51ms 128.28ms   69.32%
    Req/Sec     2.92k   452.54    15.34k    98.04%
  945815 requests in 5.10s, 56.19GB read
Requests/sec: 185480.36
Transfer/sec:     11.02GB
[info] CPU 6459.0% | Mem 922MiB

=== Best: 186711 req/s (CPU: 6454.8%, Mem: 928MiB) ===
[info] saved results/static/4096/userver.json
httparena-bench-userver
httparena-bench-userver

==============================================
=== userver / static / 6800c (tool=wrk) ===
==============================================
[info] waiting for server...
[info] server ready

[run 1/3]
Running 5s test @ http://localhost:8080
  64 threads and 6800 connections
  Thread Stats   Avg      Stdev     Max   +/- Stdev
    Latency    35.62ms   18.72ms 210.94ms   71.42%
    Req/Sec     3.04k   346.75     9.24k    95.89%
  985981 requests in 5.10s, 58.57GB read
Requests/sec: 193332.64
Transfer/sec:     11.49GB
[info] CPU 6483.1% | Mem 1.3GiB

[run 2/3]
Running 5s test @ http://localhost:8080
  64 threads and 6800 connections
  Thread Stats   Avg      Stdev     Max   +/- Stdev
    Latency    35.48ms   18.95ms 225.76ms   72.40%
    Req/Sec     3.06k   457.81    16.79k    97.67%
  993427 requests in 5.10s, 59.02GB read
Requests/sec: 194767.38
Transfer/sec:     11.57GB
[info] CPU 6573.1% | Mem 1.3GiB

[run 3/3]
Running 5s test @ http://localhost:8080
  64 threads and 6800 connections
  Thread Stats   Avg      Stdev     Max   +/- Stdev
    Latency    35.48ms   19.18ms 251.96ms   72.57%
    Req/Sec     3.07k   425.89    15.48k    97.39%
  995151 requests in 5.10s, 59.12GB read
Requests/sec: 195082.63
Transfer/sec:     11.59GB
[info] CPU 6401.7% | Mem 1.3GiB

=== Best: 195082 req/s (CPU: 6401.7%, Mem: 1.3GiB) ===
[info] saved results/static/6800/userver.json
httparena-bench-userver
httparena-bench-userver

==============================================
=== userver / async-db / 1024c (tool=gcannon) ===
==============================================
[info] waiting for server...
[info] server ready

[run 1/3]
gcannon v0.5.3
  Target:    localhost:8080/
  Threads:   64
  Conns:     1024 (16/thread)
  Pipeline:  1
  Req/conn:  25
  Templates: 5
  Expected:  200
  Duration:  10s


  Thread Stats   Avg      p50      p90      p99    p99.9
    Latency   2.71ms   1.17ms   6.06ms   27.10ms   50.80ms

  3450817 requests in 10.00s, 3450824 responses
  Throughput: 345.01K req/s
  Bandwidth:  671.31MB/s
  Status codes: 2xx=1662416, 3xx=0, 4xx=0, 5xx=1788408
  Latency samples: 3450824 / 3450824 responses (100.0%)
  Reconnects: 137680
  Per-template: 689336,690252,691094,690670,689472
  Per-template-ok: 331317,332206,332717,333049,333127

  WARNING: 1788408/3450824 responses (51.8%) had unexpected status (expected 2xx)
[info] CPU 5962.9% | Mem 355MiB

[run 2/3]
gcannon v0.5.3
  Target:    localhost:8080/
  Threads:   64
  Conns:     1024 (16/thread)
  Pipeline:  1
  Req/conn:  25
  Templates: 5
  Expected:  200
  Duration:  10s


  Thread Stats   Avg      p50      p90      p99    p99.9
    Latency   4.10ms   3.54ms   7.06ms   12.20ms   17.80ms

  2184713 requests in 10.00s, 2184715 responses
  Throughput: 218.42K req/s
  Bandwidth:  842.15MB/s
  Status codes: 2xx=2184715, 3xx=0, 4xx=0, 5xx=0
  Latency samples: 2184713 / 2184715 responses (100.0%)
  Reconnects: 87259
  Per-template: 435941,436749,438006,437779,436238
  Per-template-ok: 435941,436749,438006,437779,436238
[info] CPU 5103.4% | Mem 322MiB

[run 3/3]
gcannon v0.5.3
  Target:    localhost:8080/
  Threads:   64
  Conns:     1024 (16/thread)
  Pipeline:  1
  Req/conn:  25
  Templates: 5
  Expected:  200
  Duration:  10s


  Thread Stats   Avg      p50      p90      p99    p99.9
    Latency   3.79ms   3.28ms   6.45ms   11.50ms   17.10ms

  2326764 requests in 10.00s, 2326764 responses
  Throughput: 232.62K req/s
  Bandwidth:  896.60MB/s
  Status codes: 2xx=2325772, 3xx=0, 4xx=0, 5xx=992
  Latency samples: 2326754 / 2326764 responses (100.0%)
  Reconnects: 93013
  Per-template: 464117,465269,466319,466383,464666
  Per-template-ok: 463906,465078,466140,466181,464457

  WARNING: 992/2326764 responses (0.0%) had unexpected status (expected 2xx)
[info] CPU 5621.6% | Mem 312MiB

=== Best: 232577 req/s (CPU: 5621.6%, Mem: 312MiB) ===
[info] input BW: 15.53MB/s (avg template: 70 bytes)
[info] saved results/async-db/1024/userver.json
httparena-bench-userver
httparena-bench-userver
[info] skip: userver does not subscribe to crud
[info] skip: userver does not subscribe to fortunes
[info] skip: userver does not subscribe to baseline-h2
[info] skip: userver does not subscribe to static-h2
[info] skip: userver does not subscribe to baseline-h2c
[info] skip: userver does not subscribe to json-h2c
[info] skip: userver does not subscribe to baseline-h3
[info] skip: userver does not subscribe to static-h3
[info] skip: userver does not subscribe to gateway-64
[info] skip: userver does not subscribe to gateway-h3
[info] skip: userver does not subscribe to production-stack
[info] skip: userver does not subscribe to unary-grpc
[info] skip: userver does not subscribe to unary-grpc-tls
[info] skip: userver does not subscribe to stream-grpc
[info] skip: userver does not subscribe to stream-grpc-tls
[info] skip: userver does not subscribe to echo-ws
[info] skip: userver does not subscribe to echo-ws-pipeline
[info] rebuilding site/data/*.json
[updated] /home/diogo/actions-runner/_work/HttpArena/HttpArena/site/data/frameworks.json
[updated] /home/diogo/actions-runner/_work/HttpArena/HttpArena/site/data/api-16-1024.json
[updated] /home/diogo/actions-runner/_work/HttpArena/HttpArena/site/data/api-4-256.json
[updated] /home/diogo/actions-runner/_work/HttpArena/HttpArena/site/data/async-db-1024.json
[updated] /home/diogo/actions-runner/_work/HttpArena/HttpArena/site/data/baseline-4096.json
[updated] /home/diogo/actions-runner/_work/HttpArena/HttpArena/site/data/baseline-512.json
[updated] /home/diogo/actions-runner/_work/HttpArena/HttpArena/site/data/json-4096.json
[updated] /home/diogo/actions-runner/_work/HttpArena/HttpArena/site/data/json-comp-16384.json
[updated] /home/diogo/actions-runner/_work/HttpArena/HttpArena/site/data/json-comp-4096.json
[updated] /home/diogo/actions-runner/_work/HttpArena/HttpArena/site/data/json-comp-512.json
[updated] /home/diogo/actions-runner/_work/HttpArena/HttpArena/site/data/limited-conn-4096.json
[updated] /home/diogo/actions-runner/_work/HttpArena/HttpArena/site/data/limited-conn-512.json
[updated] /home/diogo/actions-runner/_work/HttpArena/HttpArena/site/data/pipelined-4096.json
[updated] /home/diogo/actions-runner/_work/HttpArena/HttpArena/site/data/pipelined-512.json
[updated] /home/diogo/actions-runner/_work/HttpArena/HttpArena/site/data/static-1024.json
[updated] /home/diogo/actions-runner/_work/HttpArena/HttpArena/site/data/static-4096.json
[updated] /home/diogo/actions-runner/_work/HttpArena/HttpArena/site/data/static-6800.json
[updated] /home/diogo/actions-runner/_work/HttpArena/HttpArena/site/data/upload-256.json
[updated] /home/diogo/actions-runner/_work/HttpArena/HttpArena/site/data/upload-32.json
[updated] /home/diogo/actions-runner/_work/HttpArena/HttpArena/site/data/current.json
[info] done
httparena-postgres
[info] restoring loopback MTU to 65536

@MDA2AV MDA2AV merged commit 1ae067f into MDA2AV:main Jun 6, 2026
4 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants