Helidon - Java Microservice Framework, tuned type

[tomas-langer]

Addition of stream grpc to both Helidon versions

Description

A follow up for #503, this PR adds a Helidon tuned version.

[tomas-langer]

sorry, committed too much, I will re-push

done

[MDA2AV]

/benchmark -f helidon-tuned --save

[github-actions]

👋 /benchmark request received. A collaborator will review and approve the run.

[github-actions]

🚀 Benchmark run triggered for helidon-tuned (all tests) with --save. Results will be posted here when done.

[github-actions]

Benchmark Results

Framework: helidon-tuned | Test: all tests

Test	Conn	RPS	CPU	Mem	Δ RPS	Δ Mem
baseline	512	1,707,929	6257.4%	6.8GiB	NEW	NEW
baseline	4096	1,529,299	6209.9%	6.7GiB	NEW	NEW
pipelined	512	10,901,216	6133.0%	6.5GiB	NEW	NEW
pipelined	4096	10,029,440	5994.1%	7.6GiB	NEW	NEW
limited-conn	512	577,823	4862.2%	3.8GiB	NEW	NEW
limited-conn	4096	584,873	4659.3%	3.0GiB	NEW	NEW
json	4096	757,679	6278.5%	6.4GiB	NEW	NEW
json-comp	512	370,856	5912.8%	5.8GiB	NEW	NEW
json-comp	4096	356,992	5894.8%	7.7GiB	NEW	NEW
json-comp	16384	306,148	5702.8%	12.9GiB	NEW	NEW
json-tls	4096	618,043	6211.7%	8.3GiB	NEW	NEW
upload	32	2,024	3089.6%	5.4GiB	NEW	NEW
upload	256	2,292	6142.0%	5.9GiB	NEW	NEW
api-4	256	43,456	369.0%	571.9MiB	NEW	NEW
api-16	1024	132,308	1571.7%	1.5GiB	NEW	NEW
static	1024	343,964	5833.2%	5.0GiB	NEW	NEW
static	4096	326,387	5696.8%	5.8GiB	NEW	NEW
static	6800	298,263	5476.1%	6.7GiB	NEW	NEW
async-db	1024	68,541	1860.1%	2.2GiB	NEW	NEW
baseline-h2	256	2,223,118	5910.0%	23.2GiB	NEW	NEW
baseline-h2	1024	2,027,252	5700.3%	26.1GiB	NEW	NEW
static-h2	256	265,851	6025.3%	15.0GiB	NEW	NEW
static-h2	1024	310,187	6190.0%	17.2GiB	NEW	NEW
unary-grpc	256	960,380	6053.4%	25.7GiB	NEW	NEW
unary-grpc	1024	898,409	5869.2%	27.1GiB	NEW	NEW
unary-grpc-tls	256	850,405	6111.4%	24.9GiB	NEW	NEW
unary-grpc-tls	1024	823,252	5932.9%	26.2GiB	NEW	NEW
stream-grpc	64	554,000	1086.2%	4.1GiB	NEW	NEW
stream-grpc-tls	64	513,000	1452.3%	4.2GiB	NEW	NEW
echo-ws	512	2,332,357	6439.2%	986.7MiB	NEW	NEW
echo-ws	4096	2,212,682	6415.9%	1.7GiB	NEW	NEW
echo-ws	16384	1,981,975	6154.8%	3.3GiB	NEW	NEW

Full log

  Throughput: 1.86M req/s
  Bandwidth:  12.79MB/s
  WS upgrades: 16381
  WS frames:   9282722
  Latency samples: 9282722 / 9282722 responses (100.0%)
  CPU: 6498.2% | Mem: 2.7GiB

[run 3/3]
gcannon v0.5.0 [WS]
  Target:    localhost:8080/ws
  Threads:   64
  Conns:     16384 (256/thread)
  Pipeline:  1
  Req/conn:  unlimited (keep-alive)
  Expected:  200
  Duration:  5s


  Thread Stats   Avg      p50      p90      p99    p99.9
    Latency   7.56ms   7.66ms   9.24ms   11.30ms   25.70ms

  9926259 frames sent in 5.00s, 9909876 frames received
  Throughput: 1.98M req/s
  Bandwidth:  13.63MB/s
  WS upgrades: 16383
  WS frames:   9909876
  Latency samples: 9909876 / 9909876 responses (100.0%)
  CPU: 6154.8% | Mem: 3.3GiB

=== Best: 1981975 req/s (CPU: 6154.8%, Mem: 3.3GiB) ===
[saved] results/echo-ws/16384/helidon-tuned.json
[saved] site/static/logs/echo-ws/16384/helidon-tuned.log
httparena-bench-helidon-tuned
httparena-bench-helidon-tuned
[updated] site/data/frameworks.json
[updated] site/data/api-16-1024.json
[updated] site/data/api-4-256.json
[updated] site/data/async-db-1024.json
[updated] site/data/baseline-4096.json
[updated] site/data/baseline-512.json
[updated] site/data/baseline-h2-1024.json
[updated] site/data/baseline-h2-256.json
[updated] site/data/echo-ws-16384.json
[updated] site/data/echo-ws-4096.json
[updated] site/data/echo-ws-512.json
[updated] site/data/json-4096.json
[updated] site/data/json-comp-16384.json
[updated] site/data/json-comp-4096.json
[updated] site/data/json-comp-512.json
[updated] site/data/json-tls-4096.json
[updated] site/data/limited-conn-4096.json
[updated] site/data/limited-conn-512.json
[updated] site/data/pipelined-4096.json
[updated] site/data/pipelined-512.json
[updated] site/data/static-1024.json
[updated] site/data/static-4096.json
[updated] site/data/static-6800.json
[updated] site/data/static-h2-1024.json
[updated] site/data/static-h2-256.json
[updated] site/data/stream-grpc-64.json
[updated] site/data/stream-grpc-tls-64.json
[updated] site/data/unary-grpc-1024.json
[updated] site/data/unary-grpc-256.json
[updated] site/data/unary-grpc-tls-1024.json
[updated] site/data/unary-grpc-tls-256.json
[updated] site/data/upload-256.json
[updated] site/data/upload-32.json
[updated] site/data/current.json
httparena-postgres
httparena-postgres
[restore] Restoring loopback MTU to 65536...
[restore] Restoring CPU governor to performance...
Setting cpu: 0
Setting cpu: 1
Setting cpu: 2
Setting cpu: 3
Setting cpu: 4
Setting cpu: 5
Setting cpu: 6
Setting cpu: 7
Setting cpu: 8
Setting cpu: 9
Setting cpu: 10
Setting cpu: 11
Setting cpu: 12
Setting cpu: 13
Setting cpu: 14
Setting cpu: 15
Setting cpu: 16
Setting cpu: 17
Setting cpu: 18
Setting cpu: 19
Setting cpu: 20
Setting cpu: 21
Setting cpu: 22
Setting cpu: 23
Setting cpu: 24
Setting cpu: 25
Setting cpu: 26
Setting cpu: 27
Setting cpu: 28
Setting cpu: 29
Setting cpu: 30
Setting cpu: 31
Setting cpu: 32
Setting cpu: 33
Setting cpu: 34
Setting cpu: 35
Setting cpu: 36
Setting cpu: 37
Setting cpu: 38
Setting cpu: 39
Setting cpu: 40
Setting cpu: 41
Setting cpu: 42
Setting cpu: 43
Setting cpu: 44
Setting cpu: 45
Setting cpu: 46
Setting cpu: 47
Setting cpu: 48
Setting cpu: 49
Setting cpu: 50
Setting cpu: 51
Setting cpu: 52
Setting cpu: 53
Setting cpu: 54
Setting cpu: 55
Setting cpu: 56
Setting cpu: 57
Setting cpu: 58
Setting cpu: 59
Setting cpu: 60
Setting cpu: 61
Setting cpu: 62
Setting cpu: 63
Setting cpu: 64
Setting cpu: 65
Setting cpu: 66
Setting cpu: 67
Setting cpu: 68
Setting cpu: 69
Setting cpu: 70
Setting cpu: 71
Setting cpu: 72
Setting cpu: 73
Setting cpu: 74
Setting cpu: 75
Setting cpu: 76
Setting cpu: 77
Setting cpu: 78
Setting cpu: 79
Setting cpu: 80
Setting cpu: 81
Setting cpu: 82
Setting cpu: 83
Setting cpu: 84
Setting cpu: 85
Setting cpu: 86
Setting cpu: 87
Setting cpu: 88
Setting cpu: 89
Setting cpu: 90
Setting cpu: 91
Setting cpu: 92
Setting cpu: 93
Setting cpu: 94
Setting cpu: 95
Setting cpu: 96
Setting cpu: 97
Setting cpu: 98
Setting cpu: 99
Setting cpu: 100
Setting cpu: 101
Setting cpu: 102
Setting cpu: 103
Setting cpu: 104
Setting cpu: 105
Setting cpu: 106
Setting cpu: 107
Setting cpu: 108
Setting cpu: 109
Setting cpu: 110
Setting cpu: 111
Setting cpu: 112
Setting cpu: 113
Setting cpu: 114
Setting cpu: 115
Setting cpu: 116
Setting cpu: 117
Setting cpu: 118
Setting cpu: 119
Setting cpu: 120
Setting cpu: 121
Setting cpu: 122
Setting cpu: 123
Setting cpu: 124
Setting cpu: 125
Setting cpu: 126
Setting cpu: 127

[MDA2AV]

you have 180k loc of logs, not sure if you want to check them

[tomas-langer]

I will run the benchmark locally and check the logs. Is it to a specific benchmark?

[MDA2AV]

I will run the benchmark locally and check the logs. Is it to a specific benchmark?

You can see them here in this PR commit, when we do --save they get stored

[tomas-langer]

🚀 Benchmark run triggered for helidon-tuned (all tests) with --save. Results will be posted here when done.

I will analyze the results and push an updated version to get rid of these log files. I hope this is not a bug in the product, as that would take a bit longer time to fix...

[tomas-langer]

It seems that a lot of the cases are "peer resets", where we just need to gracefully consume the exception when writing a response.

[MDA2AV]

Ah ok, I don't mind the logs. Just pointed as could be deteriorating perf. You can also see docker logs for each test on website, for each detailed test view if you click on the framework row, a popup show and on the bottom should have link to docker logs on that test

[tomas-langer]

Thanks, I will try to both update the benchmark to be less verbose, and fix this in Helidon directly, so we do not need to do a benchmark fix...