2022 02 18 bench results
Andrea edited this page Feb 18, 2022
·
2 revisions
Benchmarked on: Intel(R) Xeon(R) CPU E5-1650 v3 @ 3.50GHz, 32G RAM
Config:
cfe3081 Wed, 16 Feb 2022 10:16:44 +0100 GitHub =akka update akka-* versions (#194)
- GRPC_BENCHMARK_DURATION=120s
- GRPC_BENCHMARK_WARMUP=30s
- GRPC_SERVER_RAM=512m
- GRPC_CLIENT_CONNECTIONS=50
- GRPC_CLIENT_CONCURRENCY=1000
- GRPC_CLIENT_QPS=0
- GRPC_CLIENT_CPUS=9
- GRPC_REQUEST_SCENARIO=complex_proto
Note: I had to disable erlang and elixir benches because I couldn't get them to build successfully.
Note: It seems the ghz client is not able to generate sufficient load as before.
1 CPU Server
-----------------------------------------------------------------------------------------------------------------------------------------
| name | req/s | avg. latency | 90 % in | 95 % in | 99 % in | avg. cpu | avg. memory |
-----------------------------------------------------------------------------------------------------------------------------------------
| go_vtgrpc | 17224 | 26.95 ms | 69.28 ms | 135.67 ms | 227.63 ms | 99.11% | 25.32 MiB |
| dotnet_grpc | 17002 | 26.19 ms | 58.06 ms | 120.51 ms | 211.86 ms | 93.13% | 111.64 MiB |
| scala_akka | 16988 | 28.22 ms | 81.22 ms | 131.90 ms | 209.51 ms | 93.06% | 230.61 MiB |
| rust_tonic_st | 16954 | 28.18 ms | 63.55 ms | 131.84 ms | 220.02 ms | 63.04% | 12.59 MiB |
| crystal_grpc | 16846 | 37.06 ms | 73.30 ms | 109.96 ms | 175.06 ms | 88.48% | 34.39 MiB |
| java_quarkus | 16839 | 29.04 ms | 73.62 ms | 125.59 ms | 209.85 ms | 74.15% | 109.24 MiB |
| cpp_grpc_mt | 16810 | 27.53 ms | 55.71 ms | 143.37 ms | 246.37 ms | 80.48% | 8.16 MiB |
| java_hotspot_grpc_g1gc | 16806 | 27.64 ms | 77.66 ms | 126.32 ms | 212.06 ms | 100.65% | 144.11 MiB |
| rust_tonic_mt | 16797 | 28.11 ms | 59.40 ms | 131.40 ms | 228.64 ms | 78.03% | 11.03 MiB |
| rust_grpcio | 16797 | 27.99 ms | 64.95 ms | 143.83 ms | 237.68 ms | 75.02% | 16.09 MiB |
| cpp_asio_grpc | 16792 | 28.95 ms | 68.67 ms | 142.07 ms | 228.86 ms | 84.78% | 7.79 MiB |
| cpp_grpc_st | 16788 | 27.39 ms | 56.48 ms | 141.36 ms | 235.13 ms | 81.31% | 7.59 MiB |
| java_hotspot_grpc_zgc | 16773 | 28.74 ms | 74.47 ms | 123.41 ms | 207.70 ms | 100.69% | 112.72 MiB |
| java_hotspot_grpc_pgc | 16722 | 26.23 ms | 71.06 ms | 120.34 ms | 242.93 ms | 94.65% | 191.89 MiB |
| rust_thruster_st | 16704 | 29.64 ms | 74.34 ms | 137.19 ms | 229.37 ms | 65.77% | 9.18 MiB |
| java_hotspot_grpc_she | 16640 | 30.03 ms | 83.71 ms | 130.19 ms | 219.94 ms | 101.63% | 397.97 MiB |
| rust_thruster_mt | 16530 | 31.07 ms | 84.78 ms | 144.28 ms | 227.57 ms | 81.31% | 9.5 MiB |
| java_hotspot_grpc_sgc | 16428 | 30.06 ms | 92.93 ms | 141.32 ms | 216.59 ms | 99.26% | 95.04 MiB |
| kotlin_grpc | 15672 | 45.83 ms | 95.35 ms | 110.69 ms | 167.15 ms | 102.48% | 207.66 MiB |
| java_openj9_grpc_gencon | 15389 | 35.36 ms | 113.67 ms | 180.72 ms | 277.21 ms | 101.83% | 104.99 MiB |
| java_quarkus_native | 15021 | 56.72 ms | 75.06 ms | 86.12 ms | 107.89 ms | 100.32% | 88.38 MiB |
| go_grpc | 14331 | 51.50 ms | 89.03 ms | 99.60 ms | 127.38 ms | 97.07% | 29.63 MiB |
| java_micronaut_workstealing | 11019 | 82.60 ms | 112.24 ms | 165.93 ms | 469.38 ms | 100.34% | 470.75 MiB |
| java_micronaut | 10927 | 82.96 ms | 116.03 ms | 166.83 ms | 283.11 ms | 101.78% | 477.21 MiB |
| csharp_grpc | 6693 | 144.63 ms | 186.52 ms | 192.05 ms | 201.90 ms | 101.11% | 107.29 MiB |
| scala_zio | 5837 | 162.32 ms | 254.83 ms | 294.07 ms | 387.42 ms | 100.93% | 253.86 MiB |
| node_grpcjs_st | 3979 | 246.90 ms | 275.22 ms | 282.27 ms | 298.84 ms | 100.51% | 46.57 MiB |
| scala_fs2 | 2706 | 367.56 ms | 396.56 ms | 501.10 ms | 599.07 ms | 101.86% | 221.21 MiB |
| python_async_grpc | 2592 | 383.94 ms | 405.86 ms | 410.58 ms | 437.32 ms | 101.2% | 30.69 MiB |
| ruby_grpc | 2187 | 454.61 ms | 497.00 ms | 512.73 ms | 588.38 ms | 102.05% | 37.48 MiB |
| python_grpc | 2127 | 467.79 ms | 494.65 ms | 498.73 ms | 506.24 ms | 101.52% | 35.24 MiB |
| java_aot | 1665 | 592.25 ms | 998.13 ms | 1.18 s | 1.49 s | 100.07% | 197.02 MiB |
| php_grpc | 935 | 1.06 s | 1.10 s | 1.10 s | 1.18 s | 102.38% | 67.28 MiB |
-----------------------------------------------------------------------------------------------------------------------------------------
2 CPU Server
-----------------------------------------------------------------------------------------------------------------------------------------
| name | req/s | avg. latency | 90 % in | 95 % in | 99 % in | avg. cpu | avg. memory |
-----------------------------------------------------------------------------------------------------------------------------------------
| java_quarkus_native | 18150 | 24.45 ms | 55.17 ms | 119.92 ms | 185.30 ms | 166.93% | 88.63 MiB |
| csharp_grpc | 18147 | 21.42 ms | 31.76 ms | 125.62 ms | 222.92 ms | 191.44% | 92.41 MiB |
| java_micronaut_workstealing | 18061 | 20.97 ms | 38.70 ms | 119.08 ms | 237.52 ms | 167.53% | 345.59 MiB |
| java_quarkus | 18011 | 22.26 ms | 48.73 ms | 106.85 ms | 225.03 ms | 94.39% | 108.1 MiB |
| java_micronaut | 18011 | 20.99 ms | 43.81 ms | 119.06 ms | 231.38 ms | 165.45% | 341.78 MiB |
| java_hotspot_grpc_sgc | 18003 | 21.00 ms | 38.07 ms | 124.53 ms | 236.37 ms | 121.39% | 93.18 MiB |
| java_openj9_grpc_gencon | 17970 | 19.53 ms | 33.26 ms | 118.11 ms | 240.71 ms | 130.51% | 98.86 MiB |
| java_hotspot_grpc_g1gc | 17938 | 20.25 ms | 34.92 ms | 117.59 ms | 249.21 ms | 124.82% | 130.36 MiB |
| java_hotspot_grpc_she | 17933 | 19.95 ms | 30.24 ms | 113.12 ms | 256.53 ms | 127.53% | 399.98 MiB |
| java_hotspot_grpc_pgc | 17926 | 19.87 ms | 33.13 ms | 122.83 ms | 254.86 ms | 120.44% | 198.87 MiB |
| kotlin_grpc | 17923 | 20.27 ms | 36.14 ms | 116.54 ms | 233.97 ms | 134.76% | 212.38 MiB |
| scala_akka | 17894 | 23.53 ms | 42.46 ms | 135.65 ms | 232.72 ms | 91.72% | 229.99 MiB |
| rust_tonic_mt | 17455 | 21.48 ms | 45.02 ms | 122.80 ms | 250.33 ms | 96.57% | 12.31 MiB |
| java_hotspot_grpc_zgc | 17401 | 20.17 ms | 36.43 ms | 122.62 ms | 264.99 ms | 131.21% | 124.1 MiB |
| crystal_grpc | 17397 | 35.74 ms | 72.48 ms | 108.08 ms | 170.52 ms | 86.68% | 34.57 MiB |
| go_grpc | 17272 | 25.38 ms | 57.08 ms | 146.43 ms | 248.88 ms | 131.07% | 25.58 MiB |
| go_vtgrpc | 17223 | 25.82 ms | 63.65 ms | 140.82 ms | 245.17 ms | 101.12% | 25.1 MiB |
| dotnet_grpc | 17066 | 23.54 ms | 50.58 ms | 113.74 ms | 220.22 ms | 99.07% | 181.91 MiB |
| cpp_grpc_mt | 17045 | 22.45 ms | 50.11 ms | 117.25 ms | 244.19 ms | 100.25% | 16.88 MiB |
| cpp_asio_grpc | 17026 | 21.50 ms | 48.35 ms | 113.27 ms | 247.97 ms | 103.72% | 16.65 MiB |
| rust_grpcio | 16990 | 23.21 ms | 55.05 ms | 119.66 ms | 239.25 ms | 83.7% | 16.58 MiB |
| rust_tonic_st | 16987 | 28.14 ms | 64.76 ms | 128.36 ms | 222.20 ms | 64.64% | 12.39 MiB |
| cpp_grpc_st | 16829 | 28.09 ms | 66.53 ms | 146.50 ms | 238.50 ms | 80.87% | 7.67 MiB |
| rust_thruster_mt | 16739 | 24.98 ms | 59.85 ms | 148.22 ms | 261.50 ms | 102.57% | 9.54 MiB |
| rust_thruster_st | 16691 | 30.20 ms | 75.56 ms | 136.69 ms | 229.62 ms | 65.98% | 9.22 MiB |
| scala_zio | 14701 | 49.06 ms | 97.12 ms | 115.37 ms | 189.50 ms | 205.08% | 253.58 MiB |
| scala_fs2 | 9776 | 100.41 ms | 66.43 ms | 77.33 ms | 93.96 ms | 207.52% | 237.82 MiB |
| java_aot | 8208 | 109.69 ms | 198.45 ms | 241.20 ms | 306.54 ms | 199.09% | 210.26 MiB |
| node_grpcjs_st | 5713 | 170.65 ms | 179.04 ms | 184.58 ms | 192.09 ms | 124.97% | 46.39 MiB |
| python_async_grpc | 4145 | 240.29 ms | 261.89 ms | 267.53 ms | 275.99 ms | 139.45% | 31.44 MiB |
| python_grpc | 3424 | 290.87 ms | 303.06 ms | 306.48 ms | 312.50 ms | 143.76% | 35.57 MiB |
| ruby_grpc | 2915 | 341.41 ms | 367.95 ms | 377.99 ms | 408.19 ms | 128.79% | 36.71 MiB |
| php_grpc | 2214 | 449.21 ms | 487.87 ms | 489.44 ms | 491.85 ms | 203.85% | 65.42 MiB |
-----------------------------------------------------------------------------------------------------------------------------------------
3 CPU Server
-----------------------------------------------------------------------------------------------------------------------------------------
| name | req/s | avg. latency | 90 % in | 95 % in | 99 % in | avg. cpu | avg. memory |
-----------------------------------------------------------------------------------------------------------------------------------------
| java_quarkus_native | 18282 | 19.30 ms | 31.02 ms | 106.14 ms | 226.15 ms | 194.71% | 89.12 MiB |
| csharp_grpc | 18250 | 21.20 ms | 31.19 ms | 127.38 ms | 230.90 ms | 199.94% | 93.49 MiB |
| scala_akka | 18192 | 23.32 ms | 44.49 ms | 133.69 ms | 233.39 ms | 97.69% | 235.69 MiB |
| java_openj9_grpc_gencon | 18151 | 20.25 ms | 29.47 ms | 137.04 ms | 255.07 ms | 123.66% | 82.6 MiB |
| java_hotspot_grpc_sgc | 18128 | 20.98 ms | 36.04 ms | 135.62 ms | 250.99 ms | 126.23% | 90.76 MiB |
| java_micronaut | 18107 | 19.40 ms | 28.81 ms | 125.78 ms | 246.47 ms | 172.84% | 278.76 MiB |
| java_hotspot_grpc_g1gc | 18081 | 20.20 ms | 30.43 ms | 134.73 ms | 252.05 ms | 131.45% | 136.38 MiB |
| java_micronaut_workstealing | 18074 | 19.44 ms | 28.19 ms | 116.46 ms | 251.10 ms | 184.0% | 295.72 MiB |
| java_hotspot_grpc_pgc | 18074 | 20.65 ms | 30.77 ms | 132.31 ms | 259.21 ms | 123.24% | 204.78 MiB |
| java_hotspot_grpc_she | 18069 | 20.10 ms | 31.04 ms | 133.57 ms | 262.10 ms | 133.97% | 398.48 MiB |
| java_hotspot_grpc_zgc | 18054 | 20.40 ms | 34.19 ms | 127.41 ms | 250.88 ms | 131.29% | 126.65 MiB |
| java_quarkus | 18017 | 21.89 ms | 55.52 ms | 111.25 ms | 224.81 ms | 102.28% | 110.96 MiB |
| kotlin_grpc | 17989 | 19.91 ms | 29.27 ms | 121.55 ms | 245.08 ms | 154.18% | 214.71 MiB |
| dotnet_grpc | 17919 | 22.14 ms | 50.06 ms | 117.97 ms | 214.08 ms | 97.54% | 173.98 MiB |
| go_grpc | 17285 | 25.64 ms | 52.79 ms | 143.74 ms | 249.73 ms | 128.97% | 25.66 MiB |
| go_vtgrpc | 17213 | 25.74 ms | 52.54 ms | 141.63 ms | 251.57 ms | 102.55% | 25.21 MiB |
| cpp_grpc_mt | 17086 | 22.37 ms | 47.29 ms | 127.08 ms | 258.72 ms | 101.55% | 19.3 MiB |
| rust_tonic_mt | 17071 | 21.94 ms | 44.55 ms | 118.43 ms | 253.27 ms | 104.57% | 14.43 MiB |
| cpp_asio_grpc | 17048 | 22.14 ms | 46.65 ms | 123.77 ms | 256.20 ms | 106.49% | 20.89 MiB |
| rust_grpcio | 17010 | 23.16 ms | 52.93 ms | 118.40 ms | 248.31 ms | 93.85% | 17.85 MiB |
| scala_zio | 16997 | 26.44 ms | 90.70 ms | 140.65 ms | 216.80 ms | 245.33% | 269.49 MiB |
| rust_tonic_st | 16990 | 28.62 ms | 63.88 ms | 125.30 ms | 220.67 ms | 63.63% | 12.61 MiB |
| crystal_grpc | 16880 | 36.85 ms | 74.62 ms | 108.64 ms | 171.75 ms | 87.93% | 34.87 MiB |
| rust_thruster_mt | 16828 | 23.90 ms | 49.36 ms | 145.17 ms | 275.07 ms | 111.31% | 10.3 MiB |
| cpp_grpc_st | 16780 | 27.78 ms | 56.57 ms | 144.47 ms | 246.73 ms | 81.67% | 7.69 MiB |
| rust_thruster_st | 16654 | 30.46 ms | 81.43 ms | 142.00 ms | 237.25 ms | 65.62% | 9.04 MiB |
| java_aot | 15041 | 44.71 ms | 105.49 ms | 133.67 ms | 191.49 ms | 282.45% | 144.22 MiB |
| scala_fs2 | 13192 | 73.21 ms | 87.17 ms | 95.91 ms | 717.38 ms | 280.82% | 238.94 MiB |
| node_grpcjs_st | 5668 | 172.34 ms | 180.16 ms | 186.18 ms | 197.00 ms | 123.59% | 45.91 MiB |
| python_async_grpc | 4184 | 238.03 ms | 261.91 ms | 268.18 ms | 275.72 ms | 139.48% | 32.23 MiB |
| php_grpc | 4020 | 247.51 ms | 274.91 ms | 276.51 ms | 279.14 ms | 309.57% | 67.65 MiB |
| python_grpc | 3468 | 287.19 ms | 300.37 ms | 304.29 ms | 310.63 ms | 144.42% | 35.21 MiB |
| ruby_grpc | 2915 | 341.16 ms | 366.65 ms | 374.57 ms | 409.57 ms | 124.57% | 37.65 MiB |
-----------------------------------------------------------------------------------------------------------------------------------------