Process and plot results for benchmark using HA-proxy#75
Conversation
The benchmark uses h1load and siege client. To compare SSL libraries using h1load client the client runtime is used. The longer it takes to execute the test the worse performance. The siege client runs for 10 seconds by default attempting to perform as many requests as possible.
vavroch2010
requested review from
Dankelldesign,
jogme,
mbroz and
npajkovsky
github-project-automation
Bot
moved this from Waiting Review
to Waiting Merge
in Development Board
|
Currently HA-proxy benahcmark runs for ~7 hours. The test uses chain of 10 proxies. The this gives like ~7hours to run complete benchmark. I'm attaching the .tgz file with results. The Files with .merged suffix are processed results so they can be plotted by gnuplot. the .merged file sare also human friendly tables so one can compare the results. The .png files are coming from gnuplot. Those results are still very preliminary as no effort was put to investigate build optins for libraries. All tested libraries are being build with their default build settings. The typical option which is being set is prefix location where to install them. Same goes to ha-proxy and tools. no fancy options are being used. I'm currently re-running those tests using coniguration with single HA-proxy instead of 10 daisy chained proxy instances. According to runtime observation this makes shrinks the benchmark test run by 50%. I'll let it it finish and attach results for single HA-proxy instance once it will be done. I'm attaching result for chain of 10 proxies |
jogme
left a comment
jogme
left a comment
There was a problem hiding this comment.
some cosmetics as a first round
| # - number of processes used | ||
| # the list of files looks then as follows: | ||
| # h1load-dh-rsa-noreuse-1-openssl-3.4.out | ||
| # siege-dh-rsa-noreuse-1-openssl-3.4.out |
There was a problem hiding this comment.
The comment here belongs to merge_siege function. The function is processing the output of siege. I've missed that in my earlier PR so I'm fixing it here.
| The siege client does not build with aws-lc library. To workaround that, | ||
| all tests use siege as http client connecting to HA-proxy, which then | ||
| establishes SSL connection towards httpterm [10] server. To collect performance | ||
| data The siege client executes requests which fetch 1k of data from httpterm | ||
| server. |
There was a problem hiding this comment.
This is a little confusing; it's reads like the h1load is not run, only siege meanwhile above paragraph mentions both
| EMIT_STATS=1 | ||
| for PORT in $(seq ${BASEPORT} ${TOPPORT}) ; do | ||
| emit_https_port ${HAPROXY_CONF} ${PORT} ${PROXYCERT} | ||
| if [[ ${EMIT_STATS} -eq 1 ]] ; then |
| printf "\t${SSL_LIB}" >> ${OUTPUT_FILE} | ||
| done | ||
| printf "\n" >> ${OUTPUT_FILE} | ||
| for PROCS in `procs` ; do |
| INPUT_FILE=${RESULT_DIR}/${HANDSHAKE}-${PROCS}-${SSL_LIB}.out | ||
| # | ||
| # h1load outputs performance data combined with percentile table. Those | ||
| # parts are delimited by ^#= delimiter. The sed expression chops off | ||
| # the first part (performance table). | ||
| # tail -1 then grabs the last line where we find the test duration | ||
| # in secs. The test duration is found in the first column we read | ||
| # using awk | ||
| # | ||
| DURATION=$(sed -ne '/^#=/q;p' "${INPUT_FILE}" |tail -1 |awk '{ printf($1); }') | ||
| printf "\t${DURATION}" >> ${OUTPUT_FILE} |
There was a problem hiding this comment.
indent (the comment is fine)
| # new line | ||
| # | ||
| printf "\n" >> ${OUTPUT_FILE} | ||
| done |
There was a problem hiding this comment.
also seems like some indent stuff here
|
After dropping the chain of 10 proxies and using single SSL proxy instance the benchmark runs for two hours. To speed things up we can remove tests for openssl releases which are no longer supported, this will also help. here is results.single.tgz file with results and charts from gnuplot. |
|
I'm afraid I'm stil not able to figure out way to use h1load test right. The current test defines the load as absolute number of requests/connections to perform and measures the duration for various number of threads (1, 2, 4, 8, 16, 32, 64). This is whet tests shows for
same table for
Another way to run h1load is to limit its duration time and then compare total number of requests. For identical HA-proxy configuration I'm seeing these results. The h1load command reads as follows
It looks like I'm not able to set up things quite right. The machine where I'm trying this is 96 CPU box with 
|
|
@Sashan as you commented above; do you want to fix it in this PR or in a follow-up? |
I think it can be done if folow up PR. May be other people will have better idea to further improve the script. |
nhorman
added
the
approval: ready to merge
The benchmark uses h1load and siege client. To compare SSL libraries using h1load client the client runtime is used. The longer it takes to execute the test the worse performance. The siege client runs for 10 seconds by default attempting to perform as many requests as possible. Reviewed-by: Neil Horman <nhorman@openssl.org> Reviewed-by: Nikola Pajkovsky <nikolap@openssl.org> MergeDate: Fri Feb 13 14:48:07 2026 (Merged from #75)
|
merged to main, thank you! |
github-project-automation
Bot
moved this from Waiting Merge
to Done
in Development Board
|
So I suspect not all versions are actually doing the same thing. Things like the default key exchange got changed. Could things be configured so that they're all actually doing the same thing? |
I was kind of hoping that setting a server certificate/key is sufficient to unify key exchange accross different library versions. The HA-proxy runs with two certificate/key combination: what else should be set/configured to mitigate the effect of different library versions? thanks. |
|
No, the key type only has a limited effect. There are multiple signature schemes, key exchanges and ciphers. As you can see, 3.5 become slower. This is because of this: I'm not sure what the previous default was, but just doing X25519 is probably a fair comparison, and should be supported by all versions. I assume it also transfers data. TLS_AES_128_GCM_SHA256 is probably what you want to compare, while we default to the AES256 version. I have no idea what the other libraries default to. At least for OpenSSL you should be able to set that in the config file. |
thanks for pointing me to right direction. I'll also look at other libraries and check how to configure various parameters for handshake so the test will get results which will eliminate differences introduced by different methods/algorithms. |
5 participants
The benchmark uses h1load and siege client. To compare SSL libraries using h1load client the client runtime is used. The longer it takes to execute the test the worse performance.
The siege client runs for 10 seconds by default attempting to perform as many requests as possible.