openssl: enable readahead and increase read buffer size (improves download speed)

[edwintorok]

I ran strace on curl and noticed that it is doing a lot of very short reads, of exactly 5 bytes, followed by a larger read.

strace Before

recvfrom(4, "\27\3\3@\30", 5, 0, NULL, NULL) = 5
recvfrom(4, ")\232\36\234\354'p\247\257\366\301s\2460\372[kr\235\204\205\224I\313*\226e\35\335\373Z\232"..., 16408, 0, NULL, NULL) = 16408
write(5, "\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0"..., 4096) = 4096
write(5, "\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0"..., 12288) = 12288
recvfrom(4, "\27\3\3@\30", 5, 0, NULL, NULL) = 5
recvfrom(4, ")\232\36\234\354'p\250+\350yN^\276\343\260!\257\262e\300\216\343\251\366\35\1\301)\274\224\341"..., 16408, 0, NULL, NULL) = 16408
write(5, "\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0"..., 4096) = 4096
write(5, "\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0"..., 12288) = 12288

I tweaked the OpenSSL context settings in curl, and now strace looks much nicer: we read ~64KiB in one go
(writing is still done in several smaller writes, this could be improved in the future).

strace After

recvfrom(4, "\3\3@\30/o\342c\260\r\203S-(\247\240\315\261{\223\250\2004Ne+\37\323\263u\233u"..., 65652, 0, NULL, NULL) = 65652
write(5, "\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0"..., 4096) = 4096
write(5, "\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0"..., 12288) = 12288
write(5, "\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0"..., 4096) = 4096
write(5, "\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0"..., 12288) = 12288
write(5, "\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0"..., 4096) = 4096
write(5, "\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0"..., 12288) = 12288
write(5, "\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0"..., 4096) = 4096
write(5, "\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0"..., 12288) = 12288
recvfrom(4, "\3\3@\30/o\342c\260\r\203W\374n(\10:U\332\177u>y\227(\255z\202tzRA"..., 65652, 0, NULL, NULL) = 65652
....

Beyond making strace look "nicer" this change of course improves download speed too.
I measured that download time is reduced by ~13% when downloading a 10GiB file using HTTPS on a 100Gbit/s network link (git hash doesn't match exactly because I amended the commits to include the measurements):

ministat -A curl-8_14_1-11-gba407ee43.dat curl-8_14_1-13-g34606f4ad.dat
x curl-8_14_1-11-gba407ee43.dat
+ curl-8_14_1-13-g34606f4ad.dat
    N           Min           Max        Median           Avg        Stddev
x  30      4.596546       5.69831      5.542359     5.4684991    0.29021028
+  30      4.521651      5.151459     4.6656105     4.7356357    0.19006089
Difference at 95.0% confidence
        -0.732863 +/- 0.1268
        -13.4015% +/- 2.10608%
        (Student's t, pooled s = 0.245301)

The median speed goes from ~15.5Gbit/s to ~18.4 Gbit/s.
For reference iperf3 can do 40Gbit/s (unencrypted obviously) on a single TCP stream (and with 3 streams it can saturate the network link).

The change is tiny: just 2 commits, 1 line each. Although perhaps the buffer size should be configurable (for now I've hardcoded it).

If you want to reproduce the measurements on your own hardware I've included the commands that I've used below, they may need slight tweaks to run on your OS/HW (click to expand the details).

Experimental setup

CPU: 2x 18-core Intel Xeon Gold 6354
NIC: Intel E810-XXV, 100 Gbit/s
Kernel: 6.14.8-300.fc42.x86_64 x86_64
Mem: 251.32 GiB
OS: Fedora Linux 42
openssl version: OpenSSL 3.2.4 11 Feb 2025 (Library: OpenSSL 3.2.4 11 Feb 2025)

Script to put machine into benchmark mode

Following some of the advice from https://llvm.org/docs/Benchmarking.html.
The BIOS was configured to set the System profile to 'Performance' (this exposes P states to the OS and allows turning off turbo):

  # Prevent frequency-scaling
  sudo cpupower frequency-set -g performance

  # Prevent entering deep C-states to reduce latency and avoid menu governor bugs (there was one recently)
  # Although if we completely disable C1 then we are going to hit RAPL power limits immediately,
  # which results in a 30% perf hit on single-stream iperf.
  # This particular machine consumes 258W in C0, but RAPL long term limit is 246W and short term 204W
  # Disable just deeper C states, which avoids the big perf hit (although now we're exposed to menu governor bugs)
  sudo cpupower idle-set -D 2

  # Disable Turbo Boost for more stable measurements
  # cpupower set --turbo-boost 0 doesn't work when the driver is intel_pstate instead of cpufreq
  sudo sh -c 'echo 1 >/sys/devices/system/cpu/intel_pstate/no_turbo'

  # AMD
  sudo cpupower set -m passive
  sudo cpupower set --turbo-boost 0

  sudo cpupower -c all set --perf-bias 0

  # Bring system into a consistent state
  sudo tuned-adm profile network-throughput

  # we could use tuned-adm verify, but there is a bug where it tries to read non-existent hung_task_timeout_secs
  # that only fails on network-latency, not network-throughput though
  sudo tuned-adm verify

  # Stop unneessary services (but not NetworkManager!)
  cat >benchmark.target <<EOF
[Unit]
Description=Minimal Benchmarking mode with SSH/getty
Requires=sysinit.target getty.target network.target sshd.service
After=sysinit.target
AllowIsolate=Yes
EOF
  sudo mv benchmark.target /run/systemd/system/
  sudo systemctl daemon-reload
  sudo systemctl isolate benchmark.target
  sudo systemctl list-units --state=running --no-pager

Script to setup benchmark server

This is part of a larger script that measures also different TLS servers, I've included just the parts needed to measure curl with nginx (since that appears to be the fastest server in my tests so far).

  for P in 9443; do
    sudo firewall-cmd --add-port "${P}/tcp"
  done

# Use a single, fixed cipher for consistent tests across different tools
TLS_CIPHERS=ECDHE-RSA-AES256-GCM-SHA384
TLS_PROTO=TLSv1.2

# Use non-root ports
NGINX_PORT=9080
NGINX_TLS_PORT=9443

REPEAT=${REPEAT-30}
TEST_SIZE=${TEST_SIZE-10GiB}

PEM=stunnel.pem
TESTDIR="${PWD}/tmp"
RAMDIR=/dev/shm/stunnel-test
mkdir -p "${RAMDIR}" "${TESTDIR}"

cd "${TESTDIR}"

test -f "${PEM}" || openssl req -new -subj "/C=GB/CN=foo.example.com" -days 10 -x509 -noenc -batch -out "${PEM}" -keyout  "${PEM}" -noenc -batch

truncate --size "${TEST_SIZE}" "${RAMDIR}/${TEST_SIZE}"

cat >nginx.conf <<EOF
error_log stderr error;
pid ${TESTDIR}/nginx.pid;
daemon on;
events { }
http {
    client_body_temp_path "${TESTDIR}/nginx-client_body";
    proxy_temp_path "${TESTDIR}/nginx-proxy";
    fastcgi_temp_path "${TESTDIR}/nginx-fcgi";
    uwsgi_temp_path ${TESTDIR}/nginx-uwsgi";
    scgi_temp_path ${TESTDIR}/nginx-scgi";
    access_log off;
    tcp_nodelay on;
    sendfile on;
    server {
        gzip off;
        listen "${NGINX_PORT}" reuseport;
        listen "${NGINX_TLS_PORT}" reuseport ssl;
        ssl_certificate "${PEM}";
        ssl_certificate_key "${PEM}";
        ssl_protocols "${TLS_PROTO}";
        ssl_ciphers "${TLS_CIPHERS}";
        root "${RAMDIR}";
        location = /hello {
            default_type text/plain;
            return 200 "Hello";
        }
    }
}

killall -v -9 nginx || :

# Start servers
nginx -c "${TESTDIR}/nginx.conf"
EOF

Script to benchmark a curl change

I measure total time instead of download speed, because the correct way to average speeds would be to use a geometric mean, however ministat uses an arithmetic mean, so using ministat to process download speed measurements wouldn't be correct.

for i in $(seq 30); do src/curl https://10.70.58.43:9443/10GiB -k -o /dev/null -w '%{time_total}\n'; done >$(git describe).dat

I also measured using 8 TLS records instead of 4, but that was actually slower, and even canceled out the improvement in the first commit (so overall ministat said there was no provable difference).
I measured 2 packets instead of 4, but that would've been 5% slower:

x tls4.dat
+ tls2.dat
    N           Min           Max        Median           Avg        Stddev
x  30      4.521651      5.151459     4.6656105     4.7356357    0.19006089
+  30      4.476444      5.228927      5.059509     5.0038106    0.20104942
Difference at 95.0% confidence
        0.268175 +/- 0.101125
        5.66291% +/- 2.19331%
        (Student's t, pooled s = 0.195632)

So 4 packets seems to be the sweet spot, at least on this HW.

P.S.: this change is probably beneficial for most applications using OpenSSL, especially the readahead change, I'll submit similar changes to a few other projects (socat, stunnel, etc.). Ideally I'd submit the change to OpenSSL itself, but for backwards compatibility reasons they probably cannot change the defaults.

[edwintorok]

Looks like I'll need to add some #ifdef to support different OpenSSL versions/alternatives, it is very useful that the CI gave me feedback on that so quickly.
I'll put this PR into draft for now.

[bagder]

This is a nice find!

Looks like I'll need to add some #ifdef to support different OpenSSL versions/alternatives

Annoyingly, the docs for these functions says nothing about when they were added or other conditions as to when they are available. I presume we need to add configure/cmake checks to detect if they are present or not before we can use them.

[edwintorok]

This is a nice find!

Thanks

Annoyingly, the docs for these functions says nothing about when they were added or other conditions as to when they are available. I presume we need to add configure/cmake checks to detect if they are present or not before we can use them.

I'll look at this in more detail next week, but a quick search with git log -S turns up that:

They should both be available in OpenSSL 1.1.0:
openssl/openssl@dad78fb

Since LibreSSL is a fork of OpenSSL 1.0.x it won't have it. So perhaps some OpenSSL >= 1.1.0 could be used (unfortunately LibreSSL claims to be version 2.0 in OPENSSL_VERSION_NUMBER), but I'll figure an appropriate macro to use for detection.

[levitte]

Out of curiosity, why 0x401e? The comment above would make me think 0x4800

[edwintorok]

It is up to 0x4000 + 0x800, but can be smaller. I'll investigate with tcpdump/tshark/wireshark what a typical TLS record size is (might also depend on protocol version)

[edwintorok]

tshark says:

 Reassembled TCP length: 16413
    TLSv1.2 Record Layer: Application Data Protocol: Hypertext Transfer Protocol
        Content Type: Application Data (23)
        Version: TLS 1.2 (0x0303)
        Length: 16408

16413 is 0x401d. I'll try to find out whether that is more or less constant, or whether different TLS versions or settings could change that (well obviously according to the comment it can go up to +0x800, but it talks about compression, which is usually turned off due to security issues)

[icing]

@edwintorok nice one. Did not know of that OpenSSL feature. I had experimented with adding buffering to the underlying socket filter. That had no real impact because then it caused an additional buffer copy of the incoming data to OpenSSL. Reading directly larger chunks into OpenSSL's packet handling is much better.

I ran our scorecard against a local Caddy server, using HTTP/1.1 downloads, via:

> python3 tests/http/scorecard.py -d --samples 10 --caddy h1

with a non-debug curl build and varying the OpenSSL buffer sizes on my macOS box:

buffer size    serial(50) [cpu/rss]      parallel(50x50) [cpu/rss]
off   100MB  1365 MB/s [98.5%/10MB]        1592 MB/s [100.0%/22MB]
1     100MB  1346 MB/s [97.6%/10MB]        1746 MB/s [100.0%/22MB]
2     100MB  1380 MB/s [98.5%/10MB]        1748 MB/s [100.0%/23MB]
4     100MB  1407 MB/s [99.0%/10MB]        1740 MB/s [100.0%/25MB]
8     100MB  1238 MB/s [99.3%/10MB]        1483 MB/s [100.0%/28MB]

buffer size = n * 0x401E

Which transfers 50 times a 100MB resource, first over the same connection one after the other. Second, over 50 connections. The effect on the serial case is minimal, on the parallel one noticeable. The serial case "loses" time between end of one and start of next download after processing the request, I believe.

In the parallel case, curl is at maxed CPU and the performance benefits of the larger reads shine. The benefit comes with a 16KB buffer already with no real gain thereafter. The case of 8 * 16KB shows that it becomes counter-productive somewhere after 64KB, as you had also noticed.

Increasing the resource size does not alter the picture:

> python3 tests/http/scorecard.py -d --samples 10 --download-sizes=1gb --download-count=10 --caddy h1

buffer size    serial(10) [cpu/rss]      parallel(10x10) [cpu/rss]
off     1GB  1390 MB/s [98.8%/10MB]        1764 MB/s [100.0%/12MB]
1       1GB  1369 MB/s [97.7%/10MB]        1958 MB/s [100.0%/12MB]
2       1GB  1393 MB/s [98.8%/10MB]        1972 MB/s [100.0%/12MB]
4       1GB  1426 MB/s [99.4%/10MB]        1988 MB/s [100.0%/12MB]
8       1GB  1240 MB/s [99.7%/10MB]        1837 MB/s [100.0%/13MB]

Compatibility: there are faults of properly handling buffered TLS data visible in test failures. I have seen one IMAPS test failing for me. The trap is that a protocol waits on socket events, but the data it is expecting is already present in the OpenSSL buffers and just needs to be read. That stalls curl.

This is not a fault of this change, but a bug in the protocol handling exposed.

[bagder]

So maybe something like this?

#if (OPENSSL_VERSION_NUMBER >= 0x10100000L && \
     !defined(LIBRESSL_VERSION_NUMBER)
  #define HAVE_SSL_CTX_SET_READ_AHEAD 1
  #define HAVE_SSL_CTX_SET_DEFAULT_READ_BUFFER_LEN 1
#endif

[edwintorok]

So maybe something like this?

#if (OPENSSL_VERSION_NUMBER >= 0x10100000L && \
     !defined(LIBRESSL_VERSION_NUMBER)
  #define HAVE_SSL_CTX_SET_READ_AHEAD 1
  #define HAVE_SSL_CTX_SET_DEFAULT_READ_BUFFER_LEN 1
#endif

Thanks, I pushed a variant of this in a separate commit (with the missing ) added).
SSL_CTX_set_read_ahead was actually part of OpenSSL 0.9.2b, so we probably won't need that part.

Looks like there is a failure on BoringSSL too, I've expanded the macro. Their PORTING.md says to contact them about missing functions, they may add them, so I've done that: https://issues.chromium.org/issues/423460787

[edwintorok]

In the parallel case, curl is at maxed CPU and the performance benefits of the larger reads shine. The benefit comes with a 16KB buffer already with no real gain thereafter. The case of 8 * 16KB shows that it becomes counter-productive somewhere after 64KB, as you had also noticed.

Thanks, so we know that >4 packets is bad not just on my HW. The ideal multiple between 1 and 4 TBD, I'll have to do some testing on more HW (I also have access to HW on 10Gbit/s and 25Gbit/s links, so I can measure those too)

Compatibility: there are faults of properly handling buffered TLS data visible in test failures. I have seen one IMAPS test failing for me. The trap is that a protocol waits on socket events, but the data it is expecting is already present in the OpenSSL buffers and just needs to be read. That stalls curl.

This is not a fault of this change, but a bug in the protocol handling exposed.

Ah, latent bugs... Not quite so simple to enable this as I thought then :(.
Perhaps this could be initially enabled only for protocols where testing shows no faults (e.g. HTTPS to begin with), and then the bugs in the other protocols could be fixed before being enabled there?

Looking at the latest CI run I see these test failures, so probably FTP too, and something wrong with PROXY:

FAIL 409: 'FTPS PASV upload file' FTP, FTPS, EPSV, STOR

TESTFAIL: These test cases failed: 409

test 3201...[HTTP GET when PROXY Protocol enabled and spoofed client IP]
--p----e--- 
Error: The operation was canceled.

[bagder]

The maximum TLS record size is actually 16KB (in the protocol layer), so 64KB would make sure it can at least get 4 maximum-sized records. If that is what the buffer is used for.

Is there anything more to do here (it is still marked draft) or should we merge?

[bagder]

@icing found SSL_CTX_set_default_read_buffer_len() in the AWS-LC source code. So maybe this needs some additional tweaks - is it really confirmed to not exist in boring?

[icing]

I don't thing AWS-LC defines OPENSSL_IS_BORINGSSL. It defines OPENSSL_IS_AWSLC.

[bagder]

I'm wrong. AWS-LC uses OPENSSL_IS_AWSLC and thus it will be used there.

[icing]

My numbers with AWS-LC today on the current PR:

> python3 tests/http/scorecard.py -d --samples 10 --download-sizes=1gb --download-count=10 --caddy h1
     size  serial(10) [cpu/rss]        parallel(10x10) [cpu/rss]
off   1GB 1394 MB/s [99.2%/7MB]          1572 MB/s [100.0%/8MB]
4     1GB 1428 MB/s [99.5%/9MB]          1922 MB/s [100.0%/15MB]

Nice. All for merging this.

[bagder]

Thanks!

[edwintorok]

Is there anything more to do here (it is still marked draft) or should we merge?

I think the only outstanding action was to investigate these latent bugs/failures from the CI on certain protocols:

Compatibility: there are faults of properly handling buffered TLS data visible in test failures. I have seen one IMAPS test failing for me. The trap is that a protocol waits on socket events, but the data it is expecting is already present in the OpenSSL buffers and just needs to be read. That stalls curl.
This is not a fault of this change, but a bug in the protocol handling exposed.

Ah, latent bugs... Not quite so simple to enable this as I thought then :(. Perhaps this could be initially enabled only for protocols where testing shows no faults (e.g. HTTPS to begin with), and then the bugs in the other protocols could be fixed before being enabled there?

Sorry I haven't had time to look into these yet, I'll try to find some time later this week. (I see you've merged the PR already, that is fine, just something to keep in mind before making the next release of curl).

Thanks for all the feedback on my changes.