High CPU usage with -T .

[magisterquis]

I did this

./long_running_command | curl -T. https://my_server

The idea is to have output from the command sent to my server as it's produced.

Reproduces with

curl -T. https://example.com

I expected the following

Output to show up server-side, which it did.

Client-side, the curl's CPU usage to be nearly 0 most of the time, which it wasn't. After a while, it hits nearly 100%.

It seems to be hitting a tight loop which calls poll, read, and recvfrom:

 86911 curl     CALL  poll(0x7d6f6c2b2c80,2,0)
 86911 curl     STRU  struct pollfd [2] { fd=5, events=0x5<POLLIN|POLLOUT>, revents=0x4<POLLOUT> } { fd=3, events=0x1<POLLIN>, revents=0<> }
 86911 curl     RET   poll 1
 86911 curl     CALL  sigaction(SIGPIPE,0,0x7d6f6c2b2d70)
 86911 curl     STRU  struct sigaction { handler=SIG_IGN, mask=0<>, flags=0x2<SA_RESTART> }
 86911 curl     RET   sigaction 0
 86911 curl     CALL  sigaction(SIGPIPE,0x7d6f6c2b2d40,0)
 86911 curl     STRU  struct sigaction { handler=SIG_IGN, mask=0<>, flags=0x2<SA_RESTART> }
 86911 curl     RET   sigaction 0
 86911 curl     CALL  recvfrom(5,0x61af75375c0,0x5,0,0,0)
 86911 curl     RET   recvfrom -1 errno 35 Resource temporarily unavailable
 86911 curl     CALL  read(0,0x61af1ca4020,0x10000)
 86911 curl     RET   read -1 errno 35 Resource temporarily unavailable
 86911 curl     CALL  sigaction(SIGPIPE,0x7d6f6c2b2d70,0)
 86911 curl     STRU  struct sigaction { handler=SIG_IGN, mask=0<>, flags=0x2<SA_RESTART> }
 86911 curl     RET   sigaction 0
 86911 curl     CALL  poll(0x7d6f6c2b2c80,2,0)
 86911 curl     STRU  struct pollfd [2] { fd=5, events=0x5<POLLIN|POLLOUT>, revents=0x4<POLLOUT> } { fd=3, events=0x1<POLLIN>, revents=0<> }
 86911 curl     RET   poll 1

File descriptor 5 is the network socket:

$ fstat -p 86911
USER     CMD          PID   FD MOUNT        INUM  MODE         R/W    SZ|DV
stuart   curl       86911 text /        109747340  -rwxr-xr-x     r   321296
stuart   curl       86911   wd /        109670073  drwxr-xr-x     r     1536
stuart   curl       86911   tr /        109670150  -rw-------    rw 335611000
stuart   curl       86911    0 /        104276665  crw--w----    rw    ttyp5
stuart   curl       86911    1 /        104277329  crw-rw-rw-     w     null
stuart   curl       86911    2 /        104277329  crw-rw-rw-     w     null
stuart   curl       86911    3 pipe 0x0 state:
stuart   curl       86911    4 pipe 0x0 state:
stuart   curl       86911    5* internet stream tcp 0x0 [Internal_IP]:24787 --> 93.184.216.34:443

It's more or less the same on Linux, according to strace.

curl/libcurl version

curl 8.7.0-DEV (x86_64-unknown-openbsd7.4) libcurl/8.7.0-DEV LibreSSL/3.8.2 zlib/1.3 libidn2/2.3.0 nghttp2/1.57.0
Release-Date: [unreleased]
Protocols: dict file ftp ftps gopher gophers http https imap imaps ipfs ipns mqtt pop3 pop3s rtsp smb smbs smtp smtps telnet tftp
Features: alt-svc AsynchDNS HSTS HTTP2 HTTPS-proxy IDN IPv6 Largefile libz NTLM SSL threadsafe UnixSockets

As well as

curl 8.5.0 (x86_64-unknown-openbsd7.4) libcurl/8.5.0 LibreSSL/3.8.2 zlib/1.3 nghttp2/1.57.0 ngtcp2/0.19.1 nghttp3/0.15.0
Release-Date: 2023-12-06
Protocols: dict file ftp ftps gopher gophers http https imap imaps mqtt pop3 pop3s rtsp smb smbs smtp smtps telnet tftp
Features: alt-svc AsynchDNS HSTS HTTP2 HTTP3 HTTPS-proxy IPv6 Largefile libz NTLM NTLM_WB SSL threadsafe UnixSockets

curl 7.88.1 (aarch64-unknown-linux-gnu) libcurl/7.88.1 OpenSSL/3.0.11 zlib/1.2.13 brotli/1.0.9 zstd/1.5.4 libidn2/2.3.3 libpsl/0.21.2 (+libidn2/2.3.3) libssh2/1.10.0 nghttp2/1.52.0 librtmp/2.3 OpenLDAP/2.5.13
Release-Date: 2023-02-20, security patched: 7.88.1-10+deb12u5
Protocols: dict file ftp ftps gopher gophers http https imap imaps ldap ldaps mqtt pop3 pop3s rtmp rtsp scp sftp smb smbs smtp smtps telnet tftp
Features: alt-svc AsynchDNS brotli GSS-API HSTS HTTP2 HTTPS-proxy IDN IPv6 Kerberos Largefile libz NTLM NTLM_WB PSL SPNEGO SSL threadsafe TLS-SRP UnixSockets zstd

curl 8.6.0 (aarch64-unknown-openbsd7.4) libcurl/8.6.0 LibreSSL/3.8.2 zlib/1.3 nghttp2/1.57.0 ngtcp2/0.19.1 nghttp3/0.15.0
Release-Date: 2024-01-31
Protocols: dict file ftp ftps gopher gophers http https imap imaps ipfs ipns mqtt pop3 pop3s rtsp smb smbs smtp smtps telnet tftp
Features: alt-svc AsynchDNS HSTS HTTP2 HTTP3 HTTPS-proxy IPv6 Largefile libz NTLM NTLM_WB SSL threadsafe UnixSockets

curl 8.4.0 (x86_64-apple-darwin23.0) libcurl/8.4.0 (SecureTransport) LibreSSL/3.3.6 zlib/1.2.12 nghttp2/1.55.1
Release-Date: 2023-10-11
Protocols: dict file ftp ftps gopher gophers http https imap imaps ldap ldaps mqtt pop3 pop3s rtsp smb smbs smtp smtps telnet tftp
Features: alt-svc AsynchDNS GSS-API HSTS HTTP2 HTTPS-proxy IPv6 Kerberos Largefile libz MultiSSL NTLM NTLM_WB SPNEGO SSL threadsafe UnixSockets

operating system

OpenBSD [hostname] 7.4 GENERIC.MP#2 amd64

Linux [hostname] 6.1.0-18-arm64 #1 SMP Debian 6.1.76-1 (2024-02-01) aarch64 GNU/Linux

OpenBSD [hostname] 7.4 GENERIC.MP#2 arm64

Darwin [hostname] 23.2.0 Darwin Kernel Version 23.2.0: Wed Nov 15 21:53:18 PST 2023; root:xnu-10002.61.3~2/RELEASE_ARM64_T6000 arm64

[rsbeckerca]

The poll call is specifying a 0 timeout which means return immediately. A negative value will wait for data, so this looks like a deliberately tight loop polling for data as fast as the processor can check the IP stack for data. If there is no data, but the FDs are open, the loop will consume as many cycles as available.

[icing]

@rsbeckerca thanks for the report. I can reproduce locally that indeed curl cycles through read/recv attempts indefinitely without pausing.

[icing]

After analysis, we would need an addition to the libcurl API, basically a new option, that allows to unpause transfer sending when a socket becomes readable. Sine this is not a "few lines" fix, it might take a while.

[bagder]

I propose a work-around in #13506 that should ideally at least reduce the busy-looping.