curl consumes 100% CPU when sending a file with -F #11242

l29ah · 2023-06-03T00:04:59Z

I did this

curl -Ffile=@"VID_20230603_010956.mp4" https://0x0.st

I expected the following

Tiny CPU consumption appropriate for the pathetic ~200kB/s transfer speed on my i7 CPU.

curl/libcurl version

curl 8.1.2 (x86_64-pc-linux-gnu) libcurl/8.1.2 GnuTLS/3.8.0 (OpenSSL/1.1.1u) zlib/1.2.13 c-ares/1.19.1 libidn2/2.3.4 nghttp2/1.52.0 librtmp/2.3
Release-Date: 2023-05-30
Protocols: dict file ftp ftps http https imap imaps mqtt pop3 pop3s rtmp rtsp smtp smtps tftp
Features: AsynchDNS HTTP2 HTTPS-proxy IDN IPv6 Largefile libz MultiSSL NTLM SSL threadsafe TLS-SRP UnixSockets

operating system

Gentoo Lignux
Linux l29ah-x201 6.2.5+ #238 SMP PREEMPT_DYNAMIC Mon Mar 13 02:40:28 CET 2023 x86_64 Intel(R) Core(TM) i7-8550U CPU @ 1.80GHz GenuineIntel GNU/Linux

top of perf report

# Children      Self  Command  Shared Object          Symbol                                          
# ........  ........  .......  .....................  ................................................
#
    27.96%     0.00%  curl     [unknown]              [.] 0x0000000000020003
            |
            ---0x20003
               |          
               |--4.68%--pthread_mutex_lock@@GLIBC_2.2.5
               |          
               |--3.98%--_gnutls_io_write_flush
               |          
               |--3.81%--_gnutls_record_overhead
               |          
               |--3.61%--_gnutls_epoch_get
               |          
               |--3.18%--record_overhead_rt
               |          
               |--2.50%--__GI___pthread_mutex_unlock_usercnt
               |          
               |--1.47%--gnutls_system_mutex_lock
               |          
               |--1.35%--_mbuffer_head_get_first
               |          
               |--1.06%--gnutls_system_mutex_unlock
               |          
                --0.91%--_gnutls_writev_emu.constprop.0

    12.93%     5.91%  curl     libc.so.6              [.] __send
            |          
            |--7.02%--__send
            |          
            |--0.60%--0x3c668934bc4645c5
            |          0x6a8
            |          |          
            |           --0.60%--__send
            |          
            |--0.53%--0x83df1832731fb4aa
            |          0x3e52
            |          |          
            |           --0.52%--__send
            |          
             --0.50%--0x7734ab5ea3b468f0
                       0x35ac
                       |          
                        --0.50%--__send

    10.99%     0.00%  curl     [unknown]              [.] 0000000000000000
            |
            ---0
               |          
                --10.99%--0x5575fc882990
                          |          
                          |--5.02%--nghttp2_session_send
                          |          
                          |--1.86%--send_callback
                          |          
                          |--1.15%--nghttp2_session_mem_send_internal
                          |          
                          |--1.14%--Curl_bufq_write_pass
                          |          
                          |--1.07%--nghttp2_session_adjust_idle_stream
                          |          
                           --0.58%--nghttp2_is_fatal

The text was updated successfully, but these errors were encountered:

bagder · 2023-06-03T18:58:14Z

Does this by any chance use HTTP/2 when uploading? (if you add -v you'll see it mentioned)

Oops. sorry yes it does. The nghttp2_* symbols indicate that.

bagder · 2023-06-03T18:59:14Z

@icing when you have a minute over, can you have a look at this?

- related to curl#11242 where curl enters busy loop when sending http2 data to the server

icing · 2023-06-04T10:46:19Z

@l29ah would you be able to try #11247 in your setup? I made the code that sends the http2 data up more robust in handling EAGAIN situations.

l29ah · 2023-06-04T11:00:21Z

On Sun, Jun 04, 2023 at 03:46:30AM -0700, Stefan Eissing wrote: @l29ah would you be able to try #11247 in your setup? I made the code that sends the http2 data up more robust in handling EAGAIN situations.

Tried slapping it on top of 8.1.2, still 100% CPU.

…

-- () ascii ribbon campaign - against html mail /\ http://arc.pasp.de/ - against proprietary attachments

icing · 2023-06-04T12:07:47Z

Could you make a build with --enable-debug for testing and run curl with

CURL_DEBUG=http/2 curl -v [your args]

Thank you!

- related to #11242 where curl enters busy loop when sending http2 data to the server Closes #11247

bagder · 2023-06-15T13:40:55Z

I cannot reproduce. I tried sending a fairly large file to that same server using this curl version:

curl 8.2.0-DEV (x86_64-pc-linux-gnu) libcurl/8.2.0-DEV GnuTLS/3.7.9 zlib/1.2.13
brotli/1.0.9 zstd/1.5.4 libidn2/2.3.3 libpsl/0.21.2 (+libidn2/2.3.3)
nghttp2/1.54.0 librtmp/2.3 libgsasl/2.2.0

l29ah · 2023-06-16T10:07:39Z

CURL_DEBUG=http/2

https://tinystash.undef.im/il/2EQtfhvYcr9r9SPd1TkCyqudqFDNKSc9yaCHNAw9t4gsEdd9gKLaQUKAt2pBMiTgKow7kDcA2DhETsB8Q2ggAWsw
Grabbed a chunk of /dev/urandom and killed curl after a few seconds of uploading.

icing · 2023-06-16T10:47:17Z

In the log I see that the upload is progressing. So we have no stall or busy loop here, "just" the action of chunking your upload file into the 16KB frame data of the HTTP/2 protocol, encrypting those and passing them to the network.

I, so far, see nothing wrong, except that the implementation is not as efficient as you would like it to be.

l29ah · 2023-06-16T13:25:08Z

Downgraded curl to 8.0.1, now it consumes <1% CPU on the same scenario.

icing · 2023-06-16T14:01:06Z

Reproduced. You convinced me that this does not look right. Analyzing what I did here...

- refs curl#11242 where abnormal CPU on uploads wa reported - this is mainly due to the drain dselect_bits being set during uploads where they should not. This was continually EXPIRing the transfer - BUT toying with large uplaods on my connection against the server in the issue revealed a deeper problem with buffer handling that leads to failed uploads Outline: - when curl is faster than the TCP connection, we get EAGAIN in various stages of uploads - Then, data is left pending in buffers a) the stream->sendbuf b) the filters ctx->outbufq c) nghttp2 itself - Problem a) I addressed in this branch - Problem b) could be solved, however - Problem c) is the real issue nghttp2 hold internally the DATA frame it has collected from out callbacks, but it is not able to write it to the network due to EAGAIN. It try to write it on next opportunity. Fine. But when is that opportunity? Well, if an upload EAGAINs on the very last DATA frame, transfer.c declares it is done and never calls send again. The transfer times out and is reset by the server.

- refs curl#11242 where CPU busy loop is reported. Removed drain setting that caused this - discovered stalled uploads using the server from curl#11242 on my slow internet - added workaround on flushing the last upload chunk that, if not written fully, makes the transfer stall. This is not the fix we are looking for.

icing · 2023-06-17T15:44:27Z

Ok, if your find the time, you could give #11342 a try on your setup. This is a fix for the CPU and for stalled uploads that I observed here. Likelihood of those depend on network conditions. We are discussing how best to resolve that one.

But it would be good to know if this PR makes the upload work flawlessly for you.

updated to the new PR

- refs curl#11242 where abnormal CPU on uploads wa reported - this is mainly due to the drain dselect_bits being set during uploads where they should not. This was continually EXPIRing the transfer - BUT toying with large uplaods on my connection against the server in the issue revealed a deeper problem with buffer handling that leads to failed uploads Outline: - when curl is faster than the TCP connection, we get EAGAIN in various stages of uploads - Then, data is left pending in buffers a) the stream->sendbuf b) the filters ctx->outbufq c) nghttp2 itself - Problem a) I addressed in this branch - Problem b) could be solved, however - Problem c) is the real issue nghttp2 hold internally the DATA frame it has collected from out callbacks, but it is not able to write it to the network due to EAGAIN. It try to write it on next opportunity. Fine. But when is that opportunity? Well, if an upload EAGAINs on the very last DATA frame, transfer.c declares it is done and never calls send again. The transfer times out and is reset by the server.

- refs curl#11242 where CPU busy loop is reported. Removed drain setting that caused this - discovered stalled uploads using the server from curl#11242 on my slow internet - added workaround on flushing the last upload chunk that, if not written fully, makes the transfer stall. This is not the fix we are looking for.

- fixes curl#11242 where 100% CPU on uploads was reported - fixes possible stalls on last part of a request body when that information could not be fully send on the connection due to an EAGAIN - applies the same EGAIN handling to HTTP/2 proxying

- related to curl#11242 where curl enters busy loop when sending http2 data to the server Closes curl#11247

- fixes curl#11242 where 100% CPU on uploads was reported - fixes possible stalls on last part of a request body when that information could not be fully send on the connection due to an EAGAIN - applies the same EGAIN handling to HTTP/2 proxying Reported-by: Sergey Alirzaev Fixed curl#11242 Closes curl#11342

- related to curl#11242 where curl enters busy loop when sending http2 data to the server Closes curl#11247

- fixes curl#11242 where 100% CPU on uploads was reported - fixes possible stalls on last part of a request body when that information could not be fully send on the connection due to an EAGAIN - applies the same EGAIN handling to HTTP/2 proxying Reported-by: Sergey Alirzaev Fixed curl#11242 Closes curl#11342

bagder added the HTTP/2 label Jun 3, 2023

icing mentioned this issue Jun 4, 2023

cURL 8.1.0 POST fails, downgrading to 8.0.1 works as expected #11157

Closed

icing added a commit to icing/curl that referenced this issue Jun 4, 2023

bufq: make write/pass methods more robust

591e1db

- related to curl#11242 where curl enters busy loop when sending http2 data to the server

icing mentioned this issue Jun 4, 2023

bufq: make write/pass methods more robust #11247

Closed

bagder pushed a commit that referenced this issue Jun 5, 2023

bufq: make write/pass methods more robust

73022b5

- related to #11242 where curl enters busy loop when sending http2 data to the server Closes #11247

bagder added the needs-info label Jun 15, 2023

icing mentioned this issue Jun 17, 2023

http/2, workaround for stalled uploads #11330

Closed

icing mentioned this issue Jun 19, 2023

HTTP/2 and HTTP/2 upload handling fixes #11342

Closed

bagder closed this as completed in 65937f0 Jun 20, 2023

bch pushed a commit to bch/curl that referenced this issue Jul 19, 2023

bufq: make write/pass methods more robust

445875c

- related to curl#11242 where curl enters busy loop when sending http2 data to the server Closes curl#11247

ptitSeb pushed a commit to wasix-org/curl that referenced this issue Sep 25, 2023

bufq: make write/pass methods more robust

58565fb

- related to curl#11242 where curl enters busy loop when sending http2 data to the server Closes curl#11247

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

curl consumes 100% CPU when sending a file with -F #11242

curl consumes 100% CPU when sending a file with -F #11242

l29ah commented Jun 3, 2023

bagder commented Jun 3, 2023 •

edited

Loading

bagder commented Jun 3, 2023

icing commented Jun 4, 2023

l29ah commented Jun 4, 2023 via email

icing commented Jun 4, 2023

bagder commented Jun 15, 2023

l29ah commented Jun 16, 2023 •

edited

Loading

icing commented Jun 16, 2023

l29ah commented Jun 16, 2023

icing commented Jun 16, 2023

icing commented Jun 17, 2023 •

edited

Loading

curl consumes 100% CPU when sending a file with -F #11242

curl consumes 100% CPU when sending a file with -F #11242

Comments

l29ah commented Jun 3, 2023

I did this

I expected the following

curl/libcurl version

operating system

top of perf report

bagder commented Jun 3, 2023 • edited Loading

bagder commented Jun 3, 2023

icing commented Jun 4, 2023

l29ah commented Jun 4, 2023 via email

icing commented Jun 4, 2023

bagder commented Jun 15, 2023

l29ah commented Jun 16, 2023 • edited Loading

icing commented Jun 16, 2023

l29ah commented Jun 16, 2023

icing commented Jun 16, 2023

icing commented Jun 17, 2023 • edited Loading

bagder commented Jun 3, 2023 •

edited

Loading

l29ah commented Jun 16, 2023 •

edited

Loading

icing commented Jun 17, 2023 •

edited

Loading