Unable to upload file with curl when --limit-rate is provided #12559

Karthikdasari0423 · 2023-12-20T07:54:51Z

I did this

curl -v -k -# --http2 --limit-rate 1M https://127.0.0.1:8443/upload/BPS.pdf -T /upload/BPS.pdf

I expected the following

root@ubuntu:~# curl -v -k -# --http2 https://127.0.0.1:8443/upload/BPS.pdf -T /upload/BPS.pdf
*   Trying 127.0.0.1:8443...
* Connected to 127.0.0.1 (127.0.0.1) port 8443
* ALPN: curl offers h2,http/1.1
* TLSv1.3 (OUT), TLS handshake, Client hello (1):
* TLSv1.3 (IN), TLS handshake, Server hello (2):
* TLSv1.3 (IN), TLS handshake, Encrypted Extensions (8):
* TLSv1.3 (IN), TLS handshake, Certificate (11):
* TLSv1.3 (IN), TLS handshake, CERT verify (15):
* TLSv1.3 (IN), TLS handshake, Finished (20):
* TLSv1.3 (OUT), TLS change cipher, Change cipher spec (1):
* TLSv1.3 (OUT), TLS handshake, Finished (20):
* SSL connection using TLSv1.3 / TLS_AES_256_GCM_SHA384 / X25519 / RSASSA-PSS
* ALPN: server accepted h2
* Server certificate:
*  subject: CN=ubuntu
*  start date: Feb 10 15:24:51 2023 GMT
*  expire date: Feb  7 15:24:51 2033 GMT
*  issuer: CN=ubuntu
*  SSL certificate verify result: self-signed certificate (18), continuing anyway.
*   Certificate level 0: Public key type RSA (2048/112 Bits/secBits), signed using sha256WithRSAEncryption
* TLSv1.3 (IN), TLS handshake, Newsession Ticket (4):
* TLSv1.3 (IN), TLS handshake, Newsession Ticket (4):
* old SSL session ID is stale, removing
* using HTTP/2
* [HTTP/2] [1] OPENED stream for https://127.0.0.1:8443/upload/BPS.pdf
* [HTTP/2] [1] [:method: PUT]
* [HTTP/2] [1] [:scheme: https]
* [HTTP/2] [1] [:authority: 127.0.0.1:8443]
* [HTTP/2] [1] [:path: /upload/BPS.pdf]
* [HTTP/2] [1] [user-agent: curl/8.5.1-DEV]
* [HTTP/2] [1] [accept: */*]
* [HTTP/2] [1] [content-length: 30182355]
> PUT /upload/BPS.pdf HTTP/2
> Host: 127.0.0.1:8443
> User-Agent: curl/8.5.1-DEV
> Accept: */*
> Content-Length: 30182355
>
* We are completely uploaded and fine
< HTTP/2 201
< server: nginx/1.25.3
< date: Wed, 20 Dec 2023 07:53:05 GMT
< content-length: 0
< location: https://127.0.0.1:8443/upload/BPS.pdf
<
* Connection #0 to host 127.0.0.1 left intact
root@ubuntu:~#
root@ubuntu:~#

curl/libcurl version

root@ubuntu:~# curl -V
curl 8.5.1-DEV (x86_64-pc-linux-gnu) libcurl/8.5.1-DEV quictls/3.1.4 zlib/1.2.11 brotli/1.0.9 nghttp2/1.59.0-DEV ngtcp2/1.1.0 nghttp3/1.1.0
Release-Date: [unreleased]
Protocols: dict file ftp ftps gopher gophers http https imap imaps ipfs ipns mqtt pop3 pop3s rtsp smb smbs smtp smtps telnet tftp
Features: alt-svc AsynchDNS brotli HSTS HTTP2 HTTP3 HTTPS-proxy IPv6 Largefile libz NTLM SSL threadsafe TLS-SRP UnixSockets
root@ubuntu:~#

operating system

root@ubuntu:~# uname -a
Linux ubuntu 5.15.0-60-generic #66-Ubuntu SMP Fri Jan 20 14:29:49 UTC 2023 x86_64 x86_64 x86_64 GNU/Linux
root@ubuntu:~#

The text was updated successfully, but these errors were encountered:

Karthikdasari0423 · 2023-12-20T08:00:34Z

when i remove --limit-rate 1M and run curl command, upload is fine but when i provide --limit-rate 1M i am getting below error

>
* HTTP/2 stream 1 was not closed cleanly: PROTOCOL_ERROR (err 1)
* Connection #0 to host 127.0.0.1 left intact
curl: (92) HTTP/2 stream 1 was not closed cleanly: PROTOCOL_ERROR (err 1)

Not provided --limit-rate 1M with curl

root@ubuntu:~# curl -v -k -# --http2 https://127.0.0.1:8443/upload/BPS.pdf -T /upload/BPS.pdf
*   Trying 127.0.0.1:8443...
* Connected to 127.0.0.1 (127.0.0.1) port 8443
* ALPN: curl offers h2,http/1.1
* TLSv1.3 (OUT), TLS handshake, Client hello (1):
* TLSv1.3 (IN), TLS handshake, Server hello (2):
* TLSv1.3 (IN), TLS handshake, Encrypted Extensions (8):
* TLSv1.3 (IN), TLS handshake, Certificate (11):
* TLSv1.3 (IN), TLS handshake, CERT verify (15):
* TLSv1.3 (IN), TLS handshake, Finished (20):
* TLSv1.3 (OUT), TLS change cipher, Change cipher spec (1):
* TLSv1.3 (OUT), TLS handshake, Finished (20):
* SSL connection using TLSv1.3 / TLS_AES_256_GCM_SHA384 / X25519 / RSASSA-PSS
* ALPN: server accepted h2
* Server certificate:
*  subject: CN=ubuntu
*  start date: Feb 10 15:24:51 2023 GMT
*  expire date: Feb  7 15:24:51 2033 GMT
*  issuer: CN=ubuntu
*  SSL certificate verify result: self-signed certificate (18), continuing anyway.
*   Certificate level 0: Public key type RSA (2048/112 Bits/secBits), signed using sha256WithRSAEncryption
* TLSv1.3 (IN), TLS handshake, Newsession Ticket (4):
* TLSv1.3 (IN), TLS handshake, Newsession Ticket (4):
* old SSL session ID is stale, removing
* using HTTP/2
* [HTTP/2] [1] OPENED stream for https://127.0.0.1:8443/upload/BPS.pdf
* [HTTP/2] [1] [:method: PUT]
* [HTTP/2] [1] [:scheme: https]
* [HTTP/2] [1] [:authority: 127.0.0.1:8443]
* [HTTP/2] [1] [:path: /upload/BPS.pdf]
* [HTTP/2] [1] [user-agent: curl/8.5.1-DEV]
* [HTTP/2] [1] [accept: */*]
* [HTTP/2] [1] [content-length: 30182355]
> PUT /upload/BPS.pdf HTTP/2
> Host: 127.0.0.1:8443
> User-Agent: curl/8.5.1-DEV
> Accept: */*
> Content-Length: 30182355
>
* We are completely uploaded and fine
< HTTP/2 201
< server: nginx/1.25.3
< date: Wed, 20 Dec 2023 07:56:52 GMT
< content-length: 0
< location: https://127.0.0.1:8443/upload/BPS.pdf
<
* Connection #0 to host 127.0.0.1 left intact
root@ubuntu:~#

Provided --limit-rate 1M with curl

root@ubuntu:~# curl -v -k -# --http2 --limit-rate 1M https://127.0.0.1:8443/upload/BPS.pdf -T /upload/BPS.pdf
*   Trying 127.0.0.1:8443...
* Connected to 127.0.0.1 (127.0.0.1) port 8443
* ALPN: curl offers h2,http/1.1
* TLSv1.3 (OUT), TLS handshake, Client hello (1):
* TLSv1.3 (IN), TLS handshake, Server hello (2):
* TLSv1.3 (IN), TLS handshake, Encrypted Extensions (8):
* TLSv1.3 (IN), TLS handshake, Certificate (11):
* TLSv1.3 (IN), TLS handshake, CERT verify (15):
* TLSv1.3 (IN), TLS handshake, Finished (20):
* TLSv1.3 (OUT), TLS change cipher, Change cipher spec (1):
* TLSv1.3 (OUT), TLS handshake, Finished (20):
* SSL connection using TLSv1.3 / TLS_AES_256_GCM_SHA384 / X25519 / RSASSA-PSS
* ALPN: server accepted h2
* Server certificate:
*  subject: CN=ubuntu
*  start date: Feb 10 15:24:51 2023 GMT
*  expire date: Feb  7 15:24:51 2033 GMT
*  issuer: CN=ubuntu
*  SSL certificate verify result: self-signed certificate (18), continuing anyway.
*   Certificate level 0: Public key type RSA (2048/112 Bits/secBits), signed using sha256WithRSAEncryption
* TLSv1.3 (IN), TLS handshake, Newsession Ticket (4):
* TLSv1.3 (IN), TLS handshake, Newsession Ticket (4):
* old SSL session ID is stale, removing
* using HTTP/2
* [HTTP/2] [1] OPENED stream for https://127.0.0.1:8443/upload/BPS.pdf
* [HTTP/2] [1] [:method: PUT]
* [HTTP/2] [1] [:scheme: https]
* [HTTP/2] [1] [:authority: 127.0.0.1:8443]
* [HTTP/2] [1] [:path: /upload/BPS.pdf]
* [HTTP/2] [1] [user-agent: curl/8.5.1-DEV]
* [HTTP/2] [1] [accept: */*]
* [HTTP/2] [1] [content-length: 30182355]
> PUT /upload/BPS.pdf HTTP/2
> Host: 127.0.0.1:8443
> User-Agent: curl/8.5.1-DEV
> Accept: */*
> Content-Length: 30182355
>
* HTTP/2 stream 1 was not closed cleanly: PROTOCOL_ERROR (err 1)
* Connection #0 to host 127.0.0.1 left intact
curl: (92) HTTP/2 stream 1 was not closed cleanly: PROTOCOL_ERROR (err 1)
root@ubuntu:~#

Karthikdasari0423 · 2023-12-20T08:01:54Z

am i missing anything here?

Karthikdasari0423 · 2023-12-20T08:24:13Z

a small update
i have a curl 8.2.1-DEV and upload is fine with and without --limit-rate 1M

root@ubuntu:~# /opt/bin/curl -V
curl 8.2.1-DEV (x86_64-pc-linux-gnu) libcurl/8.2.1-DEV OpenSSL/3.1.2 zlib/1.2.11 brotli/1.0.9 nghttp2/1.55.1 ngtcp2/0.17.0 nghttp3/0.13.0
Release-Date: [unreleased]
Protocols: dict file ftp ftps gopher gophers http https imap imaps mqtt pop3 pop3s rtsp smb smbs smtp smtps telnet tftp
Features: alt-svc AsynchDNS brotli HSTS HTTP2 HTTP3 HTTPS-proxy IPv6 Largefile libz NTLM NTLM_WB SSL threadsafe TLS-SRP UnixSockets
root@ubuntu:~#

root@ubuntu:~# /opt/bin/curl -v -k -# --http2 https://127.0.0.1:8443/upload/BPS.pdf -T /upload/BPS.pdf
* processing: https://127.0.0.1:8443/upload/BPS.pdf
*   Trying 127.0.0.1:8443...
* Connected to 127.0.0.1 (127.0.0.1) port 8443
* ALPN: offers h2,http/1.1
* TLSv1.3 (OUT), TLS handshake, Client hello (1):
* TLSv1.3 (IN), TLS handshake, Server hello (2):
* TLSv1.3 (IN), TLS handshake, Encrypted Extensions (8):
* TLSv1.3 (IN), TLS handshake, Certificate (11):
* TLSv1.3 (IN), TLS handshake, CERT verify (15):
* TLSv1.3 (IN), TLS handshake, Finished (20):
* TLSv1.3 (OUT), TLS change cipher, Change cipher spec (1):
* TLSv1.3 (OUT), TLS handshake, Finished (20):
* SSL connection using TLSv1.3 / TLS_AES_256_GCM_SHA384
* ALPN: server accepted h2
* Server certificate:
*  subject: CN=ubuntu
*  start date: Feb 10 15:24:51 2023 GMT
*  expire date: Feb  7 15:24:51 2033 GMT
*  issuer: CN=ubuntu
*  SSL certificate verify result: self-signed certificate (18), continuing anyway.
* using HTTP/2
* h2 [:method: PUT]
* h2 [:scheme: https]
* h2 [:authority: 127.0.0.1:8443]
* h2 [:path: /upload/BPS.pdf]
* h2 [user-agent: curl/8.2.1-DEV]
* h2 [accept: */*]
* h2 [content-length: 30182355]
* Using Stream ID: 1
> PUT /upload/BPS.pdf HTTP/2
> Host: 127.0.0.1:8443
> User-Agent: curl/8.2.1-DEV
> Accept: */*
> Content-Length: 30182355
>
* TLSv1.3 (IN), TLS handshake, Newsession Ticket (4):
* TLSv1.3 (IN), TLS handshake, Newsession Ticket (4):
* old SSL session ID is stale, removing
* We are completely uploaded and fine
< HTTP/2 201
< server: nginx/1.25.3
< date: Wed, 20 Dec 2023 08:23:33 GMT
< content-length: 0
< location: https://127.0.0.1:8443/upload/BPS.pdf
<
* Connection #0 to host 127.0.0.1 left intact
root@ubuntu:~#

root@ubuntu:~# /opt/bin/curl -v -k -# --http2 --limit-rate 1M https://127.0.0.1:8443/upload/BPS.pdf -T /upload/BPS.pdf
* processing: https://127.0.0.1:8443/upload/BPS.pdf
*   Trying 127.0.0.1:8443...
* Connected to 127.0.0.1 (127.0.0.1) port 8443
* ALPN: offers h2,http/1.1
* TLSv1.3 (OUT), TLS handshake, Client hello (1):
* TLSv1.3 (IN), TLS handshake, Server hello (2):
* TLSv1.3 (IN), TLS handshake, Encrypted Extensions (8):
* TLSv1.3 (IN), TLS handshake, Certificate (11):
* TLSv1.3 (IN), TLS handshake, CERT verify (15):
* TLSv1.3 (IN), TLS handshake, Finished (20):
* TLSv1.3 (OUT), TLS change cipher, Change cipher spec (1):
* TLSv1.3 (OUT), TLS handshake, Finished (20):
* SSL connection using TLSv1.3 / TLS_AES_256_GCM_SHA384
* ALPN: server accepted h2
* Server certificate:
*  subject: CN=ubuntu
*  start date: Feb 10 15:24:51 2023 GMT
*  expire date: Feb  7 15:24:51 2033 GMT
*  issuer: CN=ubuntu
*  SSL certificate verify result: self-signed certificate (18), continuing anyway.
* using HTTP/2
* h2 [:method: PUT]
* h2 [:scheme: https]
* h2 [:authority: 127.0.0.1:8443]
* h2 [:path: /upload/BPS.pdf]
* h2 [user-agent: curl/8.2.1-DEV]
* h2 [accept: */*]
* h2 [content-length: 30182355]
* Using Stream ID: 1
> PUT /upload/BPS.pdf HTTP/2
> Host: 127.0.0.1:8443
> User-Agent: curl/8.2.1-DEV
> Accept: */*
> Content-Length: 30182355
>
* TLSv1.3 (IN), TLS handshake, Newsession Ticket (4):
* TLSv1.3 (IN), TLS handshake, Newsession Ticket (4):
* old SSL session ID is stale, removing
* We are completely uploaded and fine
< HTTP/2 201
< server: nginx/1.25.3
< date: Wed, 20 Dec 2023 08:07:16 GMT
< content-length: 0
< location: https://127.0.0.1:8443/upload/BPS.pdf
<
* Connection #0 to host 127.0.0.1 left intact
root@ubuntu:~#

jay · 2023-12-20T08:25:03Z

Interesting. Would you mind trying to reproduce with release tag curl-8_5_0?

Karthikdasari0423 · 2023-12-20T09:04:44Z

tried with released curl-8_5_0 and issue is not seen

root@ubuntu:/src/curl-8.5.0# ./src/curl -V
curl 8.5.0 (x86_64-pc-linux-gnu) libcurl/8.5.0 quictls/3.1.4 zlib/1.2.11 brotli/1.0.9 nghttp2/1.59.0-DEV ngtcp2/1.1.0 nghttp3/1.1.0
Release-Date: 2023-12-06
Protocols: dict file ftp ftps gopher gophers http https imap imaps mqtt pop3 pop3s rtsp smb smbs smtp smtps telnet tftp
Features: alt-svc AsynchDNS brotli HSTS HTTP2 HTTP3 HTTPS-proxy IPv6 Largefile libz NTLM SSL threadsafe TLS-SRP UnixSockets
root@ubuntu:/src/curl-8.5.0#

root@ubuntu:/src/curl-8.5.0# ./src/curl -v -k -# --http2 https://127.0.0.1:8443/upload/BPS.pdf -T /upload/BPS.pdf
*   Trying 127.0.0.1:8443...
* Connected to 127.0.0.1 (127.0.0.1) port 8443
* ALPN: curl offers h2,http/1.1
* TLSv1.3 (OUT), TLS handshake, Client hello (1):
* TLSv1.3 (IN), TLS handshake, Server hello (2):
* TLSv1.3 (IN), TLS handshake, Encrypted Extensions (8):
* TLSv1.3 (IN), TLS handshake, Certificate (11):
* TLSv1.3 (IN), TLS handshake, CERT verify (15):
* TLSv1.3 (IN), TLS handshake, Finished (20):
* TLSv1.3 (OUT), TLS change cipher, Change cipher spec (1):
* TLSv1.3 (OUT), TLS handshake, Finished (20):
* SSL connection using TLSv1.3 / TLS_AES_256_GCM_SHA384 / X25519 / RSASSA-PSS
* ALPN: server accepted h2
* Server certificate:
*  subject: CN=ubuntu
*  start date: Feb 10 15:24:51 2023 GMT
*  expire date: Feb  7 15:24:51 2033 GMT
*  issuer: CN=ubuntu
*  SSL certificate verify result: self-signed certificate (18), continuing anyway.
*   Certificate level 0: Public key type RSA (2048/112 Bits/secBits), signed using sha256WithRSAEncryption
* using HTTP/2
* [HTTP/2] [1] OPENED stream for https://127.0.0.1:8443/upload/BPS.pdf
* [HTTP/2] [1] [:method: PUT]
* [HTTP/2] [1] [:scheme: https]
* [HTTP/2] [1] [:authority: 127.0.0.1:8443]
* [HTTP/2] [1] [:path: /upload/BPS.pdf]
* [HTTP/2] [1] [user-agent: curl/8.5.0]
* [HTTP/2] [1] [accept: */*]
* [HTTP/2] [1] [content-length: 30182355]
> PUT /upload/BPS.pdf HTTP/2
> Host: 127.0.0.1:8443
> User-Agent: curl/8.5.0
> Accept: */*
> Content-Length: 30182355
>
* TLSv1.3 (IN), TLS handshake, Newsession Ticket (4):
* TLSv1.3 (IN), TLS handshake, Newsession Ticket (4):
* old SSL session ID is stale, removing
* We are completely uploaded and fine
< HTTP/2 201
< server: nginx/1.25.3
< date: Wed, 20 Dec 2023 09:02:57 GMT
< content-length: 0
< location: https://127.0.0.1:8443/upload/BPS.pdf
<
* Connection #0 to host 127.0.0.1 left intact
root@ubuntu:/src/curl-8.5.0#

root@ubuntu:/src/curl-8.5.0# ./src/curl -v -k -# --http2 --limit-rate 1M https://127.0.0.1:8443/upload/BPS.pdf -T /upload/BPS.pdf
*   Trying 127.0.0.1:8443...
* Connected to 127.0.0.1 (127.0.0.1) port 8443
* ALPN: curl offers h2,http/1.1
* TLSv1.3 (OUT), TLS handshake, Client hello (1):
* TLSv1.3 (IN), TLS handshake, Server hello (2):
* TLSv1.3 (IN), TLS handshake, Encrypted Extensions (8):
* TLSv1.3 (IN), TLS handshake, Certificate (11):
* TLSv1.3 (IN), TLS handshake, CERT verify (15):
* TLSv1.3 (IN), TLS handshake, Finished (20):
* TLSv1.3 (OUT), TLS change cipher, Change cipher spec (1):
* TLSv1.3 (OUT), TLS handshake, Finished (20):
* SSL connection using TLSv1.3 / TLS_AES_256_GCM_SHA384 / X25519 / RSASSA-PSS
* ALPN: server accepted h2
* Server certificate:
*  subject: CN=ubuntu
*  start date: Feb 10 15:24:51 2023 GMT
*  expire date: Feb  7 15:24:51 2033 GMT
*  issuer: CN=ubuntu
*  SSL certificate verify result: self-signed certificate (18), continuing anyway.
*   Certificate level 0: Public key type RSA (2048/112 Bits/secBits), signed using sha256WithRSAEncryption
* TLSv1.3 (IN), TLS handshake, Newsession Ticket (4):
* TLSv1.3 (IN), TLS handshake, Newsession Ticket (4):
* old SSL session ID is stale, removing
* using HTTP/2
* [HTTP/2] [1] OPENED stream for https://127.0.0.1:8443/upload/BPS.pdf
* [HTTP/2] [1] [:method: PUT]
* [HTTP/2] [1] [:scheme: https]
* [HTTP/2] [1] [:authority: 127.0.0.1:8443]
* [HTTP/2] [1] [:path: /upload/BPS.pdf]
* [HTTP/2] [1] [user-agent: curl/8.5.0]
* [HTTP/2] [1] [accept: */*]
* [HTTP/2] [1] [content-length: 30182355]
> PUT /upload/BPS.pdf HTTP/2
> Host: 127.0.0.1:8443
> User-Agent: curl/8.5.0
> Accept: */*
> Content-Length: 30182355
>
* We are completely uploaded and fine
< HTTP/2 201
< server: nginx/1.25.3
< date: Wed, 20 Dec 2023 09:03:44 GMT
< content-length: 0
< location: https://127.0.0.1:8443/upload/BPS.pdf
<
* Connection #0 to host 127.0.0.1 left intact
root@ubuntu:/src/curl-8.5.0#

bagder · 2023-12-20T12:16:33Z

@Karthikdasari0423 any chance you can bisect to figure out the commit that broke it?

Karthikdasari0423 · 2023-12-20T12:18:58Z

sure @bagder , i will give it a try

Karthikdasari0423 · 2023-12-20T13:47:41Z

@bagder this is the commit that broke this thing
1da640a

root@ubuntu:/src/curl_test# git log
commit 1da640abb6886aab822ff0c0da71b1df0ca89d0f (HEAD)
Author: Daniel Stenberg <daniel@haxx.se>
Date:   Mon Dec 11 19:36:27 2023 +0100

    readwrite_data: loop less

    This function is made to loop in order to drain incoming data
    faster. Completely removing the loop has a measerably negative impact on
    transfer speeds.

    Downsides with the looping include

    - it might call the progress callback much more seldom. Especially if
      the write callback is slow.

    - rate limiting becomes less exact

    - a single transfer might "starve out" other parallel transfers

    - QUIC timers for other connections can't be maintained correctly

    The long term fix should be to remove the loop and optimize coming back
    to avoid the transfer speed penalty.

    This fix lower the max loop count to reduce the starvation problem, and
    avoids the loop completely for when rate-limiting is in progress.

    Ref: #12488
    Ref: https://curl.se/mail/lib-2023-12/0012.html
    Closes #12504

commit 8706b680108919d9cdca1acd3a17fff6a57e245c
Author: Stefan Eissing <stefan@eissing.org>
Date:   Wed Dec 13 11:25:20 2023 +0100

    lib: eliminate `conn->cselect_bits`

    - use `data->state.dselect_bits` everywhere instead
    - remove `bool *comeback` parameter as non-zero
      `data->state.dselect_bits` will indicate that IO is
      incomplete.

root@ubuntu:/src/curl_test#
root@ubuntu:/src/curl_test#
root@ubuntu:/src/curl_test# ./src/curl -v -k -# --http2 --limit-rate 1M https://localhost:8443/upload/BPS.pdf -T /upload/BPS.pdf
* Host localhost:8443 was resolved.
* IPv6: ::1
* IPv4: 127.0.0.1
*   Trying [::1]:8443...
* Connected to localhost (::1) port 8443
* ALPN: curl offers h2,http/1.1
* TLSv1.3 (OUT), TLS handshake, Client hello (1):
* TLSv1.3 (IN), TLS handshake, Server hello (2):
* TLSv1.3 (IN), TLS handshake, Encrypted Extensions (8):
* TLSv1.3 (IN), TLS handshake, Certificate (11):
* TLSv1.3 (IN), TLS handshake, CERT verify (15):
* TLSv1.3 (IN), TLS handshake, Finished (20):
* TLSv1.3 (OUT), TLS change cipher, Change cipher spec (1):
* TLSv1.3 (OUT), TLS handshake, Finished (20):
* SSL connection using TLSv1.3 / TLS_AES_256_GCM_SHA384 / X25519 / RSASSA-PSS
* ALPN: server accepted h2
* Server certificate:
*  subject: CN=ubuntu
*  start date: Feb 10 15:24:51 2023 GMT
*  expire date: Feb  7 15:24:51 2033 GMT
*  issuer: CN=ubuntu
*  SSL certificate verify result: self-signed certificate (18), continuing anyway.
*   Certificate level 0: Public key type RSA (2048/112 Bits/secBits), signed using sha256WithRSAEncryption
* TLSv1.3 (IN), TLS handshake, Newsession Ticket (4):
* TLSv1.3 (IN), TLS handshake, Newsession Ticket (4):
* old SSL session ID is stale, removing
* using HTTP/2
* [HTTP/2] [1] OPENED stream for https://localhost:8443/upload/BPS.pdf
* [HTTP/2] [1] [:method: PUT]
* [HTTP/2] [1] [:scheme: https]
* [HTTP/2] [1] [:authority: localhost:8443]
* [HTTP/2] [1] [:path: /upload/BPS.pdf]
* [HTTP/2] [1] [user-agent: curl/8.5.1-DEV]
* [HTTP/2] [1] [accept: */*]
* [HTTP/2] [1] [content-length: 30182355]
> PUT /upload/BPS.pdf HTTP/2
> Host: localhost:8443
> User-Agent: curl/8.5.1-DEV
> Accept: */*
> Content-Length: 30182355
>

* HTTP/2 stream 1 was not closed cleanly: PROTOCOL_ERROR (err 1)
* Connection #0 to host localhost left intact
curl: (92) HTTP/2 stream 1 was not closed cleanly: PROTOCOL_ERROR (err 1)
root@ubuntu:/src/curl_test#
root@ubuntu:/src/curl_test#

commit before breakage is below
8706b68

root@ubuntu:/src/curl_test# git log
commit 8706b680108919d9cdca1acd3a17fff6a57e245c (HEAD)
Author: Stefan Eissing <stefan@eissing.org>
Date:   Wed Dec 13 11:25:20 2023 +0100

    lib: eliminate `conn->cselect_bits`

    - use `data->state.dselect_bits` everywhere instead
    - remove `bool *comeback` parameter as non-zero
      `data->state.dselect_bits` will indicate that IO is
      incomplete.

    Closes #12512

commit 0f052808b3a054bbbcdf31d16ad168e6fdd1252e
Author: Stefan Eissing <stefan@eissing.org>
Date:   Thu Dec 14 12:59:19 2023 +0100

    connect: refactor `Curl_timeleft()`

    - less local vars, "better" readability
    - added documentation

    Closes #12518

commit 1e4cd519355b63b26a0a9ee1f125153bfcc4995e
Author: Dmitry Karpov <dkarpov@roku.com>
Date:   Wed Dec 13 14:32:53 2023 -0800

    cookie: avoid fopen with empty file name

    Closes #12514

commit 246e0d805ac11aa2a2e14aee485e5deb58653b8b
Author: Viktor Szakats <commit@vsz.me>
Date:   Tue Dec 12 20:38:08 2023 +0000

    tests/server: delete workaround for old-mingw

    mingw-w64 1.0 comes with w32api v3.12, thus doesn't need this.

    Follow-up to 38029101e2d78ba125732b3bab6ec267b80a0e72 #11625

root@ubuntu:/src/curl_test#
root@ubuntu:/src/curl_test# ./src/curl -v -k -# --http2 --limit-rate 1M https://localhost:8443/upload/BPS.pdf -T /upload/BPS.pdf
* Host localhost:8443 was resolved.
* IPv6: ::1
* IPv4: 127.0.0.1
*   Trying [::1]:8443...
* Connected to localhost (::1) port 8443
* ALPN: curl offers h2,http/1.1
* TLSv1.3 (OUT), TLS handshake, Client hello (1):
* TLSv1.3 (IN), TLS handshake, Server hello (2):
* TLSv1.3 (IN), TLS handshake, Encrypted Extensions (8):
* TLSv1.3 (IN), TLS handshake, Certificate (11):
* TLSv1.3 (IN), TLS handshake, CERT verify (15):
* TLSv1.3 (IN), TLS handshake, Finished (20):
* TLSv1.3 (OUT), TLS change cipher, Change cipher spec (1):
* TLSv1.3 (OUT), TLS handshake, Finished (20):
* SSL connection using TLSv1.3 / TLS_AES_256_GCM_SHA384 / X25519 / RSASSA-PSS
* ALPN: server accepted h2
* Server certificate:
*  subject: CN=ubuntu
*  start date: Feb 10 15:24:51 2023 GMT
*  expire date: Feb  7 15:24:51 2033 GMT
*  issuer: CN=ubuntu
*  SSL certificate verify result: self-signed certificate (18), continuing anyway.
*   Certificate level 0: Public key type RSA (2048/112 Bits/secBits), signed using sha256WithRSAEncryption
* TLSv1.3 (IN), TLS handshake, Newsession Ticket (4):
* TLSv1.3 (IN), TLS handshake, Newsession Ticket (4):
* old SSL session ID is stale, removing
* using HTTP/2
* [HTTP/2] [1] OPENED stream for https://localhost:8443/upload/BPS.pdf
* [HTTP/2] [1] [:method: PUT]
* [HTTP/2] [1] [:scheme: https]
* [HTTP/2] [1] [:authority: localhost:8443]
* [HTTP/2] [1] [:path: /upload/BPS.pdf]
* [HTTP/2] [1] [user-agent: curl/8.5.1-DEV]
* [HTTP/2] [1] [accept: */*]
* [HTTP/2] [1] [content-length: 30182355]
> PUT /upload/BPS.pdf HTTP/2
> Host: localhost:8443
> User-Agent: curl/8.5.1-DEV
> Accept: */*
> Content-Length: 30182355
>
* We are completely uploaded and fine
< HTTP/2 201
< server: nginx/1.25.3
< date: Wed, 20 Dec 2023 13:46:40 GMT
< content-length: 0
< location: https://localhost:8443/upload/BPS.pdf
<
* Connection #0 to host localhost left intact
root@ubuntu:/src/curl_test#

bagder · 2023-12-20T14:30:14Z

Thanks a lot!

Karthikdasari0423 · 2023-12-20T15:07:47Z

np, it's my pleasure :)

This reverts commit 1da640a. That commit fixed a few issues but it seems it introduced an even worse one. We need to rethink the approach and revert this in the mean time. Fixes #12559

bagder · 2023-12-22T09:30:23Z

@icing do we run any h2 test case similar to this?

icing · 2023-12-22T09:57:47Z

@bagder we have download tests with ratelimit, but no upload tests.

Adding one locally exposes weirdness...

- refs curl#12559 - add test cases for rate limiting uploads for all http versions - fix transfer loop handling of limits. Signal a re-receive attempt only on exhausting maxloops without an EAGAIN - fix `data->state.selectbits` forcing re-receive to also set re-sending when transfer is doing this.

bagder · 2023-12-22T12:43:46Z

@Karthikdasari0423 can you check if applying #12586 fixes this problem for you?

Karthikdasari0423 · 2023-12-22T13:51:15Z

seems to me working fine @bagder

root@ubuntu:/src/curl_test# git log
commit ca626ff518a0096b91135cbdeebc074b6ce56d78 (HEAD, icing/up-ratelimit-fix)
Author: Stefan Eissing <stefan@eissing.org>
Date:   Fri Dec 22 12:27:59 2023 +0100

    Fix upload rate limiting, add test cases

    - refs #12559
    - add test cases for rate limiting uploads for all
      http versions
    - fix transfer loop handling of limits. Signal a re-receive
      attempt only on exhausting maxloops without an EAGAIN
    - fix `data->state.selectbits` forcing re-receive to also
      set re-sending when transfer is doing this.

commit 35380273b9311cf0741e386284310fa7ca4d005e
Author: Stefan Eissing <stefan@eissing.org>
Date:   Tue Dec 19 12:57:40 2023 +0100

    http2: improved on_stream_close/data_done handling

    - there seems to be a code path that cleans up easy handles without
      triggering DONE or DETACH events to the connection filters. This
      would explain wh nghttp2 still holds stream user data
    - add GOOD check to easy handle used in on_close_callback to
      prevent crashes, ASSERTs in debug builds.
    - NULL the stream user data early before submitting RST
    - add checks in on_stream_close() to identify UNGOOD easy handles

    Reported-by: Hans-Christian Egtvedt
    Fixes #10936
    Closes #12562

commit ef2cf58c77fc92960695d6669648f92ca4792e9c
Author: Daniel Stenberg <daniel@haxx.se>
Date:   Fri Dec 22 09:51:20 2023 +0100

    mprintf: overhaul and bugfixes

    In a test case using lots of snprintf() calls using many commonly used
    %-codes per call, this version is around 30% faster than previous
    version.

root@ubuntu:/src/curl_test# ./src/curl -v -k -# --http2 --limit-rate 1M https://localhost:8443/upload/BPS.pdf -T /upload/BPS.pdf
* Host localhost:8443 was resolved.
* IPv6: ::1
* IPv4: 127.0.0.1
*   Trying [::1]:8443...
* Connected to localhost (::1) port 8443
* ALPN: curl offers h2,http/1.1
* TLSv1.3 (OUT), TLS handshake, Client hello (1):
* TLSv1.3 (IN), TLS handshake, Server hello (2):
* TLSv1.3 (IN), TLS handshake, Encrypted Extensions (8):
* TLSv1.3 (IN), TLS handshake, Certificate (11):
* TLSv1.3 (IN), TLS handshake, CERT verify (15):
* TLSv1.3 (IN), TLS handshake, Finished (20):
* TLSv1.3 (OUT), TLS change cipher, Change cipher spec (1):
* TLSv1.3 (OUT), TLS handshake, Finished (20):
* SSL connection using TLSv1.3 / TLS_AES_256_GCM_SHA384 / X25519 / RSASSA-PSS
* ALPN: server accepted h2
* Server certificate:
*  subject: CN=ubuntu
*  start date: Feb 10 15:24:51 2023 GMT
*  expire date: Feb  7 15:24:51 2033 GMT
*  issuer: CN=ubuntu
*  SSL certificate verify result: self-signed certificate (18), continuing anyway.
*   Certificate level 0: Public key type RSA (2048/112 Bits/secBits), signed using sha256WithRSAEncryption
* TLSv1.3 (IN), TLS handshake, Newsession Ticket (4):
* TLSv1.3 (IN), TLS handshake, Newsession Ticket (4):
* old SSL session ID is stale, removing
* using HTTP/2
* [HTTP/2] [1] OPENED stream for https://localhost:8443/upload/BPS.pdf
* [HTTP/2] [1] [:method: PUT]
* [HTTP/2] [1] [:scheme: https]
* [HTTP/2] [1] [:authority: localhost:8443]
* [HTTP/2] [1] [:path: /upload/BPS.pdf]
* [HTTP/2] [1] [user-agent: curl/8.6.0-DEV]
* [HTTP/2] [1] [accept: */*]
* [HTTP/2] [1] [content-length: 30182355]
> PUT /upload/BPS.pdf HTTP/2
> Host: localhost:8443
> User-Agent: curl/8.6.0-DEV
> Accept: */*
> Content-Length: 30182355
>
* We are completely uploaded and fine
< HTTP/2 201
< server: nginx/1.25.3
< date: Fri, 22 Dec 2023 13:49:54 GMT
< content-length: 0
< location: https://localhost:8443/upload/BPS.pdf
<
* Connection #0 to host localhost left intact
root@ubuntu:/src/curl_test# rm -rfv /root/incoming/
removed '/root/incoming/BPS.pdf'
removed directory '/root/incoming/'
root@ubuntu:/src/curl_test#
root@ubuntu:/src/curl_test# ./src/curl -v -k -# --http3 --limit-rate 1M https://localhost:8443/upload/BPS.pdf -T /upload/BPS.pdf
* Host localhost:8443 was resolved.
* IPv6: ::1
* IPv4: 127.0.0.1
*   Trying [::1]:8443...
* QUIC cipher selection: TLS_AES_128_GCM_SHA256:TLS_AES_256_GCM_SHA384:TLS_CHACHA20_POLY1305_SHA256:TLS_AES_128_CCM_SHA256
* Skipped certificate verification
* Connected to localhost (::1) port 8443
* using HTTP/3
* [HTTP/3] [0] OPENED stream for https://localhost:8443/upload/BPS.pdf
* [HTTP/3] [0] [:method: PUT]
* [HTTP/3] [0] [:scheme: https]
* [HTTP/3] [0] [:authority: localhost:8443]
* [HTTP/3] [0] [:path: /upload/BPS.pdf]
* [HTTP/3] [0] [user-agent: curl/8.6.0-DEV]
* [HTTP/3] [0] [accept: */*]
* [HTTP/3] [0] [content-length: 30182355]
> PUT /upload/BPS.pdf HTTP/3
> Host: localhost:8443
> User-Agent: curl/8.6.0-DEV
> Accept: */*
> Content-Length: 30182355
>
< HTTP/3 201
< server: nginx/1.25.3
< date: Fri, 22 Dec 2023 13:50:30 GMT
< content-length: 0
< location: https://localhost:8443/upload/BPS.pdf
<
* We are completely uploaded and fine
* Connection #0 to host localhost left intact
root@ubuntu:/src/curl_test#

Karthikdasari0423 · 2023-12-23T02:23:59Z

Thank you @bagder & @icing

bagder added a commit that referenced this issue Dec 20, 2023

Revert "readwrite_data: loop less"

f24b6fa

This reverts commit 1da640a. That commit fixed a few issues but it seems it introduced an even worse one. We need to rethink the approach and revert this in the mean time. Fixes #12559

bagder mentioned this issue Dec 20, 2023

Revert "readwrite_data: loop less" #12566

Closed

icing mentioned this issue Dec 22, 2023

Fix upload rate limiting, add test cases #12586

Closed

bagder closed this as completed in e492c7c Dec 22, 2023

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Unable to upload file with curl when --limit-rate is provided #12559

Unable to upload file with curl when --limit-rate is provided #12559

Karthikdasari0423 commented Dec 20, 2023 •

edited

Loading

Karthikdasari0423 commented Dec 20, 2023

Karthikdasari0423 commented Dec 20, 2023

Karthikdasari0423 commented Dec 20, 2023

jay commented Dec 20, 2023

Karthikdasari0423 commented Dec 20, 2023

bagder commented Dec 20, 2023

Karthikdasari0423 commented Dec 20, 2023

Karthikdasari0423 commented Dec 20, 2023 •

edited

Loading

bagder commented Dec 20, 2023

Karthikdasari0423 commented Dec 20, 2023

bagder commented Dec 22, 2023

icing commented Dec 22, 2023

bagder commented Dec 22, 2023

Karthikdasari0423 commented Dec 22, 2023

Karthikdasari0423 commented Dec 23, 2023

Unable to upload file with curl when --limit-rate is provided #12559

Unable to upload file with curl when --limit-rate is provided #12559

Comments

Karthikdasari0423 commented Dec 20, 2023 • edited Loading

I did this

I expected the following

curl/libcurl version

operating system

Karthikdasari0423 commented Dec 20, 2023

Karthikdasari0423 commented Dec 20, 2023

Karthikdasari0423 commented Dec 20, 2023

jay commented Dec 20, 2023

Karthikdasari0423 commented Dec 20, 2023

bagder commented Dec 20, 2023

Karthikdasari0423 commented Dec 20, 2023

Karthikdasari0423 commented Dec 20, 2023 • edited Loading

bagder commented Dec 20, 2023

Karthikdasari0423 commented Dec 20, 2023

bagder commented Dec 22, 2023

icing commented Dec 22, 2023

bagder commented Dec 22, 2023

Karthikdasari0423 commented Dec 22, 2023

Karthikdasari0423 commented Dec 23, 2023

Karthikdasari0423 commented Dec 20, 2023 •

edited

Loading

Karthikdasari0423 commented Dec 20, 2023 •

edited

Loading