curl H2 PUT fails with PROTOCOL_ERROR beyond a certain payload size

[vszakats]

I did this

Run (this particular script on macOS):

#!/bin/sh

alias curl='/usr/local/opt/curl-openssl/bin/curl -sS -X PUT https://jsonplaceholder.typicode.com/put/ --data-binary @-'

# OK
dd if=/dev/zero bs=64k count=1 2> /dev/null | curl --http1.1
dd if=/dev/zero bs=64k count=2 2> /dev/null | curl --http1.1
dd if=/dev/zero bs=64k count=3 2> /dev/null | curl --http1.1
dd if=/dev/zero bs=64k count=4 2> /dev/null | curl --http1.1

# OK
dd if=/dev/zero bs=64k count=1 2> /dev/null | curl
dd if=/dev/zero bs=64k count=2 2> /dev/null | curl
# FAIL
dd if=/dev/zero bs=64k count=3 2> /dev/null | curl
dd if=/dev/zero bs=64k count=4 2> /dev/null | curl

Outputs for the FAIL cases above:

• curl output:

curl: (92) HTTP/2 stream 1 was not closed cleanly: PROTOCOL_ERROR (err 1)

• curl -v output:

[...]
> User-Agent: curl/7.53.1
> Accept: */*
> Content-Length: 196608
> Content-Type: application/x-www-form-urlencoded
>.
* Connection state changed (MAX_CONCURRENT_STREAMS updated)!
* We are completely uploaded and fine

[60-second wait]

* HTTP/2 stream 1 was not closed cleanly: PROTOCOL_ERROR (err 1)
* Closing connection 0
* TLSv1.2 (OUT), TLS alert, Client hello (1):

• Server log (nginx 1.10.2):

[...] "PUT /api/path/ HTTP/2.0" 408 0 "-" "curl/7.53.1" "-"

408 is: Request Timeout

I expected the following

Upload to finish successfully, the same way as in HTTP/1.1 mode and 64-128 KiB payloads with H2.

curl/libcurl version

macOS + Homebrew (brew install curl-openssl):

curl 7.53.1 (x86_64-apple-darwin16.4.0) libcurl/7.53.1 OpenSSL/1.0.2k zlib/1.2.8 libssh2/1.8.0 nghttp2/1.21.1
Protocols: dict file ftp ftps gopher http https imap imaps ldap ldaps pop3 pop3s rtsp scp sftp smb smbs smtp smtps telnet tftp 
Features: IPv6 Largefile NTLM NTLM_WB SSL libz TLS-SRP HTTP2 UnixSockets HTTPS-proxy 

Windows Server 2012 R2:

curl 7.53.1 (x86_64-pc-win32) libcurl/7.53.1 OpenSSL/1.1.0e zlib/1.2.11 WinIDN libssh2/1.8.0 nghttp2/1.21.1
Protocols: dict file ftp ftps gopher http https imap imaps ldap ldaps pop3 pop3s rtsp scp sftp smtp smtps telnet tftp 
Features: AsynchDNS IDN IPv6 Largefile SSPI Kerberos SPNEGO NTLM SSL libz TLS-SRP HTTP2 HTTPS-proxy 

UPDATED 2019-02-06: Applied updates necessary for recent Homebrew versions (aka use curl-openssl package instead of a curl with custom options).

[vszakats]

I'm not sure if this issue belongs to curl (or one of its dependencies), but maybe it is interesting nevertheless.

[vszakats]

Updated the test code to work with a public request-tester website. It reproduces the exact same issue.

I've also added the instruction used to obtain the tested Homebrew curl build (it doesn't include H2 by default.)

[jay]

The way you are doing it will send content type urlencoded. I do notice similar though:

curl -v -sS https://nghttp2.org/httpbin/put -T blob.bin

Every time the server will wait ~60 seconds before sending back a 502 reply, according to a number of tries I monitored in Wireshark. And as you say in --http1.1 it will echo back the content immediately.

Another thing interesting about the http2 case is curl eats CPU for the minute it is waiting for the reply. It looks like curl_multi_wait is not actually waiting.

Reproduced in:

curl 7.53.1 (i386-pc-win32) libcurl/7.53.1 OpenSSL/1.0.2k nghttp2/1.21.1
Protocols: dict file ftp ftps gopher http https imap imaps ldap pop3 pop3s rtsp smb smbs smtp smtps telnet tftp
Features: AsynchDNS Debug Largefile NTLM SSL HTTP2 HTTPS-proxy

curl 7.53.1 (i386-pc-win32) libcurl/7.53.1 wolfSSL/3.10.2 nghttp2/1.21.1
Protocols: dict file ftp ftps gopher http https imap imaps ldap pop3 pop3s rtsp smb smbs smtp smtps telnet tftp
Features: AsynchDNS Debug Largefile NTLM SSL HTTP2

curl 7.54.0-DEV (i686-w64-mingw32) libcurl/7.54.0-DEV mbedTLS/2.4.2 zlib/1.2.11 libssh2/1.8.0 nghttp2/1.19.0
Protocols: dict file ftp ftps gopher http https imap imaps ldap ldaps pop3 pop3s rtsp scp sftp smtp smtps telnet tftp
Features: AsynchDNS Debug TrackMemory IPv6 Largefile SSPI Kerberos SPNEGO NTLM SSL libz HTTP2

curl issue #1410.zip is a Wireshark capture (macs and stuff anonymized) made from openssl_dump_secrets branch and these commands:

dd if=/dev/zero of=blob.bin bs=64k count=4 2> /dev/null;
curl -v -sS -X PUT https://nghttp2.org/httpbin/put --data-binary @blob.bin

curl issue #1410.keys      <--- ssl keys for above capture
curl issue #1410.pcapng    <--- packet capture of demo

To view the packet capture load the file then load the keys:
Edit > Preferences > Protocols > SSL > Master-Secret log

It shows libcurl is sending in 16384 DATA chunks, except for the first 64k which oddly it's doing as 16383 + 1

16384 +
16384 +
16384 +
16383 +  <--- 65535 total so far
    1 +  <--- ?? hm don't know why this is split off
16384 +
16384 +
16384 +
16384 +
16384 +
16384 +
16384 +
16384 +
16384 +
16384 +
16384 +  <-- libcurl gets into tight loop here, presumably waiting for reply
16384 =  <-- libcurl sends this last DATA after receiving the 502 one min later
-------
262144

[vszakats]

@jay: Thank you for your extensive tests and info (also for your note about Content-type, I've fixed it locally). This confirms the problem is valid. Question if it's curl or nghttp2-specific.

[jay]

This may be timing dependent, I was unable to reproduce today in Windows 8.1 (wireless) and Ubuntu 16 (wired), it works fine there. My repro above was in Windows 7.

In all of them though it does start with 16384, 16384, 16384, 16383, 1. afaics libcurl tries to write 16384 via nghttp2_session_send but that calls data_source_read_callback requesting 16383, see a vs
codemap for an illustration.

Question if it's curl or nghttp2-specific.

Yes that's the question. A good next step I think will be in Windows 7 I will try a few PUTs of the same size in nghttp client, it looks like it is possible by setting the header method to PUT.

[tatsuhiro-t]

I can answer why there are 16383 and 1 byte writes separately.
Initially, server side flow control window, for both connection and stream, is 65535.
So you can send 16384, 16384, 16384, and 16383, and stop sending data further, wait for WINDOW_UPDATE from server.
OK, then after WINDOW_UPDATE arrives, we continue to send the remaining data. Previously, curl requested 16384 bytes data to send, but libnghttp2 only send 16383, leaving 1 byte. Therefore, the first write after WINDOW_UPDATE is this just 1 byte.

[vszakats]

Replacing curl with the nghttp client, the same uploads do work correctly. If my tests are correct and they can be replicated by others, it means that the issue lies somewhere inside libcurl code.

#!/bin/sh

alias nghttp="nghttp -H ':method: PUT' -H 'Content-Type: application/x-www-form-urlencoded' https://jsonplaceholder.typicode.com/put/ -d blob.bin"

# OK
dd if=/dev/zero of=blob.bin bs=64k count=1 2> /dev/null; nghttp
dd if=/dev/zero of=blob.bin bs=64k count=2 2> /dev/null; nghttp
dd if=/dev/zero of=blob.bin bs=64k count=3 2> /dev/null; nghttp
dd if=/dev/zero of=blob.bin bs=64k count=4 2> /dev/null; nghttp

nghttp version:

$ nghttp --version
nghttp nghttp2/1.24.0

Original example keeps failing using latest curl built against nghttp2 1.24.0:

$ curl -V
curl 7.54.1 (x86_64-apple-darwin16.6.0) libcurl/7.54.1 OpenSSL/1.0.2l zlib/1.2.8 libssh2/1.8.0 nghttp2/1.24.0
Release-Date: 2017-06-14
Protocols: dict file ftp ftps gopher http https imap imaps ldap ldaps pop3 pop3s rtsp scp sftp smb smbs smtp smtps telnet tftp 
Features: IPv6 Largefile NTLM NTLM_WB SSL libz TLS-SRP HTTP2 UnixSockets HTTPS-proxy 

[vszakats]

Just a keep-alive note, that the issue applies to the current latest release as well:

curl 7.55.1 (x86_64-apple-darwin16.7.0) libcurl/7.55.1 OpenSSL/1.0.2l zlib/1.2.8 libssh2/1.8.0 nghttp2/1.24.0
Release-Date: 2017-08-14
Protocols: dict file ftp ftps gopher http https imap imaps ldap ldaps pop3 pop3s rtsp scp sftp smb smbs smtp smtps telnet tftp 
Features: AsynchDNS IPv6 Largefile NTLM NTLM_WB SSL libz TLS-SRP HTTP2 UnixSockets HTTPS-proxy 

[bagder]

Does not repro:

curl 7.56.0-DEV (x86_64-pc-linux-gnu) libcurl/7.56.0-DEV OpenSSL/1.1.0f zlib/1.2.8 c-ares/1.12.1-DEV libidn2/2.0.2 libpsl/0.18.0 (+libidn2/2.0.2) nghttp2/1.23.0-DEV librtmp/2.3

Repro:
curl 7.56.0-DEV (x86_64-apple-darwin16.7.0) libcurl/7.56.0-DEV OpenSSL/1.0.2l zlib/1.2.8 nghttp2/1.21.1

[bagder]

If I reduce the buffer size in curl/curl.h to 8K:

#define CURL_MAX_WRITE_SIZE 8000

... it seems to work every time. When I redirect in the output to a file and enable the progress meter, it seems to work every now and then. Seems buffer/alignment and timing sensitive.

I added --trace-time to the command line and when I get the problem, the pattern is always a 15 second pause. It ends like this:

17:23:06.136874 == Info: We are completely uploaded and fine

15 seconds where nothing is logged. If I set a break-point where the output above is printed, it is clear that libcurl considers all data sent and nothing left to send.

17:23:21.118812 == Info: http2_recv: easy 0x7fcce5005408 (stream 1)
17:23:21.118843 <= Recv SSL data, 5 bytes (0x5) 0000: ....%
17:23:21.118870 == Info: nread=13
17:23:21.118876 == Info: on_frame_recv() header 3 stream 1
17:23:21.118881 == Info: Got frame type 3 for stream 1!
17:23:21.118891 == Info: on_stream_close(), PROTOCOL_ERROR (err 1), stream 1

[jay]

It's definitely sensitive to something, there was one computer I was able to reproduce it on but no others.

[vszakats]

With this fresh version:

curl 7.56.0 (x86_64-apple-darwin17.0.0) libcurl/7.56.0 OpenSSL/1.0.2l zlib/1.2.11 libssh2/1.8.0 nghttp2/1.26.0
Release-Date: 2017-10-04
Protocols: dict file ftp ftps gopher http https imap imaps ldap ldaps pop3 pop3s rtsp scp sftp smb smbs smtp smtps telnet tftp 
Features: AsynchDNS IPv6 Largefile NTLM NTLM_WB SSL libz TLS-SRP HTTP2 UnixSockets HTTPS-proxy 

The failure is still there, but this time I've seen it succeed with bs=64k count=4 a few times. It means I can also confirm it being sensitive to something.

[vszakats]

This mysterious error still prevails as of:

curl 7.58.0 (x86_64-apple-darwin17.4.0) libcurl/7.58.0 OpenSSL/1.0.2n zlib/1.2.11 brotli/1.0.2 libidn2/2.0.4 libssh2/1.8.0 nghttp2/1.29.0
Release-Date: 2018-01-24
Protocols: dict file ftp ftps gopher http https imap imaps ldap ldaps pop3 pop3s rtsp scp sftp smb smbs smtp smtps telnet tftp 
Features: AsynchDNS IDN IPv6 Largefile NTLM NTLM_WB SSL libz brotli TLS-SRP HTTP2 UnixSockets HTTPS-proxy