Join GitHub today
GitHub is home to over 40 million developers working together to host and review code, manage projects, and build software together.Sign up
HTTP/2 final frames not processed on server disconnect #1021
I did this
(This is from #1013. While investigating that issue this one was uncovered, but I do not believe they are related.)
Attempted to POST multipart chunked data via HTTP/2 to an amazon server that immediately returns a 403 and then attempts a clean disconnect. The 403 and its body do not appear to always be returned to the client, they are lost somewhere. Instead either an
This isn't 100% reproducible, it seems to be timing dependent. Sometimes the body and response are returned.
When things go wrong it looks like:
And then in either case:
At this point the server sends close-notify and FIN/ACK. The client however goes on to send SETTINGS and in some cases DATA before an RST from the server:
I'm not sure if this is a libcurl or nghttp2 issue. I've built with enable-debug mode in both nghttp2 and libcurl and neither output shows that server header and data have been received. To confirm I changed
Reproduce and data files
To view the packet capture load the file then load the keys:
To reproduce build the demo code and make sure it loads libcurl from curl-7_50_3-6-ge01d0f1 2016-09-16 or later. Or better, to auto decrypt in Wireshark you can use instead the branch I made from there and I adapted @Lekensteyn's sslkeylog and it will dump encryption keys if you use libcurl w/openssl and set environment variable SSLKEYLOGFILE. You will also need to set that path as the master-secret log (see above). master...jay:openssl_dump_secrets (Side note: it would be helpful if something like his code eventually made it in to libcurl)
It is easy for me to reproduce in Windows but difficult in Linux. In Linux it seems very dependent on the amount of data that is sent. The test case attempts to post a 64KB file sound.wav. You may need to increase the size of the file to reproduce. And if you decrease the size to something like 2 bytes the server response should be much more likely to be returned.
I expected the following
For the server's response to be returned in all cases.
curl/libcurl version and operating system
Win 7 x64 Enterprise:
Ubuntu 16 x64 LTS:
I played around with demo_code_modified.c, and reproduced the issues several times.
From debug logging, it looks like curl failed to send DATA frame. I saw this logging message:
And somehow curl tried to send more frames. This is probably the bug: we should return -1 here
If we fixed the above line of code, then the client now did not send more frames, but it still did not fully receive HEADERS, and DATA from remote server.
No, it's still reproducible. I appreciate Tatsuhiro's analysis as always. I tried returning -1 where he said but I still don't see the headers returned. This is discussed above, and what lib is at fault if any I don't know. This is just a time consuming issue and I've focused on other things instead.
I'm looking into this issue now as I want to go over all our currently open http2 bugs. I can't reproduce the bug mentioned using the attached code. @jay, can you take a quick look and see if you can or if you can alter it slightly to trigger it again? If not, I think we'll need to close this until we see it happen again.