"Error in the HTTP2 framing layer" after 1000 requests #3750
Comments
|
Tricky. I tried both of your versions and I've not made any of them error for me, in spite of me having run both versions tens of times. I've tried both my 7.65.0-DEV version and my 7.64.0 system install! |
|
We had a similar bug report in Fedora: https://bugzilla.redhat.com/1690971 ... but it was resolved (or worked around) server-side. |
|
I realized I forgot to add to the report that this is happening behind a HTTP proxy 😞. It didn't seem too relevant at the time of writing, but I'm now increasingly convinced that the proxy is changing some behavior on the TCP level, making the issue much more likely to occur. Yay for automatic configuration from environment variables 😄 The last two requests look like this: Those are streams 0x13 (19) and 0x15 (21). The second one is supposed to fail (or not happen at all), because http2_max_requests is 10. What stands out most are these:
In cases where it works properly, I see these:
I suspect there's an issue here around how the tls alerts are handled. But, it sounds like that may be really hard to reproduce... [edit] I'm able to reproduce this on the latest 'alpine' docker image, by running 'apk add curl' (adds 7.64.0), and then running the relevant command. Only when setting the https_proxy environment variable, though. So there's definitely some proxy magic going on on the TCP level. |
|
Is this a race condition with RST because if so I think we're pretty much SOL. Is this related: https://trac.nginx.org/nginx/ticket/1250 |
It could well be a race condition, but I don't think we're SOL just yet :) curl receives the TLS close notify before sending the next request, so it could know better than to send it out. There should also be a GOAWAY frame indicating the last accepted stream ID, and curl should use that frame to decide whether to retry the request on a new connection (but it's not logged in curl's verbose output...). |
|
I'm able to cause a curl segfault if I do concurrent h2 requests over a connection that's going away. Maybe this somehow relates to the original issue as well, so instead of creating a new report I'll add the code here: https://gist.github.com/TvdW/3bb1415313e259b79522f9128525dbf2 The code essentially does 1000 requests to a h2 server, but since http2_max_requests is set much lower it needs to recycle connections long before that, and segfaults while doing so. |
|
(Just double-checked I can actually reproduce the segfault on a different machine in a completely different environment, eg. no proxy, etc. I can. Now let's hope the two issues are related and this should be an easy bug fix 👍 ) |
|
@TvdW, I can reproduce the crash with 7.64.1 using your example but for some reason it doesn't happen with git master. Can you check if that is the same for you or just me having something weird?
If you use |
|
@bagder Yeah, I can confirm the segfault is no longer reproducible on git master. git-bisect traces it to 2f44e94... 😃 Sadly the original bug still exists, so they were independent problems... BUT, with --enable-debug, a lot more information is revealed! The "forcibly told to drain data" then goes on for many thousands of lines before I can ^C it. Maybe this reveals enough to be able to reproduce it? |
|
This piece from the log above seems to indicate that the server closes the connection but yet curl reuses it for the subsequent request and then it goes totally bananas: Can you try and see if a patch like this below makes anything better? diff --git a/lib/transfer.c b/lib/transfer.c
index a451fb74e..1c3734f41 100644
--- a/lib/transfer.c
+++ b/lib/transfer.c
@@ -622,10 +622,15 @@ static CURLcode readwrite_data(struct Curl_easy *data,
else {
/* if we receive 0 or less here, the server closed the connection
and we bail out from this! */
DEBUGF(infof(data, "nread <= 0, server closed connection, bailing\n"));
k->keepon &= ~KEEP_RECV;
+ if(!(conn->handler->flags & PROTOPT_DUAL))
+ /* Dirty, but avoids this for FTP since we get this for the data
+ transfer and it is then not a signal to kill the control connection
+ too! */
+ streamclose(conn, "Server-initiated connection close");
break;
}
/* Default buffer to use when we write the buffer, it may be changed
in the flow below before the actual storing is done. */ |
|
No joy. |
|
Try |
|
@bagder Switching to connclose makes it execute all requests successfully, but... only one request per h2 connection is executed. 😃 The "nread <= 0, server closed connection, bailing" message is shown after every request, so that may explain why. In the case where curl does the right thing, this is shown: |
|
Thanks, yes that's not the correct fix for this issue. I just wanted to verify that marking the connection as closed would at least avoid the following situation. So, let's drop that patch and instead see if we can act on the diff --git a/lib/vtls/openssl.c b/lib/vtls/openssl.c
index eff5c2106..5d2aac7d3 100644
--- a/lib/vtls/openssl.c
+++ b/lib/vtls/openssl.c
@@ -3754,11 +3754,14 @@ static ssize_t ossl_recv(struct connectdata *conn, /* connection data */
/* failed SSL_read */
int err = SSL_get_error(BACKEND->handle, (int)nread);
switch(err) {
case SSL_ERROR_NONE: /* this is not an error */
+ break;
case SSL_ERROR_ZERO_RETURN: /* no more data */
+ /* close_notify alert */
+ connclose(conn, "TLS close_notify");
break;
case SSL_ERROR_WANT_READ:
case SSL_ERROR_WANT_WRITE:
/* there's data pending, re-invoke SSL_read() */
*curlcode = CURLE_AGAIN; |
|
Works! 😃 |
Without this, detecting and avoid reusing a closed TLS connection (without a previous GOAWAY) when doing HTTP/2 is tricky. Reported-by: Tom van der Woerdt Fixes #3750
I did this
This started out as doing several thousand requests to some company's API, and getting occasional errors stating "Error in the HTTP2 framing layer". It was found out that this always happened with stream id 7d1 (2001), also known as request 1001. This lead me to http2_max_requests in nginx. A few hours of minimizing the test case later, I've found two ways to reproduce this bug, generating different errors for the same task, when they should both just work fine.
Method one
The sleep server from my previous bug report returns :) nginx is running on the server with http2_max_requests set to 10, for easier debugging. The sleep server just sleeps the given amount of milliseconds, and the issue seems to become reproducible around 5ms, but consistently so around 100ms.
Running the command above will execute 11 requests (one more than the limit). The last request times out, but there's no reason for it to.
Method two
This uses the multi socket api, and produces the "Error in the HTTP2 framing layer" error message: https://gist.github.com/TvdW/a8bf6eabc99bf812d2fd0be1d32f9313
It's a crude eventloop implementation doing the bare minimum for making the multi socket API work. This is the last version of my code before I managed to reproduce the bug using the curl command line utility. What's notable about it is that it reproduces the issue, but with a different error message...
curl/libcurl version
operating system
Linux, CentOS 7
thoughts
Some things stand out:
The text was updated successfully, but these errors were encountered: