HTTP/2 POST: Busy loop after 64k if request not read on server

[steini2000]

The test program sends a POST request using HTTP/2 and CURLOPT_READFUNCTION.
On the server the request is received, but for testing the server does not read the buffer.

If more then 65535 bytes are returned from the read function (in total, after multiple calls of the read function), the test program goes into a busy loop:

• curl_multi_poll() returns immediately
• read function is not called
• no data on the wire

The situation is not satisfying: Nothing happens, but 100% CPU is used.

The following debug log is written during the busy loop for libcurl < 7.75:
=> Send data, 0 bytes (0x0)
This curl_debug() has been disabled with commit a5bc272.

Not reading the request on the server is only the edge case. When (slowly) reading the request on the server, libcurl sometimes still uses 100% CPU and does nothing.

Same problem when using select() instead of curl_multi_poll().

The problem goes away if using CURL_HTTP_VERSION_1_1:
More then 64k data is sent.

The problem does not appear if server sends status code >= 300:
Info: HTTP error before end of send, stop sending
-> RST_STREAM
If the status code is e.g. 204, problem is there.

The problem does not happen with new Tomcat versions. Most likely the relevant change is in 9.0.18:
When Tomcat writes a final response without reading all of an HTTP/2 request, reset the stream to inform the client that the remaining request body is not required. (markt)
But nevertheless the old Tomcat behaviour is valid and could happen with other HTTP/2 implementations.

Tested versions:
7.68.0 with nghttp2 1.40 on x64
7.79.1 with nghttp2 1.41 on arm
Tomcat 9.0.16, does not happen with 9.0.53

Server not reading the buffer is a valid state (at least temporary), libcurl should not busy loop after 64k. Different behaviour for HTTP 1.1 and HTTP/2.

Test programs and log

[steini2000]

When returning <= 65535 bytes from the read callback, everything is fine: idle loop. curl_multi_poll() returns after the timeout.

[bagder]

When returning <= 65535 bytes from the read callback

You then mean in total right? Because the way this code works it will never return that large single values from the read callback?

[steini2000]

When returning <= 65535 bytes from the read callback

You then mean in total right? Because the way this code works it will never return that large single values from the read callback?

Yes, in total. Using multiple calls of the read callback.

[bagder]

I don't know how to reproduce this. I don't have a tomcat installation and I really can't make myself go down that rabbit hole for this purpose. Any publicly accessible URL that shows the same issue?

[steini2000]

Not at the moment. But I should be able to setup one. Please allow some time.
I would feel better if I would not have to post it here. Is it OK to mail it?

[bagder]

Yes, thanks, that would be excellent!

[bagder]

[I received a URL to a test server setup by @steini2000 and it is most useful!]

I think this is a curl bug that is most noticeable because of a server bug. It turns out that the server doesn't enlarge the remote window, so it drains (thus not allowing more data to get sent) but curl doesn't notice (partly because nghttp2_session_want_write() still returns TRUE in this situation) and then busy-loops around the trying to send.

Fixing the busy-loop (by checking the remote window before waiting for socket writability) doesn't make the operation succeed because the server doesn't enlarge the window so there's no permission to send data.

The fact that this behavior is gone in a newer tomcat release could perhaps be a sign that this bug is fixed already.

PR coming up.

[steini2000]

This seems to fix the busy loop. Thanks!

Newer Tomcat versions always send a RST_STREAM when sending a response without reading all of the request. I have not tested without sending a response. There are a lot of changes in Tomcat regarding the flow control window. But this is not important any more: libcurl is prepared!