libcurl: stop reading from connection when client has paused receivin… #3240
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
…g data
Goal: Allow libcurl pause-unpause behavior to control flow of data from server with limited memory consumption.
Setup: Using curl_multi_perform to transfer GBs of data from a server that is setup using a curl_easy_handle. (HTTP 1.1 and HTTP 2) When it is not possible to accept data, the WRITEFUNCTION returns CURL_WRITEFUNC_PAUSE. Curl stops sending data to the write function.
Problem:
Use curl_easy_pause(easy_handle, CURLPAUSE_CONT) in the main thread to 'unpause' the transfer that was paused earlier. The un-paused transfer starts and calls the WRITEFUNC again and after a few invocation gets CURL_WRITEFUNC_PAUSE. At this point, libcurl continues to read data from the connection and has to store it off of the easy handle because it cannot deliver the data to the WRITEFUNCTION that has paused. When reading GBs of data, this ultimately leads to Out of memory error from CURL and the transfer is aborted.
Root cause, in my opinion:
curl_multi_perform/curl_easy_pause ultimately calls lib/transfer.c:readwrite_data to read data off the connection from http server and delivers to WRITEFUNCTION using Curl_client_write. When WRITEFUNC requests pause, this state is set in the easy-handle but readwrite_data does not check this state and continues to read data from socket but cannot deliver to the WRITEFUNC.
Solution:
readwrite_data should check the KEEP_RECV_PAUSE flag in the easy handle and stop reading when it is set.