/ curl Public
multi: fix queueing of pending easy handles #1358
Add this suggestion to a batch that can be applied as a single commit. This suggestion is invalid because no changes were made to the code. Suggestions cannot be applied while the pull request is closed. Suggestions cannot be applied while viewing a subset of changes. Only one suggestion per line can be applied in a batch. Add this suggestion to a batch that can be applied as a single commit. Applying suggestions on deleted lines is not supported. You must change the existing code in this line in order to create a valid suggestion. Outdated suggestions cannot be applied. This suggestion has been applied or marked resolved. Suggestions cannot be applied from pending reviews. Suggestions cannot be applied on multi-line comments. Suggestions cannot be applied while the pull request is queued to merge.
Multi handles "try and maintain a FIFO queue so the pipelined requests are in order". That is why newly added easy handles are added to end of the multi->easyp linked list.
When using the multi_socket API with the total number of connections limited by CURLMOPT_MAX_TOTAL_CONNECTIONS, this FIFO behaviour can be lost. This can be demonstrated by the slightly modified "multi-uv" example. We create a multi handle, and limit CURLMOPT_MAX_TOTAL_CONNECTIONS to 1 (this wouldn't make much sense in real life, but it helps to demonstrate the problem clearly). Then we add 16 easy handles (HTTP GET requests to the same URL) and log when these requests are finished (see: multi-pending-fix-test.txt).
It turns out that the requests are finished in this order:
This is caused by the repeated inversion of the list of easy handles waiting for connections (multi->pending) in a multistep process.
Let's begin from a state where one easy handle is executing and the others are in STATE_CONNECT_PEND.
- this means that the first timeout to be chosen and removed from the splay tree will be the one which is the largest smaller than the current time and, as a result, the expired timeouts will be iterated from the largest (most recently expired) to the smallest (least recently expired)
- in case when multiple timeouts (easy handles) belong to the same key (timestamp) in the splay tree, a linked list of timeouts are used
- when a node is inserted to the splay tree with a key already existing in the tree, it will be inserted in place of the current timeout (to the head of the list of same timeouts) and when removed, the head of this list will be removed first, and replaced by the next element of the list
- this means that the first element removed from the list of same timeouts will the one added last which is consistent with the demonstrated behaviour of Curl_splaygetbest
- therefore Curl_splaygetbest exhibits a FILO behaviour
- the last easy_handle added will take the available connection, the rest will be added to multi->pending in a reverse order
This process repeats until no more handles are left, inverting the list of pending handles at every iteration and giving the available connection to the first easy handle in the list, causing the observed 1-15,2-14,3-13,... pattern.
This particular problem could probably be solved by a quick-and-dirty solution: iterating on the multi->pending list in a reverse order in Curl_multi_process_pending_handles, negating the effect of the described process.
I would argue, however, that this problem is inherent to the workings of Curl_splaygetbest and the definition of "best" in the context of expired timeouts.
I think we should deal with timeouts which expired the longest time ago first, and define "best" in the context of expired timeouts as "smallest not larger than now".
This would fix this issue, and it seems generally the logical approach to me. However, I recognize that this may disable some optimalization I do not yet comprehend.
The patch that I propose contains the following modifications: