Repository navigation
Description
I did this
I'm doing requests for uploading ~2.4M small analytics events from a standalone tool as roughly 800 concurrent HTTPS POST requests to a single host (TLS 1.3, HTTP/1.1). Pool is constrained with:
CURLMOPT_MAX_HOST_CONNECTIONS = 16CURLMOPT_MAX_TOTAL_CONNECTIONS = 256CURLMOPT_CONNECTTIMEOUT = 30s- No
CURLOPT_TIMEOUT - Multi interface, repeated
curl_multi_perform, one easy handle per request, added in a tight loop.
The easy handles are created via a CURLSH-shared DNS/connection pool on the multi handle, same config that has been in use for years.
I expected the following
All requests complete successfully (this is what happens with curl 8.15.0 : zero errors).
What goes wrong
About 49 out of ~800 requests fail with CURLE_OPERATION_TIMEDOUT after 30s. The info cache of every failing request is dominated by repeated "No more connections allowed to host" messages (~20-40 of them in a row), followed finally by a late TCP connect attempt that never completes within the remaining budget:
libcurl info message cache 0 (No more connections allowed to host)
libcurl info message cache 1 (No more connections allowed to host)
...
libcurl info message cache 18 (Hostname ___ was found in DNS cache)
libcurl info message cache 19 ( Trying IP:443...)
libcurl info message cache 20 (SSL reusing session with ALPN 'http/1.1')
libcurl info message cache 21 (ALPN: curl offers http/1.1)
libcurl info message cache 22 (TLSv1.3 (OUT), TLS handshake, Client hello (1):)
libcurl info message cache 23 (Connection timed out after 30000 milliseconds)
libcurl info message cache 24 (closing connection #815)
Each failing request spends the bulk of its 30s in MSTATE_PENDING waiting for a slot, gets woken late, and runs out of its connect-timeout budget mid-TLS handshake.
he same workload (same network, same host, same binary except the linked libcurl) produces 0 errors with libcurl 8.15.0.
Total connection IDs also differ: 8.17 reports connection numbers up to #815 (so MaxHost=16 isn't being effectively reused -> connections leave and come back). 8.15 creates a similar number of total connections, but never lets a PENDING handle hit the connect timeout.
Bisect / suspicion
Comparing the source:
Curl_cpool_check_limitsand the dest-limit switch case increate_conn_helper/url.care effectively unchanged between 8.15 and 8.17.- What changed in 8.16 (per release notes & source diff):
multi: process pending, one by one [90]-process_pending_handlesnow wakes only a single pending handle per call (present through master).multi: replace remaining EXPIRE_RUN_NOW [67]->move_pending_to_connect()now callsCurl_multi_mark_dirty(data)instead ofCurl_expire(data, 0, EXPIRE_RUN_NOW). Previously the PENDING handle was re-queued via the splay tree for immediate retry; now it is only marked dirty and depends on a futureprocess_pending_handles()call to be woken.- In 8.17 the happy-eyeballing code was extracted to
cf-ip-happy.cand the connection filter lifecycle was reworked. This also coincides with "vtls: properly handle SSL shutdown timeout [433]" and schannel shutdown tweaks; I haven't isolated whether the connection-filter rework keepsCONN_INUSEtrue longer. - The end result is that under sustained saturation (all 16 live connections busy serving slow remote POSTs), some PENDING handles never get woken before their connect timeout expires.
Things I tried locally in 8.17 that did not fully fix it:
- Revert
process_pending_handlesto wake all pending handles (8.15 behavior). - Reset
TIMER_STARTSINGLE+ re-scheduleEXPIRE_CONNECTTIMEOUTinsidemove_pending_to_connect()so PENDING wait time doesn't count against the connect timeout. - Schedule a 1s
EXPIRE_CONNECTTIMEOUTretry when entering PENDING so the splay tree polls for a slot. - Cap concurrent
cf_ip_attemptentries incf-ip-happy.cat 2 (to match 8.15's two-baller model). - Move
process_pending_handles(data->multi)inmulti_done()to afterCurl_cpool_do_locked(... multi_done_locked ...)so the slot is actually free when pending handles wake.
- None of those reliably eliminate the failures.
The only thing that reliably gives 0 errors is making Curl_cpool_check_limits return CPOOL_LIMIT_OK when the dest bundle is full (i.e. disabling MAX_HOST_CONNECTIONS). That implies the regression isn't about wake/timer semantics but about the real connection lifecycle under 8.17 - something keeps connections counting against dest_limit longer than 8.15 did.
How to reproduce (minimal)
- Multi interface.
CURLMOPT_MAX_HOST_CONNECTIONS = 16,CURLMOPT_MAX_TOTAL_CONNECTIONS = 256.CURLOPT_CONNECTTIMEOUT = 30, noCURLOPT_TIMEOUT.- Queue ~800 simultaneous HTTPS POSTs to the same remote host (small payload, e.g. 100 KB of JSON, against any real TLS server).
- Repeatedly call
curl_multi_performuntil all requests complete.
Happy to put together a standalone C reproducer if that would help - the failure is fully deterministic on my setup with this pattern.
Related
- Multi performance regression 8.13.0 -> 8.14.1 #18017 (introduced the "one by one" pending wake change in 8.14, then adjusted in 8.16)
- Requests start failing after long running connections on 8.17 release #20105 (also 8.15 -> 8.17 connection failures)
curl/libcurl version
libcurl/8.17.0 (OpenSSL 1.1.1t, zlib 1.3, nghttp2 1.64.0, HTTP/2) Windows x64, static lib, built (tried with both debug enabled and disabled)
operating system
Windows 11 / Windows Server 2022 (reproduced on both dev machines).
HTTPS via Schannel not used here -> built with OpenSSL..
Activity
[-]egression (8.15 -> 8.17): CURLMOPT_MAX_HOST_CONNECTIONS causes PENDING handle timeouts under sustained HTTPS load[/-][+]Regression (8.15 -> 8.17): CURLMOPT_MAX_HOST_CONNECTIONS causes PENDING handle timeouts under sustained HTTPS load[/+]bagder commentedon Apr 21, 2026
Do you still see the problem if you run your test case with curl from current git master?
juanbelonepic commentedon Apr 21, 2026
I would need to integrate latest with UE to test, I'd rather fix the issue in 8.17 if we can do that. Happy to bisect commits between 8.17 and master on lib/multi.c / lib/cf-ip-happy.c if you can point me at likely candidates, or put together a standalone C reproducer if that would help test on master.
bagder commentedon Apr 21, 2026
If you make a reproducer I can certainly test that against master.
icing commentedon Apr 21, 2026
We support here mainly the current version of the software and we do not make fixes for past releases, e.g. a curl 8.17.1. We need to reproduce in the current master to make sure the issue still is there and that we have it fixed in the next release.
To your report: since you have only connection timeout configured, it would seem that the connect for a new connection takes more than 30 seconds. That seems to match the part of the trace you listed. It may also be that there is/was a bug in calculating the connect timeout. Or it may be that the server did not respond for 30 seconds.
A reproducer would certainly help. Otherwise a trace with timestamps could clarify this.
juanbelonepic commentedon Apr 21, 2026
I was wrong, I can't find a way to reproduce the issue purely in curl 8.17 :( , need to investigate our code base more sorry.
I've made a standalone reproducer but without UE in the loop can't repro the bug..., with only libcurl 8.17.0 that runs clean in 2-3 seconds. The UE binary using the same libcurl reliably fails ~25-49/800 requests with CURLE_OPERATION_TIMEDOUT.
So the regression exists but the trigger is UE-side something I haven't isolated. I'll keep digging on our side, probably by stubbing out subsystems in the UE HTTP module one at a time.
Thanks for your time.
Juan
juanbelonepic commentedon Apr 22, 2026
Found the bug, the theory is that
CURLOPT_RESOLVEpre-pins the IP for our datarouter url, curl skips DNS resolution, connections establish faster, the host-connection pool doesn't stay saturated,PENDINGhandles get their turn before the 30s andCURLOPT_CONNECTTIMEOUTfires.This standalone C reproducer simulates an analytics-style flush: main thread records events into payloads, then bursts ~500 ~100KB JSON POSTs onto a shared queue; an HTTP thread drains the queue into a single CURLM at 200Hz.
Repro parameters (see top of file):
MAX_HOST_CONNECTIONS=16
MAX_TOTAL_CONNECTIONS=256
CONNECTTIMEOUT=30
HTTP_TICK_MS=5
DEFAULT_URL=https://httpbin.org/delay/1
Optional control run that pre-pins DNS via CURLOPT_RESOLVE (does not change the outcome meaningfully):
repro_pending_timeout.exe https://httpbin.org/delay/1 httpbin.org:443:3.220.42.21What we think changed
Between 8.15 and 8.17 (likely in the 8.16 work around process_pending_handles / Curl_multi_mark_dirty), handles in STATE_PENDING appear to wait passively for process_pending_handles() to be invoked from another code path, rather than actively re-polling via the splay tree. When the pool is saturated by the first N connections, and those first N take meaningful wall-clock time to reach MSTATE_DO (TLS handshake against a normal public HTTPS endpoint is enough), the remaining handles' CURLOPT_CONNECTTIMEOUT counts down before they ever get a slot.
Questions
Thanks
icing commentedon Apr 22, 2026
It also does in 8.15. I just built that version, set a fixed limit of max host connections to 1 and ran:
> curl -o /dev/null https://curl.se/download/curl-8.19.0.tar.xz --limit-rate 48k --http1.1 --connect-timeout 1 -o /dev/null https://curl.se/download/curl-8.19.0.tar.xz --parallelThe second url times out after 1 second.
I hope it is. The question also is: are they fairly rescheduled. We currently try the first pending handle in the bitset and that may be unfair to handles that got assigned higher
midvalues. We should probably address that.Other than configuring a larger connect timeout, I can think of none.
icing commentedon Apr 22, 2026
I just made #21412 which makes scheduling of pending transfers more fairly. The patch should apply pretty cleanly to 8.15 code as well, if you want to try it.
juanbelonepic commentedon Apr 23, 2026
thanks @icing , I had to modify it a bit to make it works in 8.17 + UE:
thank you!
icing commentedon Apr 23, 2026
Does it solve your problem?
Just saw your first part: resetting the connect timeout is not what we do. You have your own curl now!
What I'd really like to know: does #21412 solve your problem without your additional timeout resetting? Otherwise, do you want to patch your own curl going forward forever?
juanbelonepic commentedon Apr 23, 2026
Your #21412 does fix the standalone C reproducer I sent, yes.
I haven't re-run without your fix to double-check since I had both patches stacked, but as I was seeing 25-30% timeouts and with yours #21412 applied the repro drops to 0 failures against the same public endpoint.
So the PENDING-bitset fairness work is doing what it should.
The important update on my end was that while validating your patch I discovered our internal failing test had an additional cause I had been ignoring, our UE-side HTTP layer's debug callback was doing expensive per-buffer work (UE_LOG + a lock-protected activity-timer reset) synchronously inside the curl thread, thousands of times per burst upload. That was starving
curl_multi_performand producing the sameCURLE_OPERATION_TIMEDOUT+ "No more connections allowed to host" symptoms your PR addresses.First we are going to try if this 2 part patch works, then we will need to come to a version that we dont need to patch so much, we save a patch file to apply per version but the idea is to keep the patches at minimum so we can go forward in versions as far as possible and without need to adapt too much code.
icing commentedon Apr 23, 2026
Ok, then I consider the issue fixed from our side on the data that we have. Thanks.
multi: enhance pending handles fairness
multi: enhance pending handles fairness
multi: enhance pending handles fairness