Windows x64: possible crawl wedge past --max-time (engine side of the #595 runner hang)

[xroche]

PR #595 fixed the test-harness watchdog that let a wedged crawl take down the Windows x64 runner ("the hosted runner lost communication with the server"). That fix reaps and bounds a hang, but it does not explain why a crawl ran long enough to matter.

Every CI crawl runs with --max-time=120, so no single one should exceed about two minutes. During the 2026-07-15 x64 failure burst the job instead ran its full 45-minute step budget. That points to one of two causes: a crawl ran well past --max-time, or the harness alone accounts for the time (a native httrack.exe that timeout could not signal, left orphaned, plus a failure-path re-run that had no watchdog). #595 fixed the harness side. This issue is about the engine side.

--max-time is enforced by a periodic poll: back_checkmirror / back_mirror_limit in src/htsback.c request a smooth stop past the deadline and hard-stop after a grace (added in #481). The poll only runs while the crawl loop runs. Reads go through select() with a timeout rather than blocking (htsback.c:2641), so a stuck transfer should still let the loop tick, which makes a real --max-time failure less likely than it first looked. Winsocks select()does diverge from POSIX here (#579: a failed connect shows up in the exception set, not as writable), so a Windows-only path where the loop stalls without--max-time` firing is possible but unconfirmed.

There is no reproducer, and it is intermittent and x64-only. Now that #595 caps every test at 600s, a crawl that genuinely wedges past --max-time will show up as a clean 600s timeout of a specific *_local-*.test, not a silent runner death. Concrete next step: if such a timeout appears on x64, capture that tests hts-log.txt and check whether the "More than 120 seconds passed.. giving up" line is present, along with the stuck slots state. If nothing recurs, #595 resolved the symptom and this can be closed.

[xroche]

Looked at the engine side. On POSIX I couldn't make the crawl loop outlast --max-time in any stall shape I tried, and it stays idle (near 0% CPU) while doing so, so I don't think the 45-minute burst was an engine --max-time overrun. It reads much more like the harness hang that #595 already reaps.

How I checked: built the engine with OpenSSL against a stall server that could accept and go silent, send partial headers then hang, send full headers plus Content-Length then hang mid-body, and over TLS either finish the handshake and hang mid-body or accept the TCP connect and never start the handshake. With --max-time=2 (5s grace) every mode exited in about 7s with "More than 2 seconds passed.. giving up", rc 0. Sampling CPU during the stalls showed the process asleep in nanosleep, not spinning.

The code agrees. back_checkmirror runs at the top of every back_wait and on every pass of the per-link wait loop, and once maxtime+grace is exceeded the abort at the top of back_wait force-closes every slot in states 1..999, which covers CONNECTING, WAIT_DNS, SSL_WAIT_HANDSHAKE and the receive states (htsbasenet.h). So a slow or dead socket can't outlast maxtime: reaching the poll reaps it. The Winsock select() quirk from #579 hits the connect path (already fixed), not the data path, and a stalled transfer still lets the loop tick and gets reaped by maxtime+grace.

One real engine gap does defeat --max-time, but it isn't the loopback CI case: DNS resolution is synchronous and unbounded. host_wait is a stub, so a STATUS_WAIT_DNS slot resolves inline through getaddrinfo with no app-level timeout, while holding opt->state.lock. A black-hole resolver would wedge the whole crawl past --max-time, and Windows has no resolv.conf-style cap, so it fits the Windows-only shape. It needs a real hostname though, and CI crawls numeric loopback, so it isn't this symptom. Filed as #606. (A smaller related gap, #607: a stalled TLS handshake is reaped only by --max-time, not --timeout.)

I'd leave this open as you suggested. With #595 capping each test at 600s, a genuine engine overrun would now surface as a clean 600s timeout of a specific *_local-*.test with its hts-log.txt, which is what we'd need to pin it. Nothing I ran reproduces it.

[xroche]

Recommending closure, with the caveat below.

I went through every windows-build run since #595 merged (eb46d8d, 15:33:18Z), and they are all green. One failure carries a post-merge timestamp and looks like a recurrence at first glance: run 29501141975, x64 job 1h2m, "the hosted runner lost communication with the server". It isn't one. That run is attempt 2 of PR #601 on head c826882, which does not have eb46d8d as an ancestor, so it ran the pre-fix harness. Once the same PR's branch picked up the fix (75c735c), it went green in 16m42s: an hour-long hang on the pre-fix head, 16 minutes and a pass on the post-fix one.

The caveat is the sample. That is roughly 90 minutes and about seven post-fix runs, against a wedge that was intermittent and x64-only, so it is consistent with #595 having resolved the symptom rather than proof that it did.

One detail from that hang is worth recording. Its x64 log is gone (BlobNotFound) and no x64 artifact was uploaded, because the runner died before the upload step ran. When the runner dies, GitHub keeps nothing, so the evidence this issue asks for could never have been recovered from that failure in the first place.

A smaller version of that gap survived #595, which is what #612 addresses. A test killed by the 600s watchdog left one truncated line in the artifact: local-crawl.sh sends the crawl's output to its tmpdir and hts-log.txt lives there too, while only tests/*.log is uploaded. #612 salvages those into the artifact, so next time the question this issue poses ("More than 120 seconds passed.. giving up", or not?) can be answered from the artifact.

No engine wedge has ever reproduced, the earlier investigation spun the two real engine gaps out as #606 and #607, and nothing has recurred since the harness fix. I would close this as resolved by #595 rather than keep it open as an unreproducible placeholder. If a *_local-*.test does time out at 600s on x64, that is the signal to reopen, and it will come with the crawl log attached.

[xroche]

Closing this. Nothing has recurred since #595 capped every test at 600s. The one run that looked like a recurrence was a re-run whose timestamp postdates the fix but which executed the pre-fix tree; the same PR passed in 16 minutes once it picked the fix up. A handful of post-fix runs against an intermittent bug is consistent with resolved rather than proof of it, so that caveat stands.

The two engine gaps this investigation did turn up are closed on their own terms: #606 (synchronous DNS could outlast --max-time) by #613, and #607 (a stalled TLS handshake ignored --timeout) by #611. Neither explains the CI symptom, since the CI crawls target a numeric loopback address and never resolve a hostname.

If it does come back, #612 salvages the killed crawl's hts-log.txt into the Windows artifact, so the next occurrence should answer what this issue could not: whether the "More than 120 seconds passed.. giving up" line is present. Reopen on a 600s timeout of a specific test rather than a silent runner death.