Reap the native process tree when a Windows test hangs#595
Merged
Conversation
The Windows x64 leg intermittently died with "the hosted runner lost communication with the server": a crawl wedged and the timeout(1) watchdog around each test could not kill it. MSYS signals don't reach a native httrack.exe, so timeout reaped only the outer bash and left the engine and its python server spinning; the orphans accumulated until the runner starved. The failure-path traced re-run had no watchdog at all, so a deterministic hang burned the full 45-minute job budget. Replace timeout(1) with run_with_timeout/kill_tree in testlib.sh: it polls for the deadline and, on Windows, ends the whole tree by Windows PID via taskkill //T; on POSIX it runs the job in its own process group and signals the group. A hang now fails as 124 in bounded time with the tree reaped, rather than taking the runner down. 57_watchdog.test proves it: exit codes pass through, an overrun reports 124 near the deadline, and a native (Windows) or orphaned (POSIX) grandchild is confirmed dead. It runs on every platform under make check and in the Windows suite. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Signed-off-by: Xavier Roche <roche@httrack.com>
…ously The tree-reap assertions were negative-only: the Windows branch trusted `tasklist` reporting zero PING.EXE, which also holds if ping never launched or the filter were mistyped, and the POSIX branch trusted the marker being absent, which also holds if the grandchild never ran. Since the Windows path can't be exercised off a Windows runner, a vacuous pass there would go unnoticed. Prove the mechanism first: on Windows, confirm a live ping is visible to the filter and that kill_tree reaps it before asserting the grandchild is gone; on POSIX, have the grandchild mark that it ran so marker-absent means reaped, not never-started. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Signed-off-by: Xavier Roche <roche@httrack.com>
The Windows positive control fired in CI: a live ping.exe was running yet the survivor count came back zero. The workflow exports MSYS2_ARG_CONV_EXCL='*', which stops MSYS from folding the `//FI ... //NH` switches down to `/FI ... /NH`, so tasklist saw a mangled filter, matched nothing, and every `-eq 0` check passed without testing anything. Scan the unfiltered tasklist for the image name instead, which takes no switch args. This is exactly the vacuous pass the positive control was added to catch. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Signed-off-by: Xavier Roche <roche@httrack.com>
Signed-off-by: Xavier Roche <roche@httrack.com>
# Conflicts: # tests/Makefile.am
…e count The tasklist filter fix worked (a live ping was detected), but counting every ping.exe globally raced the still-dying ping left by the timing sub-test just above, so the survivor count came back non-zero. Match the grandchild's exact Windows PID instead (tasklist column 2), and have it record that PID so a non-empty value doubles as the positive control. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Signed-off-by: Xavier Roche <roche@httrack.com>
… out The bash -c backgrounded the ping and exited immediately after recording its PID, so run_with_timeout saw it finish in a blink and returned 0, not 124 (and the orphaned ping was never the watchdog's to reap). Add `wait` so the bash blocks on the ping past the deadline, which is what makes the timeout fire and taskkill //T reap the whole tree. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Signed-off-by: Xavier Roche <roche@httrack.com>
A real watchdog defect the new test exposed, not a test-only bug. The workflow sets MSYS_NO_PATHCONV=1 and MSYS2_ARG_CONV_EXCL='*' (so httrack sees URL paths unmangled), which makes MSYS pass every argument verbatim. `taskkill //F //T //PID` therefore reached taskkill with the switches still doubled, was rejected, and did nothing (masked by `|| true`); only the direct child died, via kill -9 translating to TerminateProcess on its winpid. A native grandchild (the wedged httrack.exe in production) would have survived. Use single-slash switches, which pass through cleanly under no-conversion, so taskkill /T actually reaps the tree. Real crawls never hang, so this path was never exercised by the green runs; 58_watchdog.test is what drove it out. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Signed-off-by: Xavier Roche <roche@httrack.com>
This was referenced Jul 16, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Master's Windows x64 leg fails intermittently with "the hosted runner lost communication with the server": the job runs its full 45-minute budget, then the runner dies with no logs uploaded. It's flaky, not a regression (the same commits pass on re-run), but it hides real failures and burns CI minutes.
The cause is the per-test watchdog. Each test ran under
timeout 600, which can't reap a nativehttrack.exe: MSYS signals don't reach non-Cygwin processes, as testlib.sh already notes for the python server. So when a crawl wedged,timeoutkilled only the outer bash and left the engine spinning, and the orphans piled up until the runner starved. The failure-path traced re-run had no watchdog at all, so a deterministic hang went straight to the cap.run_with_timeout/kill_treein testlib.sh replacetimeout: they poll for the deadline and end the whole tree by Windows PID withtaskkill //T(on POSIX, in its own process group, signalling the group). A hang now fails as a bounded 124 with everything reaped.57_watchdog.testguards it: exit codes pass through, an overrun reports 124 near the deadline, and a native or orphaned grandchild is confirmed dead (I checked the oracle fails a watchdog that reaps only the direct child). It doesn't address why the engine's own--max-time=120sometimes fails to fire on a stuck x64 socket, the deeper bug behind the wedge.