Capture thread stacks when the nightly's Bloom hangs (BL-16612)

[andrew-polk]

CI-only instrumentation so that the next time the nightly's visual-regression suite hangs, we learn why. No product code — nothing here can change Bloom's behavior for users, or the behavior of the run it is watching.

The problem it solves

The suite hangs intermittently (BL-16612) — 2 of the 4 scheduled nightlies so far. Bloom stops making progress, stops writing to its log, and all 12 cases then time out at 120s each. Every one of those runs has told us only that something stopped, never what.

What would identify it is each thread's stack while it was stuck. We cannot get that after the fact: the suite's own afterAll kills Bloom with taskkill /T /F before any later workflow step could look at the process.

What this adds

A watchdog that runs alongside the tests (build/ci/watch-for-hung-bloom.ps1), started just before the visual-regression step. It polls for a Bloom.exe that has been alive far longer than a healthy run needs — ~5 minutes healthy versus 20+ when hung — and captures dotnet-stack report for it while it is still wedged. On the observed failures Bloom sat stuck for about 24 minutes, so there is a wide window to catch it.

It takes up to 3 captures a few minutes apart, plus a 30-second process sample line (CPU, thread count, working set). Both matter:

• identical stacks minutes apart prove it is genuinely stuck rather than crawling;
• the CPU trend distinguishes a starved-but-idle server from a spinning renderer.

dotnet-stack, deliberately not dotnet-dump. This repo is public and workflow artifacts are widely downloadable, so we must not publish a memory dump of a process that holds CI secrets. dotnet-stack emits text call stacks only — no heap contents.

Diagnostics are now collected and uploaded on success too, not only on failure. Chasing an intermittent fault means the healthy runs are evidence as well: without a known-good log and sample to compare against, a failing run's numbers mean little on their own. Discarding everything from the green runs is a large part of why this has been so hard to pin down.

Safety

• No product code. Only the workflow, a new CI script, and two .gitignore lines for the scratch folders those steps create.
• Purely observational. The watchdog reads process state; it never signals, kills, or otherwise touches Bloom.
• Cannot fail the nightly. Every operation is best-effort and swallowed; the step ends in exit 0. If the watchdog does not start, the step says so loudly in the log rather than being discovered as an empty artifact the next day.

Why this lands ahead of the actual fix

There is a fix for this hang on a separate branch (#8110). It is deliberately not in this PR. The fix suppresses the failure — so if it lands first, the hang may simply stop happening and the root cause stays unknown, leaving mitigation in place justified by a hypothesis we could never test. The diagnostic opportunity is perishable; the fix can wait a few days.

Verified locally

• The script parses cleanly, and its no-process and polling paths were exercised.
• The detached Start-Process launch writes where expected — worth checking, because Start-Process resolves relative paths against the process working directory rather than PowerShell's location, so a wrong path would mean a watchdog that silently never starts.
• dotnet-stack report against a live .NET process returns real per-thread managed stacks (exit 0, 4.7 KB of text, no memory contents).

Ref: https://issues.bloomlibrary.org/youtrack/issue/BL-16612

Devin review

---

This change is