Windows: investigate `restart_service` crash blocking component-load test paths

[heskew]

Context

While trying to expand Windows integration test coverage (#525, #546, #547), the actual underlying blocker came into focus. Multiple integration test files have per-test skips for restart_service operations with the same explanation:

'Windows: Skipping restart_service to avoid crash'

Found in:

• integrationTests/apiTests/tests/23_blob.mjs (2 occurrences)
• Likely elsewhere — grep -rE "Windows.*Skipping restart_service|restart_service.*crash" for the full set

These skips look surgical and isolated but they're load-bearing in suite flow: any test that needs Harper to pick up a newly-written component file depends on restart_service to actually load it. With the operation skipped, downstream tests fail with "schema not found" / "component not loaded" cascades.

This was the latent root cause behind #546/#547. The outer suite-level skip on 23_blob.mjs ({ skip: process.platform === 'win32' }) is effectively a workaround for this — undocumented, but real.

See #547 (closed) for the cascade trace: the file-write portion of the blob suite passes on Windows, then "Confirm Blob schema and table created" fails because the schema isn't there, because the component was never loaded, because restart_service was skipped.

Goal

Make restart_service work on Windows, OR replace the operation in the test paths with a Windows-compatible alternative that achieves the same effect (Harper picks up a newly-written component).

Approach (suggested)

1. Reproduce the crash locally on Windows. Run a minimal test that calls restart_service (e.g., one of the currently-skipped tests in 23_blob.mjs). Capture the actual crash signature — stack trace, exit code, hdb.log output.
2. Identify the failure mode. Likely candidates based on Kris's prior Windows fixes:
   • File-locking / EPERM during the restart sequence (7823d97a "workaround database drop crashes on windows" suggests this pattern is present)
   • Worker-process spawn ordering specific to Windows process semantics
   • Socket / pipe handling for IPC during the restart handshake
   • Database file handles not released cleanly across the restart
3. Fix or work around. Either fix the crash in Harper itself, or introduce a Windows-specific code path (e.g., separate-process restart, retry-with-backoff on EPERM, explicit file-handle release before restart).
4. Un-skip the affected tests. Once restart_service works on Windows, remove the 'Windows: Skipping restart_service to avoid crash' skips. The outer suite-level skip on 23_blob.mjs becomes removable as a follow-up — that work is well-covered by Windows integration tests: investigate and reduce skips in 23_blob.mjs #546's framing, just gated on this issue first.

Success criteria

• A Windows integration test can successfully call restart_service and observe the newly-loaded component active after the call.
• The 'Windows: Skipping restart_service to avoid crash' skips can be removed from 23_blob.mjs (and any other files with the same pattern).
• run-integration-apiTests-windows CI job continues to pass.

Out of scope

• Un-skipping the broader 23_blob.mjs suite — that's Windows integration tests: investigate and reduce skips in 23_blob.mjs #546's territory, gated on this issue.
• Other Windows skips unrelated to restart_service (e.g., 8_deleteTests.mjs "Windows does not allow * in directory names" — that's a fundamentally different OS-semantic mismatch).

References

• PR test(windows): remove suite-level skip from 23_blob.mjs, narrow to per-test #547 (closed) — cascade diagnosis showing the load-bearing dependency
• Issue Windows integration tests: investigate and reduce skips in 23_blob.mjs #546 (closed) — superseded by this; same goal but blocked on restart_service
• Issue Windows Integration Testing #525 (Kris's original) — broader Windows expansion framing
• PR Investigate Integration Test Flakiness #528 (merged) — Kris's prior Windows fixes
• Related Windows commits: 7823d97a, a9f55dc4, aa034530, 351d1a40

(🤖)

[heskew]

Investigation update

Tracing the call path and looking at related Windows skips, the picture is sharper than the initial issue body suggested. This is almost certainly a native-binding cleanup crash during worker thread termination, not a Harper application-code bug.

Call path

`restart_service` with `service: 'http_workers'` resolves to:

1. `components/operations.js` operation handler → `server/serverHelpers/serverUtilities.ts` → `bin/restart.js` `restartService()`
2. `restartService()` posts an ITC `RESTART` message to the main thread (line 113-117)
3. Main thread handler (`bin/restart.js:30-34`) calls `restartService({ service: 'http_workers' })` recursively, this time on the main thread
4. Main thread path goes to `server/threads/manageThreads.js` → `restartWorkers('http')`
5. `restartWorkers` posts SHUTDOWN to each worker, waits for exit (with a forced `worker.terminate()` fallback if the worker doesn't self-exit within `threadTerminationTimeout * 2`)

Related skips form a pattern

Two operations are skipped on Windows specifically because they crash the Harper process:

Operation	Skip location	Commit
`drop_schema`	`integrationTests/apiTests/utils/schema.mjs` (function-level)	`7823d97a` (2026-05-13) "workaround database drop crashes on windows"
`restart_service` (http_workers)	`23_blob.mjs` (per-test)	`0751644f` (2026-05-14) "skip http_worker restart tests on Windows to avoid crashing"

Kris's commit message on the restart_service skip explicitly says "Like dropSchema, we skip this natively on win32" — connecting the two crashes as the same class of problem.

What `drop_schema` and `restart_service` have in common:

• Both tear down native-bound resources (LMDB file maps + workers' open DB handles)
• Both involve closing file handles that Windows is strict about
• Both happen at process / worker / database lifecycle boundaries

Strong hypothesis

The crash is in native binding cleanup during worker termination on Windows. When `restartWorkers('http')` terminates the HTTP worker thread, the worker had open database handles. Native LMDB (and/or RocksDB) cleanup runs as the thread exits. Windows file-handle semantics differ from Linux/macOS (handles can't always be released while memory-mapped, file deletion blocked while open, etc.), and the native binding's cleanup path hits an unrecoverable error → segfault → Harper process dies.

Supporting evidence:

• `7fe73a53` "Fix flaky unit tests: timing races and LMDB segfault on Node v24" — confirms the LMDB native module is fragile under non-Linux/non-Node22 conditions. `test:unit:lmdb` is currently skipped on Node v24 entirely.
• Kris is the author of `lmdb-js`. He chose to skip both the dropSchema and restart_service tests rather than fix the underlying crash — suggesting either (a) the fix is in the native binding and requires deeper work, or (b) the fix is in Harper's worker-shutdown ordering and isn't obvious.
• Existing Windows-aware code in the thread layer (`server/threads/socketRouter.ts` forces threadCount=1 on Windows, `server/threads/threadServer.js` disables SO_REUSEPORT) shows Windows-specific quirks have already required code paths. This is more of the same.

Why this is not a good `claude-fix:*` target right now

• No reproducible local Windows environment described in the repo
• The actual crash stack trace isn't captured anywhere (hdb.log on Windows CI is uploaded as an artifact but only on the test runs that already skip the operation; the crash itself has never been captured)
• Fix is likely either in `lmdb-js` native code or in worker-shutdown ordering — both require iterative test cycles on Windows to validate, which the agent can't drive

Recommended next step (a different, much narrower issue)

Before attempting a fix, capture the actual crash signature. Concrete spike:

1. Add a separate Windows CI job (or a one-shot manual workflow) that:
   • Calls `restart_service { service: 'http_workers' }` on a fresh Harper install
   • Captures hdb.log, stdout, stderr, and any core dump
   • Uploads them as an artifact
2. Run it once. See what the crash looks like.
3. Then decide between (a) fix in Harper, (b) fix in `lmdb-js`, (c) workaround in worker shutdown ordering.

That spike IS agent-suitable (small workflow file edit, no Windows debugging skills needed). Want me to file it as a sub-issue?

What this issue should NOT do

• Try to fix the crash blind without first capturing the signature
• Be labeled `claude-fix:*` until step 1 above produces a real stack trace

(🤖)

[heskew]

Filed #554 as the prerequisite diagnostic step. Once that workflow lands and produces a crash artifact, this issue can be picked up with a real stack trace to work from.

(🤖)

[heskew]

Update on the diagnostic step from #554 (now closed):

Downloaded the `harper-integration-api-test-logs-windows` artifact from PR #547's CI run and found that `restart_service { service: 'http_workers' }` actually ran successfully on Windows in that exact run (hdb.log lines 9687-9799). It was triggered from one of the other apiTests files in the suite (likely `17a_addComponents.mjs` or `27_headerTests.mjs` — the only ones still calling restart in their flow). Workers shut down, new workers started, schema-change events propagated, no crash.

So the crash Kris documented ("the 'restart_service' operation for 'http_workers' in 23_blob.mjs causes the HarperDB process to crash") is context-specific to the blob test, not to `restart_service` itself. The crash needs:

• A deployed blob-using component (per the test's setup steps just before the restart)
• Open file handles for blob storage
• Then `restart_service` on top of that state

Likely root cause refinement: the worker shutdown sequence in `restartWorkers()` (`server/threads/manageThreads.js:237`) tries to clean up the blob component's open file handles, and Windows file-locking semantics prevent the cleanup from completing — the file handle release sequence that works on Linux either fails or segfaults on Windows.

This is consistent with the `drop_schema` crash pattern from `7823d97a` — same class (Windows file-handle cleanup) but a different trigger.

Revised next step

Skip the diagnostic-workflow approach. Instead, push a spike branch that removes only the per-test restart_service skips in 23_blob.mjs (keeping the outer suite-level skip removal we already proved works at the file-write phase). The existing `run-integration-apiTests-windows` CI job will trigger, the blob context will crash, and the existing hdb.log artifact upload will capture the actual stack trace.

This uses zero new infrastructure. Branch can be closed without merge after the artifact is captured.

I'll pick this up when there's time — it's a single file edit + push.

(🤖)

[kriszyp]

Might be possible this got fixed with 6f7be5c ?

[heskew]

Crash signature captured

Spike branch (PR #556, will close without merge) caught it on `run-integration-apiTests-windows`. Downloaded the `harper-integration-api-test-logs-windows` artifact from CI run 25924041636.

When it crashes

The SECOND `restart_service { service: 'http_workers' }` in the suite crashes. The first one succeeds. This is the key signal — the operation itself isn't unconditionally broken; the crash is triggered by accumulated state between the two restarts.

Test flow:

Time	Event	Outcome
`14:54:17.495`	1st `Restarting http_workers`	✅ Succeeded; new http/14 worker emerges at 14:54:26
Between	`Create blob` / `Verify blob created in db` / `Verify blob created on filesystem` / `Read blob` / `Delete blob` / `Verify blob deleted from the filesystem`	LMDB blob writes / reads / deletes happen in the http worker
`14:55:21.332`	2nd `Restarting http_workers`	❌ Crash

Crash trace

```
2026-05-15T14:55:21.304Z [job/15] [notify]: Starting job: 970bb686-f52e-4047-b12c-ea293536ab9c
2026-05-15T14:55:21.332Z [main/0] [notify]: Restarting http_workers
2026-05-15T14:55:21.335Z [main/0] [debug]: []
2026-05-15T14:55:21.335Z [main/0] [info]: All root applications loaded
<HARD CRASH — no further log output>
```

The log ends mid-restart. No graceful shutdown, no error logged, log buffer not flushed. Process is dead within ~3ms of the last log line. By `14:56:33` the test runner is hitting `ECONNREFUSED ::1:9925` / `127.0.0.1:9925` — Harper's not listening. This matches Kris's commit message in `0751644f` exactly: "cascading ECONNREFUSED timeouts in subsequent tests."

Comparison to the successful first restart

Both restarts begin identically:

```
[main/0] [notify]: Restarting http_workers
[main/0] [debug]: []
[main/0] [info]: All root applications loaded
```

The first one continues with a "JavaScript execution has taken too long" warning ~8 seconds later, then the new http/14 worker emerges. The second one just stops. So the divergence is AFTER "All root applications loaded" — meaning the crash happens inside the worker-shutdown loop in `restartWorkers()` (`server/threads/manageThreads.js:237`), specifically in the region between `await loadRootComponents()` and the first worker's exit event.

Code-level localization

In `server/threads/manageThreads.js:restartWorkers()`, the relevant span is roughly lines 252-280:

```js
const { resetRestartNeeded } = require('../../components/requestRestart.ts');
resetRestartNeeded();
if (startReplacementThreads) {
const { loadRootComponents } = require('../loadRootComponents.js');
await loadRootComponents(); // ← logs "All root applications loaded"
}
module.exports.restartNumber++;
// ...
for (let worker of workers.slice(0)) {
if ((name && worker.name !== name) || worker.wasShutdown) continue;
harperLogger.trace('sending shutdown request to ', worker.threadId);
worker.postMessage({ ... SHUTDOWN ... }); // ← IPC to a worker with stale LMDB state
worker.wasShutdown = true;
worker.emit('shutdown', {}); // ← synchronous shutdown-event dispatch
// ...
}
```

The crash sits between "All root applications loaded" and the first `worker.emit('shutdown')` actually completing.

Likely root cause

Native cleanup of LMDB resources held by the http worker, on Windows, when the worker is asked to shut down WITH accumulated blob-write state.

Supporting evidence:

• The crash is state-dependent (1st restart fine, 2nd fails after blob ops)
• Hard crash with no Node-level error → native code segfault
• Kris's `7823d97a` commit notes the same class for `drop_schema` ("database drop crashes on windows")
• LMDB native module already known fragile on non-stable conditions (`7fe73a53` skipped `test:unit:lmdb` on Node v24 entirely)
• Kris is the author of `lmdb-js` and chose to skip rather than fix in two adjacent commits — strong signal the fix is in the native binding, not in Harper application code

Reproducer (for whoever picks this up)

Branch: `spike/blob-windows-crash-capture` (PR #556). Removes only the three Windows skips in `23_blob.mjs`. Run the `run-integration-apiTests-windows` CI job and the crash reproduces deterministically.

To get a stack trace beyond what hdb.log captures, would need either:

1. Native crash dumps enabled on the Windows runner (`%LOCALAPPDATA%\CrashDumps`)
2. `node --report-on-fatalerror` enabled in the Harper start command
3. Local Windows reproduction with a debugger attached

Fix candidates

Not implementing any here — this is diagnostic only. But the surface area to investigate:

• `server/threads/manageThreads.js:restartWorkers()` worker-shutdown sequence — could add explicit native-resource cleanup (LMDB envs, etc.) BEFORE `worker.postMessage(SHUTDOWN)` on Windows
• `lmdb-js` native binding cleanup on Windows worker thread exit (Kris's repo)
• Alternative: serialize the cleanup (close LMDB envs first, then exit worker) via a coordinated shutdown handshake

Status

PR #556 closing without merge. Spike served its purpose.

(🤖)

[heskew]

You're probably right, @kriszyp. Re-reading my own evidence with your hypothesis in mind, I overstated the "crash" conclusion.

The first `Restart Service` test actually passed in my spike (CI run 25924041636: 16.0s wall time, equivalent to 1s pre-sleep + restart + 15s post-sleep). The very next test (`Confirm Blob schema and table created`) failed instantly with ECONNREFUSED. So the symptom is "restart returned 200, test waited 15s, http workers still not back online" — which is exactly what your headers.test.mjs hit with the same fixed-15s sleep, just on a different test.

The hdb.log truncation I took as a crash signature is also consistent with a hung-but-alive Harper — main thread blocked in some long native syscall (LMDB cleanup, worker thread join, …), log buffer never flushes, but the process isn't actually dead. From the test's perspective, indistinguishable from a crash because the symptom is identical.

Your `restartHttpWorkers` with the readiness probe is the right discriminator:

• Harper recovers within 60s → test passes, issue was just timing on Windows, your approach fixes it
• Polling hits 60s deadline → clean error message proves there's a deeper recovery problem worth investigating

Either outcome is more informative than my fixed-sleep + ECONNREFUSED data point. Happy to re-spike using your helper once 23_blob.mjs is on the new framework (or I can pre-port it temporarily on a throwaway branch if useful before #555 lands).

(🤖)

[heskew]

Might be possible this got fixed with 6f7be5c ?

Saw that and thought it might, but let claude keep digging to see what it came up with. Going to have it stand-down on this one for now. :)