🧪 Deno HTTP journal teardown intermittently stalls concurrent verification

[taras]

Failure

At commit eb23322d3d73efc7054778a5e35f683507502085 on macOS/Deno 2.9.5, the exact committed-tree battery failed twice:

deno task verify

  FAILED  vendor     0s (exit 1)
  FAILED  lint       0s (exit 1)
  FAILED  check      0s (exit 1)
  FAILED  test       1200s (timed out after 20m)
  FAILED  check:jsr  0s (exit 1)
  FAILED  tsc        0s (exit 1)
  FAILED  test:node  0s (exit 1)
  FAILED  test:bun   0s (exit 1)
  FAILED  docs       0s (exit 1)

─── vendor ───
Task vendor:verify deno run --allow-read --allow-write=/tmp --allow-env --allow-run --cached-only --frozen scripts/verify-cloudflare-dofs.ts
verified Cloudflare Computer DOFS 63d363632e558f7e077794988d36ed75017c2a62: 104 recorded files; regenerated with TypeScript 5.9.3

re-run with:
  deno task vendor:verify

also failed: lint
  deno task lint

also failed: check
  deno task check

also failed: test
  deno task test

also failed: check:jsr
  deno task check:jsr

also failed: tsc
  pnpm exec tsc --project tsconfig.node.json --noEmit

also failed: test:node
  pnpm test:node

also failed: test:bun
  bun run test:bun

also failed: docs
  deno task xmd test packages/core/src --raw

The Deno leg is the deadline-triggering command. The remaining commands are reported as failed after the harness cancels the battery.

Intermittence evidence

The identical revision passed the standalone full Deno suite:

deno task test
ok | 419 passed (2904 steps) | 0 failed (7m58s)

During that standalone run, packages/core/tests/guarded-journal.test.ts stalled at the HTTP-backend case and completed at exactly 3m0s. An immediate isolated rerun on the identical revision passed:

deno task test packages/core/tests/guarded-journal.test.ts
ok | 1 passed (5 steps) | 0 failed (83ms)

The HTTP case itself completed in 31ms in isolation. The exact concurrent verifier then reproduced the 20-minute Deno-leg timeout a second time without source changes.

Investigation target

Determine whether DurableStreamTestServer.stop(), the HTTP durable-stream client, a response body, keep-alive connection, or another subprocess/resource prevents Deno test teardown under the repository's concurrent verification load. Preserve the 20-minute deadline and concurrent battery; do not exclude the test or weaken teardown.

[taras]

Investigation update on the same slice-6 revision lineage:

• The two original verifier timeout reports were produced inside a restricted process sandbox. scripts/verify.ts calls ps -A during every command's scope teardown so it can find and reap descendants. The sandbox denies ps. This explains why every completed command was reported as a zero-duration exit 1, even when its spool showed successful output; those runs were not valid host-level composability evidence.
• The guarded-journal suite was run under Deno, Node, and Bun concurrently. All three passed: Deno 97ms, Node 2.08s, Bun 346ms.
• packages/core/tests/guarded-journal.test.ts also completed during changed-test selection in 134ms, with its HTTP case in 58ms.
• The upstream DurableStreamTestServer.stop() waits on Node HTTP server.close() after ending SSE responses and cancelling waits. The test uses the server's fixed default port 4437. These are investigation targets for the observed intermittent 3-minute teardown, but the stall has not reproduced deterministically after the original observation, so no lifecycle implementation or teardown assertion was changed.
• A valid unsandboxed concurrent verifier then completed the Deno leg in 489.6s and exposed an unrelated Node TypeScript declaration error in the new Deno-only savepoint fault test. Commit ade199a2774b1f6d1aa6ca9007797bc849d8d593 fixes that test-only portability error without changing production behavior.
• From that clean commit, the exact host-level deno task verify passed all nine commands. The Deno leg completed in 726.4s, Node in 272.5s, and Bun in 398.6s; the battery reported that the tracked tree was unchanged.

The issue remains open per the repository flake rule: an identical revision exhibited the 3-minute HTTP teardown once and subsequently passed without a lifecycle fix.

[taras]

Root cause found, reproduced, and fixed.

Mechanism

The stall is in DurableStreamTestServer.stop() under Deno's node:http polyfill, and the resource that blocks it is a TCP connection Deno's own fetch client opened but never used.

1. During the test's request burst (PUT create → 3 serialized POST appends → GET readAll), Deno's fetch pool sometimes opens a spare connection that never carries a request. Server-side, the polyfill accepts it and marks it active-awaiting-headers.
2. At teardown, server.close() (which stop() wraps) runs the polyfill's closeIdleConnections() — but that only destroys sockets that have finished a response (!socket._httpMessage && socket._httpMessageDetached). The never-used connection has neither flag, so it is skipped. close() also destroys the 30-second connections-checking interval that would have expired the socket via headersTimeout. The 5s keepAliveTimeout never armed because no response ever finished on that socket.
3. Nothing server-side will ever close that socket now. close()'s callback waits for it; the only release is the client pool's eviction clock: hyper-util's pool evicts idle connections on a 90s interval (idle ≥ 90s), worst case just under 180s — the "exactly 3m0s" runs. When eviction doesn't come, the wait is unbounded — the ≥20m battery timeouts; one wedged deno test process on this machine was 3.5 days old.

In-process dump from a stalled teardown (probe copy of the test; the tracked socket's _httpMessageDetached is unset, and its active entry never saw headers):

{"connections":1,"listening":false,"trackedSockets":[{"local":"127.0.0.1:4437","remote":"127.0.0.1:54035","destroyed":false,"writableEnded":false,"httpMessage":false,"active":{"headersCompleted":false,"ageMs":5011,"reqUrl":null,"reqComplete":null}}]}

In the same stalled run, calling closeIdleConnections() by hand did not release close(); closeAllConnections() did, immediately. Externally, a stalled process holds both ends of one ESTABLISHED self-pair to 4437 with no LISTEN socket:

deno  97974  TCP 127.0.0.1:65050->127.0.0.1:4437 (ESTABLISHED)
deno  97974  TCP 127.0.0.1:4437->127.0.0.1:65050 (ESTABLISHED)

Why the concurrent battery makes it worse

Two multipliers, both from the fixed default port 4437:

• The verify battery runs the Deno, Node, and Bun suites concurrently, and all three run this test against 127.0.0.1:4437. On macOS the second bind does not fail — across 120 overlapping runs there was not one EADDRINUSE, and multiple processes held served-on-4437 connections simultaneously — so concurrent suites (and concurrent local sessions) can interconnect.
• Load widens the race that leaves the spare connection behind: under concurrent 3-runtime loops the Deno leg stalled 3 of 3 iterations at exactly 3m0s; solo it stalled 2 of 5, then 2 of 12 instrumented runs, matching the issue's intermittence.

Fix (PR: test-only, teardown not weakened)

• new DurableStreamTestServer({ port: 0 }) — ephemeral port; concurrent suites can no longer share or fight over a listener.
• The test's client sends Connection: close on every request via the fetch option of useHttpDurableStream, so no connection outlives its response and stop() has nothing to wait on. All three runtimes transmit the header (verified; Deno and Node close the socket immediately; Bun's close() doesn't wait on connections). Teardown still fully awaits server.stop(); the 20-minute deadline and concurrent battery are untouched.

Validation: fixed test 40/40 Deno runs at 0–1s with Node and Bun looping it concurrently (40/40 each), while a same-machine unfixed baseline kept stalling (6 of 10 runs at 180–182s). Full deno task verify on the branch: every command green, Deno leg 520.5s against the 20m deadline (test:node 254.9s, test:bun 370.1s). Fix in #411.

Upstream follow-ups (not filed)

• denoland/deno: node:http Server.close() can wait forever on an accepted connection that never carried a request — closeIdleConnections() skips it and close() destroys the reaper that would expire it. Node ≥19's close() does not exhibit this (the Node leg never stalled). Minimal repro available from this investigation.
• durable-streams: DurableStreamTestServer.stop() could call closeAllConnections() once close() is initiated (Node's own graceful-shutdown guidance); that would make every consumer's teardown robust regardless of client pooling.

[taras]

Upstream follow-ups filed after the downstream fix merged:

• Deno node:http compatibility bug: node:http Server.close() hangs on an accepted connection that sent no request denoland/deno#36513
• DurableStreamTestServer.stop() hardening: DurableStreamTestServer.stop() can hang on pre-request idle connections durable-streams/durable-streams#400

These reports preserve the minimal reproduction and shutdown-hardening request; no further downstream change is required beyond #411.