world-local stream tail readers poll with a full readdir of the global chunks directory — O(world chunks) per tick per reader

[AndrewBarba]

Summary

world-local stream tail readers poll every 100 ms, and every poll lists the entire world-global streams/chunks/ directory (listChunkFilesForStream → fs.readdir(chunksDir) → prefix filter). Cost per tick is O(total chunk files in the world), per open reader — independent of the stream being read.

Any workload that holds tail readers open while the world accumulates chunks degrades superlinearly. The sharpest producer of long-lived readers is a serialized AbortSignal in step arguments (one tail reader per step invocation for the step's whole duration — see #2795): with 20 000 chunk files on disk, a trivial signal-bearing step goes from ~600 ms to 24–38 seconds per step, while the same step without a signal stays at 84 ms.

Real-world impact: a 100-turn single-session stress e2e (each turn writes ~15 chunk files to the shared directory) collapses — per-turn latency compounds from ~0.9 s until a workflow replay exceeds its 240 s timeout around turn 40. The identical fixture on the Vercel world (service-backed streams, no directory scans) is unaffected.

Versions observed

• @workflow/world-local@5.0.0-beta.22 (createLocalWorld, in-process handler, WORKFLOW_TURBO=0)
• @workflow/core@5.0.0-beta.26, Node v24; reproduced on macOS and Linux CI

Numbers

20 sequential trivial steps; "seeded" = 20 000 pre-created empty chunk files in streams/chunks/:

no signal,  clean world:  ~84ms/step   (total 4.0s)
no signal,  20k chunks:   ~84ms/step   (total 4.0s)   ← step path itself unaffected
signal,     clean world:  ~600ms/step  (total 13.0s)  ← fixed cost, see #2795
signal,     20k chunks:    600ms → 800ms → 1.4s → 24s → 26s → 38s per step (total 188s)

Note the degradation compounds within one run: each step's own stream writes add chunks that every concurrently-open reader re-lists on its next tick.

Repro

Use the workflow from #2795, pre-seed the chunks directory, and run with withSignal: true:

for (let i = 0; i < 20_000; i++) {
  await writeFile(join(dataDir, "streams", "chunks", `strm_seed${i}_user-chnk_SEED.bin`), "");
}

Code pointers

@workflow/world-local dist/streamer.js:

• listChunkFilesForStream() → listChunkEntries(chunksDir) → fs.readdir(chunksDir) — full directory listing, then prefix filtering per call.
• pollInterval = setInterval(async () => { ... listChunkFilesForStream(...) ... }, 100) — every open reader re-lists everything every 100 ms (the cross-process fallback path; it runs even when the in-process emitter is delivering).

Expected

Per-stream chunk lookup that doesn't scale with unrelated streams — e.g. shard chunks into per-stream subdirectories, keep a per-stream index, or skip filesystem polling when the in-process emitter is authoritative for the writer.

Actual

O(world-total chunks) filesystem work per 100 ms per open reader; worlds with long-lived sessions degrade until replay timeouts kill healthy runs.

Related

• Serializing an AbortSignal into a step adds ~0.5s fixed latency per step invocation on world-local #2795 — per-step AbortSignal fixed cost (the main producer of long-lived tail readers in our workload).
• Recreating a hook token after dispose() in the same run self-conflicts: suspension processing validates the new hook_created before the pending hook_disposed persists #2777/Hook token claim is not released in order with hook_disposed: a fast next-run claimant gets HookConflictError against an already-disposed hook #2778/A hook_received while a step attempt is in flight re-dispatches the step; both attempts run concurrently in-process #2780/resumeHook racing hook disposal is journaled after hook_disposed; the owner's replay then diverges into CorruptedEventLogError #2781 — same downstream investigation (turn cancellation).