Fix concurrency and event emission across storage and task systems

[sroussey]

Summary

This PR addresses several concurrency, event emission, and error handling issues across the storage, task-graph, and worker systems. Key fixes include proper cross-instance connection guarding in DuckDB storage, vector similarity search event emission, task graph dependency validation, and worker error rehydration.

Key Changes

Storage & Concurrency

• DuckDbTabularStorage: Added connectionHandle() and guardedWrite() methods to properly serialize writes across multiple storage instances sharing the same database connection. This prevents interleaving of BEGIN/COMMIT blocks when sibling instances call putBulk() concurrently.
• DuckDbTabularStorage: Refactored withTransaction() to use runInTransactionOnConnection() for proper cross-instance transaction coordination.
• BaseTabularStorage: Replaced the two literal NUL bytes inside the index-dedup join separators with the unicode escape sequence (backslash-u0000). Runtime behavior is byte-identical — the change makes the file valid UTF-8 text again so grep/ripgrep stop classifying it as binary and silently skipping it in symbol searches.

Vector Storage Events

• SqliteAiVectorStorage, SqliteVectorStorage, PostgresVectorStorage, SupabaseVectorStorage: Added emitSimilaritySearch() helper method to emit the similaritySearch event on all vector search results (previously only the InMemory and IndexedDb backends emitted the event declared on IVectorStorage). The inherited events emitter is widened to carry vector-specific events.

Task Graph & Dependencies

• TaskGraph: run() now spreads the full run config instead of hand-copying a field whitelist — enforceEntitlements and matchAllEmptyInputs were silently dropped before.
• TaskJSON: Moved dependency validation and Dataflow creation from JsonTask into createGraphFromDependencyJSON() so that all graph construction (including recursed subtasks) properly wires dataflows and validates that dependency IDs exist in the graph.
• JsonTask: Removed duplicate dependency wiring logic now handled by the factory function.

Task Execution & Validation

• IteratorTask: Changed executeStream() to throw immediately if called (instead of yielding its input as the finish payload), since subclass output schemas exclude x-stream and runs should never route here. Failing loudly prevents silent wrong results.
• ReduceTask: Strip x-stream annotation from child task output schemas when building the aggregate schema, ensuring the reduce output is never treated as a live stream (previously a reduce whose body ended in a streaming task returned its input without running an iteration).
• AiChatWithKbTask: Delegated getJobInput() to base class to ensure timeoutMs, outputSchema, and future base fields are always populated. Added explicit gateOrThrow() call in executeStream() since the override doesn't call super.executeStream().
• Vector tasks (Distance, Divide, DotProduct, Multiply, Subtract, Sum): Changed generic Error throws to TaskInvalidInputError for proper error classification.
• DateFormatTask: Invalid-date throw now uses TaskInvalidInputError for consistent error classification.

Worker Error Handling

• WorkerManager: Extracted error rehydration logic into new rehydrateWorkerError() function in scrubStack.ts. All three error message handlers now use this function for consistent stack scrubbing and error reconstruction — previously the stream and run-fn paths skipped the stack scrubbing the call path applied, leaking absolute container paths.
• scrubStack: Added rehydrateWorkerError() function that rebuilds an Error from worker message payloads with defense-in-depth stack scrubbing.

Event Emission Safety

• IndexedDbTabularStorage: Replaced direct this.events.emit() calls with safeEmit() — a throwing subscriber inside an IDB onsuccess callback previously unwound before resolve(), leaving put()'s promise permanently unsettled.

Job Queue

• Queue storages (InMemory, IndexedDb, Postgres, Sqlite, Supabase): add() no longer clobbers a caller-set future visible_at, so send(..., { delaySeconds }) actually delays delivery instead of being a silent no-op.
• JobQueueClient: the job_error client event now forwards the errorCode the server already carries.
• genericJobQueueTests: Added assertion that deferred visibility is preserved in stored job rows (not clobbered to "now" on add).

Package Configuration

• @workglow/javascript: Updated sideEffects to ["./dist/task.js"] (was false) to preserve task registration side effects, and added registerJavaScriptTasks() so serialized graphs containing a JavaScriptTask node can be rehydrated.
• @workglow/mcp: McpListTask now declares the optional CREDENTIAL entitlement on the HTTP branch, matching its three siblings.
• @workglow/util: the ./worker export's default condition now pairs worker-node.d.ts with worker-node.js (types previously pointed at worker-entry.d.ts, hiding the Worker.node exports from TypeScript), and the tsconfig now emits the worker-node/browser/bun.d.ts declarations the exports map names — the browser/bun types conditions previously pointed at files that were never built.
• All packages: Added "license": "Apache-2.0" field to package.json files for compliance.

Documentation

• EXECUTION_MODEL.md: Clarified that ABORTING state transitions to FAILED when the abort surfaces as an error.
• TaskJSON JSDoc: Updated return type documentation to clarify that dataflows are wired for all dependencies including in recursed subtasks.

https://claude.ai/code/session_01TQFDxK7AKh4w4DCjRJUpQm

[github-actions]

Coverage Report

Status	Category	Percentage	Covered / Total
🔵	Lines	62.33%	28173 / 45199
🔵	Statements	62.2%	29212 / 46958
🔵	Functions	62.55%	5371 / 8586
🔵	Branches	51.39%	13924 / 27094

File Coverage

File	Stmts	Branches	Functions	Lines	Uncovered Lines
Changed Files
packages/storage/src/tabular/BaseTabularStorage.ts	18.55%	7.35%	10.14%	20.37%	80-119, 204, 206, 224, 227-229, 233, 245-252, 260, 262-272, 286, 289, 299-300, 306-309, 313-319, 328-329, 332, 337-914, 935-940, 960-969, 974-975, 978-979, 1002-1048, 1056-1144

Generated in workflow #2866 for commit 69eed2b by the Vitest Coverage Report Action

[sroussey]

Code review (xhigh, --fix) — 10 finder angles + 5 adversarial verifiers. Fixes pushed in fc3c29a.

Bugs confirmed and fixed:

• TaskGraph.run forwarded outputCache: config?.outputCache || this.outputCache — an explicit outputCache: false (the documented "disable all caching" signal TaskGraphRunner branches on) was swallowed and the graph's repo substituted. Now ??, with a regression test proving a second run re-executes under false.
• DuckDbTabularStorage.guardedWrite's !this.inTransaction short-circuit was a hole: a write issued during the instance's own open transaction queued only on the instance mutex, woke after COMMIT, and could run on the shared connection with no chain slot — free to interleave inside a sibling instance's next BEGIN/COMMIT. The chain is now always consulted (runOnConnection inlines genuine tx descendants and queues unrelated calls; verified deadlock-free since the tx callback gets the proxy). Added a deterministic scripted-stub test asserting statement-level ordering.
• IndexedDbQueueStorage.add() unconditionally recomputed fingerprint, silently replacing a caller-supplied one (the other four backends use ?? makeFingerprint) — custom-fingerprint dedup could never match on IDB. Pre-existing, adjacent to this PR's visible_at hunk. Fixed + pinned in the generic suite.
• The new delayed-send assertion compared visibleAt - createdAt >= 150, coupling two timestamps taken at different points with 50 ms of headroom — flaky under CI load. Rewritten as a [t0 + 200, t1 + 200] bracket on the client clock.
• WorkerManager logged the raw worker payload (including the unscrubbed data.stack the scrubbing commit exists to remove) at the run-path debug site and the preview-path warn; both now log the rehydrated error.
• AiChatWithKbTask.getJobInput's spread embedded sessionId inside the serialized taskInput; it's now set on the job input after delegating to the base, staying a top-level field only.

Quality: the six identical emitSimilaritySearch blocks (4 added here + InMemory/IndexedDb inline copies) collapsed into one shared helper in @workglow/storage typed against the exported VectorEventListeners; examples/web's now-redundant dependency re-wiring deleted (verified harmless — addEdge dedupes on Dataflow.id — but dead); util tsconfig formatting churn reverted. Added TaskJSON tests covering the new nested-subtask wiring (top-level, nested, out-of-scope id throw, toDependencyJSON round-trip — the round-trip silently lost every subgraph edge before this PR).

Investigated and cleared: WhileTask does not have ReduceTask's x-stream leak (it overrides executeStream with the full condition/maxIterations loop); the nested-dependency validation throw is intentional strictness (anything that now throws was already running mis-wired — worth a line in the PR description); the run() spread newly activating enforceEntitlements/matchAllEmptyInputs is the fix working as described.

Verified: build:packages clean, tsc -b clean, storage/queue/graph/task/util/ai vitest sections green (section re-run in flight at push time; will follow up if CI disagrees).

---

Generated by Claude Code