sleep() causes 'Unconsumed event in event log' when called concurrently via Promise.all

[desenmeng]

Bug Report

Description

When multiple tool calls execute in parallel via Promise.all (as DurableAgent does internally), concurrent sleep() calls cause event log corruption during replay, resulting in:

Unconsumed event in event log: eventType=wait_completed, correlationId=wait_xxx, eventId=evnt_xxx.
This indicates a corrupted or invalid event log.

Reproduction

1. Create a DurableAgent with tools that use sleep() for polling (e.g., polling an external API queue)
2. Ask the AI to call 5+ tools simultaneously
3. DurableAgent executes tools via Promise.all (line 189 of durable-agent.js)
4. Each tool calls sleep() concurrently
5. On replay, the event log has unconsumed wait_completed events

Minimal Example

export async function myWorkflow() {
  "use workflow";

  const agent = new DurableAgent({
    model: "anthropic/claude-sonnet-4.6",
    tools: {
      generateImage: tool({
        description: "Generate an image",
        inputSchema: z.object({ prompt: z.string() }),
        execute: async (params) => {
          // Submit to external queue
          const requestId = await submitStep(params.prompt);

          // Poll with sleep — breaks when multiple tools run concurrently
          for (let i = 0; i < 100; i++) {
            await sleep("5s");
            const status = await checkStatusStep(requestId, i);
            if (status === "COMPLETED") break;
          }

          return await getResultStep(requestId);
        },
      }),
    },
  });

  await agent.stream({ messages, writable: getWritable() });
}

When the LLM returns 5 generateImage tool calls, DurableAgent runs them all via Promise.all, creating 5 concurrent sleep() calls. On replay, the execution order of these concurrent sleeps may differ, causing event correlation mismatches.

Root Cause Analysis

sleep() uses order-based correlation IDs internally. With Promise.all, the order in which concurrent sleep() calls execute is non-deterministic. During replay, the runtime may match a wait_completed event to the wrong sleep() call because the execution order differs from the original run.

This contrasts with "use step" functions, which are identified by file + function + args (content-based), making them safe for concurrent use.

Suggested Fix

Add an optional token or name parameter to sleep() for content-based correlation:

// Current — order-based, breaks with Promise.all
await sleep("5s");

// Proposed — content-based, safe for concurrent use
await sleep("5s", { token: `poll-${requestId}-${iteration}` });

This would align with how createWebhook({ token }) and createHook({ token }) already support deterministic correlation via explicit tokens.

Environment

• workflow@4.1.0-beta.63
• @workflow/ai@4.0.1-beta.54
• @workflow/core@4.1.0-beta.63
• Platform: Vercel (Next.js)

Workaround

Currently working around this by avoiding sleep() in tools that may execute concurrently — using fal.subscribe() (blocking poll) for fast operations and planning to use createWebhook() for long-running ones.

[pranaygp]

@desenmeng Promise.all is not non-deterministic - so the correlation IDs for sleep should be consistent across replays even when multiple sleeps are being created in parallel. Explicit tokens won't help here (as your suggested fix)

I'll dig in and see if there's something else going wrong potentially

Can you please also confirm which version of workflow you're using and which world you're seeing this issue it (local/vercel/postgres?)

[pranaygp]

Independent Claude Investigation Results

I attempted to reproduce this issue and was unable to reproduce it. Concurrent sleep() calls via Promise.all appear to be safe in the current implementation. Here's the analysis:

Why concurrent sleep is safe

1. Deterministic ULID generation: The workflow VM uses a seeded PRNG (seedrandom) for Math.random(), and ULIDs are generated via monotonicFactory using this seeded RNG. The seed is derived from the run ID and startedAt timestamp, so every replay generates the exact same ULID sequence.

2. CorrelationId-based matching (not position-based): The events consumer matches events by correlationId — each sleep() subscriber checks event.correlationId !== correlationId. Even if wait_completed events arrive in a different order in the event log, the correct sleep gets resolved.

3. promiseQueue ensures deterministic resolution order: All promise resolutions are chained through ctx.promiseQueue, which processes in event log order. This means continuations (the code after await sleep()) always execute in a deterministic order, ensuring that subsequent ULID generations stay in sync.

Test verification

I wrote a comprehensive test that models the exact scenario from the issue (3 concurrent "tools" each running a sleep → step → sleep → step polling loop via Promise.all). The test verified:

• Same correlation IDs across replays: Running the workflow multiple times with the same event log produces identical ULID sequences
• Full loop completion: The workflow successfully replays through 2 iterations of sleep → checkStatus for all 3 concurrent tools without "unconsumed event" errors
• Event order doesn't matter: Even when wait_completed events arrive in reverse order, the downstream step correlation IDs are identical — the system is order-agnostic for event matching

Clarification on the root cause claim

The issue states that sleep() uses "order-based correlation IDs" — this isn't accurate. Sleep uses ULID-based correlation IDs from a deterministic seeded PRNG, with event matching by correlationId (not position). Promise.all is not non-deterministic in terms of which promises are created — the .map() call iterates synchronously, generating ULIDs in the same order every time.

Next steps

Could you help us narrow down the real cause? Specifically:

• Which world backend are you using? (local / vercel / postgres)
• Can you share the actual event log that shows the corruption? (the events leading up to the "unconsumed event" error)
• Is there any non-deterministic code in the workflow function itself? (e.g., accessing external state, using host Math.random() instead of the VM's, conditional logic based on wall-clock time, etc.)

[iNishant]

[Workflow] Workflow run failed with 1 uncommitted operation(s): step "step//@workflow/ai@4.1.0-beta.57//writeToolOutputToUI". Did you forget to await a step, hook, or sleep call? { workflowRunId: 'wrun_01KM8TWMC04ERJBRAET9ZBHMY5' }
[Workflow] Error while running workflow {
  workflowRunId: 'wrun_01KM8TWMC04ERJBRAET9ZBHMY5',
  errorName: 'WorkflowRuntimeError',
  errorStack: 'WorkflowRuntimeError: Unconsumed event in event log: eventType=hook_disposed, correlationId=hook_01KM8TWMCG11YH0MFY2Q04WYB3, eventId=evnt_01KM8V0B06TQNZ6X0DMDW5XDYN. This indicates a corrupted or invalid event log.\n' +
    '\n' +
    'Learn more: https://useworkflow.dev/err/corrupted-event-log\n' +
    '    at EventsConsumer.onUnconsumedEvent 

Got a similar error in case of parallel tool calls.

[pranaygp]

Ok following up here

• There was a race condition that allowed multiple wait_completed events to happen which corrupted the event log. This was already fixed across Materialize waits as entities to prevent duplicate wait_completed events #1057 and [world-postgres, world-local] Fix TOCTOU races in entity state transitions #1434 - so I think the initial issue is correct @desenmeng but the root cause was incorrect
• @iNishant from your message, I found a similar race condition existed that allowed multiple hook_disposed events to get posted simultaneously which causes the error you saw. I'm working on a fix rn in Fix race condition allowing duplicate hook_disposed events #1523