Skip to content

A hook_received while a step attempt is in flight re-dispatches the step; both attempts run concurrently in-process #2780

Description

@AndrewBarba

Summary

While a workflow is suspended awaiting a step, receiving a payload on an unrelated hook wakes the run, and the wake's replay re-dispatches the in-flight step under the same correlation id. The original attempt is still running in the same process and is not cancelled: both attempts execute the step body to completion, duplicating every side effect. Only one step_completed is recorded, so the duplication is invisible to the event log's completion accounting.

For steps whose side effects are externally visible — stream writes, resumeHook calls to other runs, third-party API calls — the duplicate is user-observable. In our case a duplicated resumeHook from a re-executed step also produced a resume landing on an already-disposed hook, corrupting the receiving run's replay (filed separately, see below).

At-least-once execution is expected for crash recovery. This report is about the live-process case: the runtime re-dispatches a step whose original attempt is demonstrably still running on the same event loop, and the trigger is any unrelated hook_received — routine traffic for hook-driven workflows.

Versions observed

  • @workflow/core@5.0.0-beta.26
  • @workflow/world-local@5.0.0-beta.22 (via createLocalWorld; in-process handler, WORKFLOW_TURBO=0)
  • Node v24

Reproduced 4/4 attempts with the minimal repro below.

Event log

One run; the step's side-effect log shows its body ran twice (starts=2):

step_started   corr=step_…CN0S  spikeSlowStep      ← attempt 1 (with input)
step_created   corr=step_…CN0S  spikeSlowStep
hook_received  corr=hook_…      (unrelated wake token)
step_started   corr=step_…CN0S  spikeSlowStep      ← attempt 2, same correlation id
step_completed corr=step_…CN0S  spikeSlowStep      ← one completion

Repro

import { appendFile } from "node:fs/promises";
import { createHook } from "workflow";

export async function spikeSlowStep(input: { logPath: string }) {
  "use step";
  await appendFile(input.logPath, "slow-step\n");           // side-effect marker
  await new Promise((resolve) => setTimeout(resolve, 1500));
  return "slow-done";
}

export async function wakeMidStepWorkflow(input: { token: string; logPath: string }) {
  "use workflow";
  const hook = createHook<{ n: number }>({ token: input.token });
  const iterator = hook[Symbol.asyncIterator]();
  const pending = iterator.next();                           // unrelated wake source

  await spikeSlowStep({ logPath: input.logPath });           // in flight when the wake lands

  const first = await pending;
  hook.dispose();
  return { received: first.done ? -1 : first.value.n };
}

Client:

const run = await start(wakeMidStepWorkflow, [{ token, logPath }]);
await waitForHookRegistered(token);
await sleep(400);                       // let the slow step start
await resumeHook(token, { n: 7 });      // unrelated wake, mid-step
await run.returnValue;                  // resolves fine: { received: 7 }
// logPath now contains "slow-step" twice

Expected

A step in created/started but not completed state whose attempt is still executing in the current process is recognized as pending during a wake's replay and not re-dispatched (or the superseded attempt is torn down before the new one starts, so at most one body's side effects land).

Actual

The step body executes twice concurrently in one process; all side effects flush from both attempts; one completion is recorded.

Impact / workaround

Any workflow that holds a hook open while running steps (the natural shape for cancellation, inbox, or control-channel patterns) gets duplicate step side effects under routine hook traffic. We had to (a) make cancellation-related step results pure markers, (b) move stream-emitting settle logic into a different run whose wake sources are quieter, and (c) add an in-process single-flight around the remaining emission step.

Related

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Fields

    No fields configured for issues without a type.

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions