Skip to content

feat(testing): invoker state and checkpoint stamping#515

Draft
yaythomas wants to merge 1 commit into
mainfrom
pr1/invoker-state-checkpoint-stamping
Draft

feat(testing): invoker state and checkpoint stamping#515
yaythomas wants to merge 1 commit into
mainfrom
pr1/invoker-state-checkpoint-stamping

Conversation

@yaythomas

@yaythomas yaythomas commented Jul 7, 2026

Copy link
Copy Markdown
Contributor

What

Align the local test runner's data-plane responses with the real service, validated against the JS example suite as a cross-language conformance check.

Changes

Invoker state and checkpoint stamping:

  • Track invocation state (PRE_INVOKE/INVOKING/COMPLETED) and gate checkpoints so at most one handler invocation is in flight per execution.
  • Record per-update timestamps and stamp checkpoint operations with a monotonic sequence so delta computation and pagination share one snapshot.
  • Offload handler invocation off the scheduler event loop so concurrent map/parallel branches no longer serialize on a single thread.
  • Return a locked snapshot from in-process state reads.

History generation (GetDurableExecutionHistory):

  • Emit one history event per checkpoint update so step retries report the correct StepStarted/StepFailed counts and attempt numbers.
  • Reconstruct events from recorded update transitions using the real operation for details, with a fallback for operations completed outside the checkpoint path.

Fidelity fixes:

  • Store the RETRY payload as the step result so waitForCondition state replays, and map a RETRY without error to StepSucceeded.
  • Set callback timeout errorType to Callback.Timeout/Callback.Heartbeat with the correct message strings.
  • Introduce CheckpointCore.match_cached as the single source of truth for idempotent checkpoint replay detection (both web and in-process paths delegate to it).

Invocation timeout:

  • Add execution_timeout and invocation_timeout to the runner constructor and CLI. A timed-out invocation leaves the step STARTED and replays.

Testing

  • 1311 passed, 95.67% coverage (gate 95%)
  • Typecheck clean (102 files)
  • Format clean
  • Cross-language conformance: JS example suite failures reduced from 27 to 7 (remaining are CHAINED_INVOKE feature gap and one upstream JS SDK bug)

Stack

This is PR 1 of 2. PR 2 (per-execution worker serialization #516) builds on top.

Closes #464, #460, #462, #438, #439

@@ -33,7 +33,10 @@


class ValidActionsByOperationTypeValidator:

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

ooof, this belongs elsewhere.

@yaythomas yaythomas force-pushed the pr1/invoker-state-checkpoint-stamping branch from 99be1b4 to 5017a88 Compare July 7, 2026 18:39
@@ -1,66 +0,0 @@
"""Validator for valid actions by operation type."""

from __future__ import annotations

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

moved to CheckpointValidator

Align the local test runner's data-plane responses with the deployed
service, validated against the JS example suite as a cross-language
conformance check (failures reduced from 27 to 7; the remaining 7 are
the unimplemented CHAINED_INVOKE operation and an upstream JS SDK
replay bug).

Invoker state and checkpoint stamping:
- Track invocation state (PRE_INVOKE/INVOKING/COMPLETED) and gate
  checkpoints so at most one handler invocation is in flight per
  execution
- Record per-update timestamps and stamp checkpoint operations with a
  monotonic sequence so delta computation and pagination share one
  snapshot
- Offload handler invocation off the scheduler event loop so concurrent
  map/parallel branches no longer serialize on a single thread
- Return a locked snapshot from in-process state reads

History generation (GetDurableExecutionHistory):
- Emit one history event per checkpoint update so step retries report
  the correct StepStarted/StepFailed counts and attempt numbers, instead
  of one pair per operation
- Reconstruct events from recorded update transitions using the real
  operation for details, with a fallback for operations completed
  outside the checkpoint path

Fidelity fixes:
- Serialize Error and ReplayChildren in ContextDetails so child-context
  failures reconstruct the correct error class on replay
- Store the RETRY payload as the step result so waitForCondition state
  replays, and map a RETRY without error to StepSucceeded
- Set callback timeout errorType to Callback.Timeout/Callback.Heartbeat
  with the expected message strings

Invocation timeout:
- Add execution_timeout and invocation_timeout to the runner
  constructor and CLI to simulate the Lambda function Timeout; a
  timed-out invocation leaves the step STARTED and replays

Closes #464
Closes #460
Closes #462
Closes #438
Closes #439
@yaythomas yaythomas force-pushed the pr1/invoker-state-checkpoint-stamping branch from 5017a88 to 11eebdc Compare July 7, 2026 19:25
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

[Testing] Add single instance execution control

1 participant