Add durable-execution support to Lambda Test Tool v2 (preview)#2495
Closed
GarrettBeatty wants to merge 6 commits into
Closed
Add durable-execution support to Lambda Test Tool v2 (preview)#2495GarrettBeatty wants to merge 6 commits into
GarrettBeatty wants to merge 6 commits into
Conversation
Adds a local durable-execution service emulator to the Test Tool, enabled with --durable-execution. A durable function whose AWS_ENDPOINT_URL_LAMBDA is pointed at the Test Tool can now run a full workflow locally. Phase 1 (data plane): CheckpointDurableExecution and GetDurableExecutionState endpoints hosted alongside the Runtime API, backed by the operation store + checkpoint processor ported from Amazon.Lambda.DurableExecution.Testing and retyped to plain System.Text.Json request DTOs (the AWSSDK model types use ConstantClass and can't be STJ-deserialized). Response timestamps are emitted as unix seconds to match the AWSSDK rest-json unmarshaller. Phase 2 (control plane): invoking with X-Amz-Durable-Execution-Name starts an execution that DurableExecutionDriver drives to completion, re-invoking the function over the Runtime API across the replay cycle. Timers are time-skipped by default (--durable-time-skip). Chained durable invokes and external callbacks are not yet supported and fail fast with a clear message. Includes end-to-end tests exercising both the data plane and the full start-hook/drive-loop cycle through a real AmazonLambdaClient and Runtime API.
Callbacks: SendDurableExecutionCallback{Success,Failure,Heartbeat} endpoints
resolve a pending CALLBACK operation by its minted id and wake the parked
drive loop. The driver now parks on a pending callback (instead of failing)
and resumes when the callback is resolved; park/resume is lock-guarded so a
resolve that races ahead of the park is not lost.
Web UI: a new Durable Execution page lists local executions, renders each
one's operation timeline (id/type/status/detail), and can send a callback
from the browser. The page shows a friendly hint when --durable-execution is
not enabled.
Includes an end-to-end test covering the full start -> park-on-callback ->
SendDurableExecutionCallbackSuccess -> resume -> Succeeded cycle.
…ator (Phase 4) Chained durable invokes: a CHAINED_INVOKE operation is now resolved by starting the target function as a nested durable execution and stamping its result (or error) back onto the parent's operation, so ctx.InvokeAsync works locally. The sibling runs the same drive loop under its own function name. StopDurableExecution: the /stop endpoint cancels the execution's drive loop and marks the EXECUTION operation STOPPED, returning a StopTimestamp. Payload caps: checkpoint updates whose operation payload exceeds the service limits (256 KB general, 1 MB for chained invokes) are rejected with 413. Also fixes a latent bug: the drive loop awaited EventContainer.WaitForCompletion on a threadpool thread, a blocking 15-minute wait that ignored cancellation, so Stop could not interrupt an in-flight invocation. Replaced with a cancellation-aware status poll. The durable web-host tests now run in a non-parallel xUnit collection to avoid starving their Runtime-API poll loops under CPU pressure. Full Test Tool unit suite passes (207 passed, 1 pre-existing skip).
The callback-id and result inputs now bind on oninput (not the default onchange) so a value is committed as the user types, and the Send Success button is no longer gated on a disabled attribute that consumed the first click to commit the field. Verified end to end in a real browser (Playwright): starting a durable workflow, parking on a callback, clicking Send Callback to prefill the id, and clicking Send Success resumes the execution to Succeeded.
Adds a 'Start as durable execution' checkbox (with optional execution name) to the Function Tester page so users can start a durable workflow from the UI without manually setting the X-Amz-Durable-Execution-Name header. The checkbox only appears when the tool is started with --durable-execution. Fixes Re-Invoke for durable functions: the queued/executed events for a durable function are internal replay envelopes, so re-posting one was a no-op replay. Re-Invoke now detects a durable envelope, extracts the original user payload from the EXECUTION operation, and starts a fresh durable execution. Adds a durable-execution sample request and documents the feature in the README (Durable Execution Mode) and the in-app Documentation page. Verified end to end in a browser (Playwright): the checkbox starts an execution that appears on the Durable Execution page, and Re-Invoke creates a new execution.
Fixes --durable-time-skip effectively defaulting to false: as a plain bool Spectre.Console.Cli option it bound presence->true/absence->false regardless of the C# field initializer, so timers were never skipped even though the default was documented as true. Now uses [DefaultValue(true)] and accepts an explicit value (--durable-time-skip false). Adds a live 'Time-skip timers' toggle to the Durable Execution page. Time-skip is now read on each checkpoint (via a provider) rather than captured at store construction, so it can be flipped at runtime and applies to subsequent checkpoints across all executions. Fixes the timeline showing a completed WAIT as STARTED: when time-skip is off and a timer elapses, the SDK proceeds without emitting a SUCCEEDED checkpoint, leaving the op STARTED forever. The driver now stamps elapsed WAITs as SUCCEEDED before re-invoking (mirroring the service firing the timer), so the store and UI reflect that the wait completed. Verified end to end in a browser (Playwright): time-skip defaults on (execution parks in ~1s), the WAIT shows SUCCEEDED, the UI toggle flips interactively, and the checkbox-start and callback flows still work.
GarrettBeatty
force-pushed
the
feature/testtool-durable-execution
branch
from
July 23, 2026 01:46
86c38b6 to
2d4df13
Compare
Contributor
Author
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What this adds
Durable-execution support in the Lambda Test Tool v2 — run a durable Lambda workflow locally, end to end, with a web UI.
Enable with
--durable-execution. The tool emulates the durable-execution service: it stores checkpoints, drives the workflow across its replay cycle, resolves timers, and manages callbacks. Point the function'sAWS_ENDPOINT_URL_LAMBDAat the tool to use it.Commits (each independently reviewable)
CheckpointDurableExecution/GetDurableExecutionStateendpoints) + the re-invocation driver that starts an execution from theX-Amz-Durable-Execution-Nameheader and drives it across the replay cycle (steps, waits, retries, child contexts, parallel/map).SendDurableExecutionCallback{Success,Failure,Heartbeat}) with park/resume, plus a Durable Execution web page (execution list, operation timeline, Send Callback).StopDurableExecution, payload-size caps.--durable-time-skipdefault (was effectively false), completed WAITs showing SUCCEEDED, Send Callback interactivity.Design
See
Libraries/src/Amazon.Lambda.DurableExecution/docs/design/testtool-integration-plan.md.Verification
Backend covered by the Test Tool xUnit suite (data plane + full start/drive/callback/chained-invoke cycle). The UI flows (checkbox start, Re-Invoke, callbacks, time-skip toggle) were verified end to end in a real browser against the sample durable function.
Not yet