Add durable-execution support to Lambda Test Tool v2 (on shared kernel)

.autover/changes/50f64d33-d1b0-445c-aac3-b2c95ae10ff6.json

@@ -0,0 +1,16 @@
+{
+  "Projects": [
+    {
+      "Name": "Amazon.Lambda.TestTool",
+      "Type": "Minor",
+      "ChangelogMessages": [
+        "Add durable-execution service emulator (preview): with --durable-execution, the Test Tool hosts CheckpointDurableExecution and GetDurableExecutionState endpoints alongside the Lambda Runtime API so durable workflows can checkpoint and read state locally. Point the function's AWS_ENDPOINT_URL_LAMBDA at the Test Tool to use it. Timers are time-skipped by default (--durable-time-skip).",
+        "Support running a durable workflow locally end to end: invoking a function with the X-Amz-Durable-Execution-Name header starts a durable execution that the Test Tool drives to completion, re-invoking the function across the replay cycle (steps, waits, retries, child contexts, parallel/map). Chained durable-to-durable invokes are not yet supported.",
+        "Add external callback support and a Durable Execution web UI: SendDurableExecutionCallbackSuccess/Failure/Heartbeat endpoints resume a workflow parked on a callback, and a new Durable Execution page lists executions, shows each operation timeline, and can send a callback from the browser.",
+        "Complete local durable workflow support: chained durable-to-durable invokes (ctx.InvokeAsync) now run the target function as a nested execution and return its result to the caller; StopDurableExecution halts a running execution; and checkpoint payloads exceeding the service size limits are rejected.",
+        "Start durable executions from the web UI: the Function Tester page adds a 'Start as durable execution' checkbox (with an optional execution name) so a durable workflow can be started without setting the X-Amz-Durable-Execution-Name header manually. Re-Invoke on a durable function now starts a fresh execution from the original user payload instead of replaying an internal envelope. Adds a durable-execution sample request and documents the feature in the README and in-app documentation.",
+        "Fix --durable-time-skip defaulting to false (timers were never skipped); it now defaults to true and accepts an explicit value (--durable-time-skip false). Add a live 'Time-skip timers' toggle to the Durable Execution page. Completed WAIT operations now show SUCCEEDED in the timeline instead of remaining STARTED after their timer elapses."
+      ]
+    }
+  ]
+}

.autover/changes/b7e2f4a1-9c33-4d15-8e6b-1f0a2c9d7e42.json

@@ -0,0 +1,11 @@
+{
+  "Projects": [
+    {
+      "Name": "Amazon.Lambda.TestTool",
+      "Type": "Patch",
+      "ChangelogMessages": [
+        "Consume the shared Amazon.Lambda.DurableExecution.LocalEmulation checkpoint state machine and operation store in the durable-execution emulator instead of a private fork, so the emulator and the durable-execution testing package cannot drift. No user-facing behavior change."
+      ]
+    }
+  ]
+}

Tools/LambdaTestTool-v2/README.md

@@ -78,6 +78,33 @@ Use this mode when you want to test Lambda functions directly without API Gatewa
 dotnet lambda-test-tool start --lambda-emulator-port 5050
 ```
 
+### Durable Execution Mode (preview)
+Use this mode to run [durable Lambda functions](../../Libraries/src/Amazon.Lambda.DurableExecution) locally. The Test Tool emulates the durable-execution service: it stores checkpoints, drives the workflow across the replay cycle, resolves timers, and manages callbacks — so you can run a full multi-step workflow (and attach a debugger) without deploying to AWS.
+
+Enable it with `--durable-execution`:
+
+```
+# Start the Lambda emulator with the durable-execution service emulator
+dotnet lambda-test-tool start --lambda-emulator-port 5050 --durable-execution
+```
+
+By default, timers (`WaitAsync` and retry backoff) are **time-skipped** so workflows advance immediately. Pass `--durable-time-skip false` to wait real wall-clock time instead.
+
+**Point your function at the emulator.** Run your durable function as a normal local process (`dotnet run`) with these environment variables set:
+
+| Variable | Value | Why |
+|---|---|---|
+| `AWS_LAMBDA_RUNTIME_API` | `localhost:5050/<FunctionName>` | Runtime API endpoint. **Must include the `/<FunctionName>` suffix** — the tool partitions events by that path segment, and it must match the function name you invoke with. |
+| `AWS_ENDPOINT_URL_LAMBDA` | `http://localhost:5050` | Redirects the durable checkpoint/state/callback calls to the Test Tool. |
+| `AWS_REGION` | e.g. `us-east-1` | Required for request signing. |
+| `AWS_ACCESS_KEY_ID` / `AWS_SECRET_ACCESS_KEY` | any dummy value | Required for signing; the emulator ignores the signature. |
+
+**Start a durable execution.** In the Test Tool web UI, select your function, check **Start as durable execution** (optionally name it), and click **Invoke**. Then open the **Durable Execution** page to watch the operation timeline (steps, waits, callbacks) as the workflow runs.
+
+**Callbacks.** When a workflow parks on `WaitForCallbackAsync`, the Durable Execution page shows a **Send Callback** action to resolve it and resume the workflow. Callbacks can also be sent with the AWS SDK's `SendDurableExecutionCallbackSuccess`/`Failure`/`Heartbeat` against the tool's endpoint.
+
+**Current limitations (preview):** chained durable-to-durable invokes run each target as a nested execution against a sibling function that must also be running locally; large-payload paging fidelity is approximate.
+
 ### API Gateway Emulator Mode
 Use this mode when you want to test Lambda functions through API Gateway endpoints. **Note: Running this mode by itself will not work, you will still need have the lambda runtime client running elsewhere and reference it in the `Endpoint` parameter in the `APIGATEWAY_EMULATOR_ROUTE_CONFIG` env varible (see below [Required Configuration](#required-configuration))** Api gateway mode requires additional configuration through environment variables.
 

Tools/LambdaTestTool-v2/src/Amazon.Lambda.TestTool/Amazon.Lambda.TestTool.csproj

@@ -40,6 +40,20 @@
     <PackageReference Include="BlazorMonaco" Version="3.2.0" />
   </ItemGroup>
 
+  <!-- Durable-execution support is referenced from in-repo source (like RuntimeSupport below)
+       rather than as a published package:
+       - Amazon.Lambda.DurableExecution supplies the public wire model (Operation/ErrorObject/
+         OperationTypes/…) the durable-service emulator reuses so its checkpoint responses are
+         byte-for-byte the envelope the SDK expects.
+       - Amazon.Lambda.DurableExecution.LocalEmulation is the shared in-memory checkpoint state
+         machine (InMemoryOperationStore/CheckpointProcessor), used via InternalsVisibleTo so the
+         emulator and the .Testing package cannot drift. A source reference (not a preview package
+         pin) keeps a single Amazon.Lambda.DurableExecution assembly identity across both. -->
+  <ItemGroup>
+    <ProjectReference Include="..\..\..\..\Libraries\src\Amazon.Lambda.DurableExecution\Amazon.Lambda.DurableExecution.csproj" />
+    <ProjectReference Include="..\..\..\..\Libraries\src\Amazon.Lambda.DurableExecution.LocalEmulation\Amazon.Lambda.DurableExecution.LocalEmulation.csproj" />
+  </ItemGroup>
+
   <Target Name="GetRuntimeSupportTargetFrameworks">
     <Exec Command="dotnet msbuild ../../../../Libraries/src/Amazon.Lambda.RuntimeSupport/Amazon.Lambda.RuntimeSupport.csproj --getProperty:TargetFrameworks" ConsoleToMSBuild="true">
       <Output TaskParameter="ConsoleOutput" PropertyName="RuntimeSupportTargetFrameworks" />

Tools/LambdaTestTool-v2/src/Amazon.Lambda.TestTool/Commands/Settings/RunCommandSettings.cs

@@ -97,6 +97,33 @@ public sealed class RunCommandSettings : CommandSettings
     [Description("The absolute path used to save global settings and saved requests. You will need to specify a path in order to enable saving global settings and requests.")]
     public string? ConfigStoragePath { get; set; }
 
+    /// <summary>
+    /// Enables the durable-execution service emulator: HTTP endpoints emulating the Lambda
+    /// durable-execution data plane (checkpoint / get-state) are hosted alongside the Runtime
+    /// API. Point the function's <c>AWS_ENDPOINT_URL_LAMBDA</c> at the emulator to run a durable
+    /// workflow locally.
+    /// </summary>
+    [CommandOption("--durable-execution")]
+    [Description("Enable the durable-execution service emulator (checkpoint/get-state endpoints) hosted alongside the Lambda Runtime API.")]
+    public bool DurableExecution { get; set; }
+
+    /// <summary>
+    /// When the durable-execution emulator is enabled, resolves timers (WaitAsync, retry
+    /// backoff) immediately instead of waiting for wall-clock time. Defaults to <c>true</c> so
+    /// local workflows advance without real delays. Use <c>--durable-time-skip false</c> to wait
+    /// real wall-clock time.
+    /// </summary>
+    /// <remarks>
+    /// Declared as a nullable bool with an explicit <see cref="DefaultValueAttribute"/> so the
+    /// Spectre.Console.Cli binder honors the "default true" behavior. A plain <c>bool</c> option
+    /// binds presence→true / absence→false regardless of a C# field initializer, which would make
+    /// the default effectively <c>false</c> and prevent <c>--durable-time-skip false</c> from parsing.
+    /// </remarks>
+    [CommandOption("--durable-time-skip <TRUE_OR_FALSE>")]
+    [Description("When durable execution is enabled, resolve timers/retry backoff immediately rather than waiting for wall-clock time. Default: true.")]
+    [DefaultValue(true)]
+    public bool DurableTimeSkip { get; set; } = true;
+
     /// <summary>
     /// Validate that <see cref="ConfigStoragePath"/> is an absolute path.
     /// </summary>

Tools/LambdaTestTool-v2/src/Amazon.Lambda.TestTool/Components/Layout/MainLayout.razor

@@ -25,6 +25,14 @@
                         Function Tester
                     </NavLink>
                 </li>
+                <li class="nav-item">
+                    <NavLink class="nav-link d-flex align-items-center gap-1" href="/durable-execution">
+                        <svg xmlns="http://www.w3.org/2000/svg" width="18" height="18" fill="currentColor" class="bi bi-diagram-3-fill" viewBox="0 0 16 16">
+                            <path fill-rule="evenodd" d="M6 3.5A1.5 1.5 0 0 1 7.5 2h1A1.5 1.5 0 0 1 10 3.5v1A1.5 1.5 0 0 1 8.5 6v1H14a.5.5 0 0 1 .5.5v1a.5.5 0 0 1-1 0V8h-5v.5a.5.5 0 0 1-1 0V8h-5v.5a.5.5 0 0 1-1 0v-1A.5.5 0 0 1 2 7h5.5V6A1.5 1.5 0 0 1 6 4.5zm-3 8A1.5 1.5 0 0 1 4.5 10h1A1.5 1.5 0 0 1 7 11.5v1A1.5 1.5 0 0 1 5.5 14h-1A1.5 1.5 0 0 1 3 12.5zm6 0a1.5 1.5 0 0 1 1.5-1.5h1a1.5 1.5 0 0 1 1.5 1.5v1a1.5 1.5 0 0 1-1.5 1.5h-1A1.5 1.5 0 0 1 9 12.5z"/>
+                        </svg>
+                        Durable Execution
+                    </NavLink>
+                </li>
                 <li class="nav-item">
                     <NavLink class="nav-link d-flex align-items-center gap-1" href="/documentation">
                         <svg xmlns="http://www.w3.org/2000/svg" width="18" height="18" fill="currentColor" class="bi bi-file-text-fill" viewBox="0 0 16 16">

Tools/LambdaTestTool-v2/src/Amazon.Lambda.TestTool/Components/Pages/Documentation.razor

@@ -11,3 +11,25 @@
     Please see the README file <a href="https://github.com/aws/aws-lambda-dotnet/blob/master/Tools/LambdaTestTool-v2/README.md">here</a> for more information.
 </p>
 
+<h3 style="padding: 0 15px">Durable Execution (preview)</h3>
+<div style="padding: 0 15px">
+    <p>
+        Run <a href="https://github.com/aws/aws-lambda-dotnet/tree/master/Libraries/src/Amazon.Lambda.DurableExecution">durable Lambda functions</a>
+        locally. Start the tool with <code>--durable-execution</code> and the tool emulates the durable-execution
+        service: it stores checkpoints, drives the workflow across its replay cycle, time-skips timers, and manages callbacks.
+    </p>
+    <ol>
+        <li>Start the tool with <code>--durable-execution</code> (timers are time-skipped by default; use <code>--durable-time-skip false</code> for real waits).</li>
+        <li>
+            Run your durable function as a local process with these environment variables:
+            <ul>
+                <li><code>AWS_LAMBDA_RUNTIME_API=localhost:&lt;port&gt;/&lt;FunctionName&gt;</code> — must include the <code>/&lt;FunctionName&gt;</code> suffix.</li>
+                <li><code>AWS_ENDPOINT_URL_LAMBDA=http://localhost:&lt;port&gt;</code> — redirects checkpoint/state/callback calls to the tool.</li>
+                <li><code>AWS_REGION</code> plus dummy <code>AWS_ACCESS_KEY_ID</code>/<code>AWS_SECRET_ACCESS_KEY</code> — required for request signing; the emulator ignores the signature.</li>
+            </ul>
+        </li>
+        <li>On the Function Tester page, select your function, check <b>Start as durable execution</b> (optionally name it), and click <b>Invoke</b>.</li>
+        <li>Open the <a href="/durable-execution">Durable Execution</a> page to watch the operation timeline. When a workflow parks on a callback, use <b>Send Callback</b> to resume it.</li>
+    </ol>
+</div>
+

Tools/LambdaTestTool-v2/src/Amazon.Lambda.TestTool/Components/Pages/DurableExecution.razor

@@ -0,0 +1,154 @@
+@page "/durable-execution"
+@using Amazon.Lambda.DurableExecution
+
+<PageTitle>Durable Execution</PageTitle>
+
+<div class="container-fluid py-3">
+    <h3>Durable Execution</h3>
+
+    @if (Driver is null)
+    {
+        <div class="alert alert-info" role="alert">
+            The durable-execution emulator is not enabled. Restart the Test Tool with
+            <code>--durable-execution</code> and point your function's
+            <code>AWS_ENDPOINT_URL_LAMBDA</code> at this host to run durable workflows locally.
+        </div>
+    }
+    else
+    {
+        <p class="text-body-secondary">
+            Durable executions started against this Test Tool. Invoke your function with the
+            <code>X-Amz-Durable-Execution-Name</code> header (the AWS SDK sets this from
+            <code>DurableExecutionName</code>) to start one.
+        </p>
+
+        <div class="form-check form-switch mb-3">
+            <input class="form-check-input" type="checkbox" role="switch" id="timeSkipSwitch"
+                   checked="@_skipTime" @onchange="OnSkipTimeToggled" />
+            <label class="form-check-label" for="timeSkipSwitch">
+                <b>Time-skip timers</b>
+                <span class="text-body-secondary">
+                    — when on, <code>WaitAsync</code> and retry backoff resolve immediately instead of
+                    waiting real time. Applies to subsequent checkpoints.
+                </span>
+            </label>
+        </div>
+
+        <div class="row">
+            <div class="col-4">
+                <h5>Executions</h5>
+                @if (_executions.Count == 0)
+                {
+                    <div class="text-body-secondary fst-italic">No executions yet.</div>
+                }
+                else
+                {
+                    <div class="list-group">
+                        @foreach (var exec in _executions)
+                        {
+                            <button type="button"
+                                    class="list-group-item list-group-item-action @(_selectedArn == exec.Arn ? "active" : "")"
+                                    @onclick="() => SelectExecution(exec.Arn)">
+                                <div class="d-flex justify-content-between align-items-center">
+                                    <span class="text-truncate" title="@exec.Arn">@ShortName(exec.Arn)</span>
+                                    <span class="badge @PhaseBadgeClass(exec.Phase)">@exec.Phase</span>
+                                </div>
+                                @if (!string.IsNullOrEmpty(exec.FailureReason))
+                                {
+                                    <small class="d-block text-danger text-truncate" title="@exec.FailureReason">@exec.FailureReason</small>
+                                }
+                            </button>
+                        }
+                    </div>
+                }
+            </div>
+
+            <div class="col-8">
+                @if (_selectedArn is null)
+                {
+                    <div class="text-body-secondary fst-italic">Select an execution to view its timeline.</div>
+                }
+                else
+                {
+                    <h5 class="text-truncate" title="@_selectedArn">Timeline</h5>
+                    <table class="table table-sm align-middle">
+                        <thead>
+                            <tr>
+                                <th>Operation</th>
+                                <th>Type</th>
+                                <th>Status</th>
+                                <th>Detail</th>
+                            </tr>
+                        </thead>
+                        <tbody>
+                            @foreach (var op in _operations)
+                            {
+                                <tr>
+                                    <td><code>@op.Id</code>@if (!string.IsNullOrEmpty(op.Name)) { <span class="text-body-secondary"> (@op.Name)</span> }</td>
+                                    <td>@op.Type@if (!string.IsNullOrEmpty(op.SubType)) { <span class="text-body-secondary">/@op.SubType</span> }</td>
+                                    <td><span class="badge @StatusBadgeClass(op.Status)">@op.Status</span></td>
+                                    <td>
+                                        @{ var pendingCb = PendingCallbackId(op); }
+                                        @if (pendingCb is not null)
+                                        {
+                                            <button class="btn btn-sm btn-outline-primary" @onclick="() => PrefillCallback(pendingCb)">
+                                                Send Callback (@pendingCb)
+                                            </button>
+                                        }
+                                        else
+                                        {
+                                            <span class="text-body-secondary small">@DetailText(op)</span>
+                                        }
+                                    </td>
+                                </tr>
+                            }
+                        </tbody>
+                    </table>
+
+                    <div class="card mt-3">
+                        <div class="card-header">Send Callback</div>
+                        <div class="card-body">
+                            <div class="mb-2">
+                                <label class="form-label mb-0">Callback ID</label>
+                                <input class="form-control form-control-sm" @bind="_callbackId" @bind:event="oninput" placeholder="cb-..." />
+                            </div>
+                            <div class="mb-2">
+                                <label class="form-label mb-0">Result (JSON)</label>
+                                <input class="form-control form-control-sm" @bind="_callbackResult" @bind:event="oninput" />
+                            </div>
+                            <button class="btn btn-primary btn-sm" @onclick="SendCallbackSuccess">
+                                Send Success
+                            </button>
+                            @if (_callbackFeedback is not null)
+                            {
+                                <div class="mt-2 small">@_callbackFeedback</div>
+                            }
+                        </div>
+                    </div>
+                }
+            </div>
+        </div>
+    }
+</div>
+
+@code {
+    private static string ShortName(string arn)
+    {
+        // Show the trailing "GROUP/NAME" of the durable-execution ARN for readability.
+        var idx = arn.IndexOf("/durable-execution/", StringComparison.Ordinal);
+        return idx >= 0 ? arn[(idx + "/durable-execution/".Length)..] : arn;
+    }
+
+    private static string DetailText(Operation op)
+    {
+        if (op.StepDetails?.Result is { } r) return Truncate(r);
+        if (op.ContextDetails?.Result is { } cr) return Truncate(cr);
+        if (op.CallbackDetails?.Result is { } cbr) return Truncate(cbr);
+        if (op.ChainedInvokeDetails?.Result is { } ir) return Truncate(ir);
+        if (op.ExecutionDetails?.InputPayload is { } ip) return "input: " + Truncate(ip);
+        if (op.StepDetails?.Error?.ErrorMessage is { } em) return "error: " + em;
+        return string.Empty;
+    }
+
+    private static string Truncate(string s) => s.Length <= 80 ? s : s[..77] + "...";
+}