5.5.1.2. Checkpoints and Recovery

Activity Checkpoints

A checkpoint persists the values and network marking of an Orleans Activity without requiring its streaming network to run to exhaustion. It is a firing-boundary barrier, not serialization of a running .NET task and not cancellation.

Use it to establish a durable restart point during Activity streaming. A checkpoint covers the state that reached its barrier, not an arbitrary list of previously acknowledged batches as completed business work.

Standalone checkpoint

Use the dedicated event when there is no application event to send:

using Stateflows.Common;

var result = await activity.CheckpointAsync();
if (result.Status != EventStatus.Consumed)
{
    throw new InvalidOperationException(
        $"Checkpoint did not complete: {result.Status}.");
}

CheckpointAsync is an IActivityBehavior helper for:

var result = await activity.SendAsync(new ActivityCheckpoint());

The event does not inject tokens, activate accept-event nodes as an application trigger, or implicitly initialize an idle Activity. A successful result returns after the snapshot commits; it does not mean the network completed. It is supported only for Orleans Activities; unsupported runtimes and other behavior types reject it.

The standalone event follows the regular inbox order. It does not jump ahead of an active regular transaction.

Checkpoint together with a regular event

Attach the Checkpoint header when an application event's transactional effects should be included in the snapshot:

var result = await activity.SendAsync(
    new SetThreshold(20m),
    new Dictionary<string, EventHeader>
    {
        [nameof(Checkpoint)] = new Checkpoint()
    });

ActivityCheckpoint is the standalone event; Checkpoint remains the header. They use the same barrier, persistence, timeout, and resumption machinery.

The header first establishes a barrier, then lets the requesting regular event's transactional part execute against that live context. Its state is committed with the checkpoint. Normal event acceptance still applies: a header does not make an otherwise unhandled application event acceptable. Inspect its result rather than treating any completed task as a successful application command.

A header on an unrelated status query also requests a checkpoint, but CheckpointAsync() expresses the standalone intent directly.

What the barrier does

1. Stop starting new network firings; hold new stream inputs for the next epoch.
2. Allow already-running node bodies and their flow pipelines to drain. Route their outputs into marking, recording eligible successors as ready work rather than starting them.
3. Execute the requesting regular event's transactional part, if there is one, and persist the snapshot with that call.
4. Release the barrier and resume ready work and later admissions.

The network does not need to exhaust all possible execution. A long chain can pause between two action firings. Conversely, a body that never reaches its boundary can block the checkpoint indefinitely unless a timeout is configured.

New input arriving during the barrier/commit does not become part of the snapshot being written. Do not infer checkpoint inclusion solely from when the client started a send.

Persisted marking and recovery

Values and marking are persisted together, including waiting tokens and active node configuration. Ready work records the node, upstream edge, execution thread, and selection tokens needed to restart a firing.

Suspended structured/iterative/parallel work records continuation frames: original inputs, remaining inputs, pending/completed partitions, and accumulated output. The runtime does not serialize stack frames, task objects, or local variables of an action body.

Interleaved iterative/parallel invocations release completed partition records and disposed scopes as they run. Their continuation frames retain unfinished work and accumulated boundary output; recovery does not replay the released partitions. The iterative execution identity is preserved even after all earlier partitions have been released.

On reactivation, the network resumes checkpointed ready work before serving the next ordinary call. Structural initialization is not repeated; unfinished partitions continue and the structural finalizer runs once. Partition selection tokens remain available to resumed typed actions. Idle tokens remain idle until their missing inputs or events arrive.

This is recovery from the last durable snapshot, not exactly-once execution of arbitrary application effects. Work after that snapshot may be lost or replayed following failure. For external operations, use idempotency keys or an application-owned durable completion record.

Resumed streamed invocations expose IExecutionContext.ExecutionMode == ExecutionMode.Streamed with null ExecutionTrigger, ExecutionTriggerId, and Headers. Neither the checkpoint event nor a later status query is substituted as their cause. Use resumed input/selection-token APIs for their data; no source-event lineage is reconstructed.

Timeout and unsuccessful checkpoints

StateflowsRuntimeOptions.ActivityStreamBarrierTimeout defaults to Timeout.InfiniteTimeSpan. Set a finite value when an unbounded wait is unacceptable:

silo.AddStateflowsRuntime(options =>
{
    options.ActivityStreamBarrierTimeout = TimeSpan.FromSeconds(30);
});

Valid values are nonnegative durations or Timeout.InfiniteTimeSpan. The timeout governs waiting for the stream to reach its firing boundary; it is not a deadline for all storage operations or downstream external effects.

When the barrier times out, the checkpointing call fails with a timeout, its deferred publications are discarded, and the barrier is released so live work can resume. Already-applied live effects are not rolled back. A failed or uncertain checkpoint must not be counted as durable success.

If a storage or transport failure leaves the commit outcome uncertain, use application-level recovery/idempotency rather than assuming either that the checkpoint definitely happened or that retrying is harmless.

Checkpointed reset

To replace a busy network without resuming its drained successors, attach the checkpoint header to reset:

var result = await activity.ResetAsync(
    ResetMode.KeepVersionAndSubscriptions,
    new Dictionary<string, EventHeader>
    {
        [nameof(Checkpoint)] = new Checkpoint()
    });

Active bodies drain to the barrier; reset clears Activity values and marking and discards ready successors. The reset mode still controls the other reset policy, such as subscriptions/version preservation. A standalone checkpoint followed by a separate reset is not the same atomic boundary: the network can resume between those calls.

Checkpoint, quiescence, or finalization?

Boundary	Purpose	Stops the Activity?
Natural quiescence	Automatically persist when no runnable/in-flight work remains	No; waiting nodes/tokens can remain
Standalone ActivityCheckpoint	Persist a busy network at a firing boundary	No; ready work resumes
Checkpoint header	Include a regular event's transactional effects in the snapshot	No, unless that event resets/finalizes the model
Queued/immediate Finalize	Stop admission and cancel/drain active work	Yes, after the finalization lifecycle finishes

Checkpoint persistence depends on the configured Orleans storage provider. In-memory grain storage cannot provide crash/restart durability across silo shutdown. See Orleans Runtime for durable storage requirements and Activity Streaming for finalization guarantees.

SQL Server recovery verification

Tests/Orleans.IntegrationTests/ActivityStreamingSqlRecoveryTests.cs verifies interleaved iterative and bounded parallel recovery using real Orleans ADO.NET SQL Server storage. It tests both checkpoint entry points, stops and disposes the entire host, then starts a fresh host with the same storage identity. The cases cover completed output, suspended successors, queued later arrivals and their common inputs, partition selection, iterative identity, concurrency bounds, and single initialization/finalization. A second restart verifies that completed work does not run again.

Run these cases from the framework repository:

dotnet test .\Tests\Orleans.IntegrationTests\Orleans.IntegrationTests.csproj --filter FullyQualifiedName~ActivityStreamingSqlRecoveryTests

They reuse the SQL test suite's local Stateflows database, with STATEFLOWS_SQLSERVER available as a connection-string override. Each case uses a unique Orleans service identity and removes only its own storage rows. SQL Server must be available; otherwise the existing SQL test setup marks the cases inconclusive. These are full-host restart tests after a committed checkpoint, not abrupt process-kill or database-outage tests, and do not establish exactly-once external side effects.