Pilot: prove durable-runtime gap, then evaluate Restate for AES

[sergstack]

Goal

First prove that AI-OS has a recurring, material durable-runtime gap that the current Dual Surface / AES execution model does not already solve. Only if that gate passes, evaluate Restate as a bounded durable runtime for AES.

The issue must answer two sequential questions:

1. Gap gate: Is there observed execution friction that requires durable runtime capabilities such as crash-safe resume, durable waits, retry persistence, side-effect deduplication, or long-lived continuation?
2. Restate fit gate: If yes, can Restate solve that gap while preserving AES as the only semantic owner and without creating a second source of truth?

Status

• Decision status: candidate decision
• Production promotion: NOT AUTHORIZED
• Architecture adoption: NOT AUTHORIZED
• Scope: evidence-first, reversible pilot only
• Supersedes: LangGraph pilot issue Pilot: evaluate LangGraph as a bounded AES runtime adapter #341

Current evidence boundary

CURRENT_STATUS.md currently records:

• production_promotion: no;
• runtime_smoke_status: candidate;
• agentic workflows remain blocked;
• three real Dual Surface live tests passed with manual orchestration: 0;
• broad Phase 2 is currently NOT REQUIRED by available evidence;
• operating principle: use -> observe -> record material friction -> fix only recurring evidenced gaps.

Therefore this issue must not assume that a new runtime layer is required.

Target architecture if the gap is proven

                         Owner
                           │
                    Goal / decision
                           │
                           ▼
                  AI-OS CONTROL PLANE

      Goal Mode / routing / governance / evidence
                 AES semantic contract
                           │
                           ▼
              DURABLE EXECUTION PLANE

                     Restate
       persistence / resume / waits / retries
          journal / reliable invocation
                           │
          ┌────────────────┼────────────────┐
          ▼                ▼                ▼
       Thinking         Analytics          LLM
          │                │                │
          └────────────────┼────────────────┘
                           ▼
                         Codex
                           │
              RIOS / local models / tools

Architectural rule:

AI-OS decides WHAT MUST BE TRUE
AES owns execution semantics
Restate guarantees durable execution mechanics
workers perform bounded work within authority
human acts only at material authority boundaries

Ownership boundaries

AI-OS Control Plane owns

• Goal Mode;
• routing and project ownership;
• governance and evidence rules;
• AES semantics;
• acceptance criteria;
• Judge / QA / authority separation;
• handoff contracts;
• rollback policy.

AES remains the only semantic owner of

• execution_state;
• requirements and requirement IDs;
• defects and defect IDs;
• iterations;
• validation / freshness semantics;
• overall_delivery;
• qa_status;
• judge_verdict;
• authority_status;
• merge_status;
• production_status;
• Closure Review;
• continuation acceptance boundary.

Restate may own only execution mechanics

• durable step execution;
• process-crash recovery;
• persistence and resume;
• retry persistence for eligible transient operations;
• durable timers and waits;
• human / external signal waiting;
• reliable service invocation;
• execution journal / runtime observability;
• side-effect recovery mechanics;
• bounded parallel execution only if separately authorized later.

Restate journal/state is runtime evidence, not an alternate governance record.

Workers remain specialized

• [Thinking] — strategy, alternatives, risks;
• [Analytics] — deterministic data / calculations;
• [LLM] — reusable prompt / model workflow ownership;
• [Codex] — implementation and tests;
• RIOS — evidence / research service, not orchestration authority;
• local models — bounded worker pool for suitable extraction, classification, summarization, judging or other explicitly routed tasks.

No worker may silently become the lifecycle owner of the execution.

Human boundary

The runtime should not require owner intervention for deterministic policy transitions.

Examples that should remain machine-executable when authority and evidence are sufficient:

• record validation failure;
• register a recoverable defect;
• refresh stale evidence;
• retry an eligible transient operation within policy limits;
• rerun affected validation;
• enter Closure Review.

The owner must still be required for material authority decisions such as:

• scope expansion;
• residual-risk acceptance;
• owner approval;
• merge / deploy / production authorization;
• business or governance decisions not already authorized by policy.

Phase 0 — prove the durable-runtime gap

Do not integrate Restate into a real execution path yet.

Collect real execution evidence from current AI-OS / AES usage. A qualifying gap must be recurring or materially consequential and must not be solvable by a simpler bounded correction to the existing Dual Surface.

Candidate gap classes:

• manual recovery after process / session interruption;
• repeated context reconstruction for the same execution;
• loss of execution IDs, requirement IDs, defect IDs or evidence boundary across continuation;
• inability to wait durably for owner / external signal;
• duplicated side effects after retry / restart;
• incorrect or manual retry coordination;
• stale evidence used after source change;
• long-running execution that cannot reliably resume;
• repeated manual next-step routing despite deterministic policy;
• premature closure caused by lost runtime state.

Phase 0 evidence record

For each observed case capture:

• execution / case ID;
• observed failure or friction;
• frequency / recurrence;
• material impact;
• current workaround;
• whether Dual Surface / existing AES can solve it without a new runtime;
• evidence refs;
• owner intervention required;
• candidate minimal fix;
• why a durable runtime would materially improve the case.

Phase 0 acceptance gate

Proceed to Restate only if all are true:

• at least one material durable-runtime gap is directly observed;
• the gap is recurring or has material downside;
• it is not merely a UI / prompt / routing / documentation defect;
• a simpler bounded fix to the existing execution model is insufficient or clearly inferior;
• Restate capabilities map directly to the observed gap;
• expected benefit is measurable;
• rollback remains local.

If this gate fails: close the Restate implementation path as not_planned / retain issue evidence. Do not introduce a runtime layer.

Phase 1 — isolated Restate fit test

Only after Phase 0 passes.

Build a minimal isolated Restate-backed AES execution prototype.

Target flow:

START
 ↓
load canonical AES execution record
 ↓
scope
 ↓
execute
 ↓
validate
 ├── PASS ──> closure_review ──> owner gate / COMPLETE
 └── FAIL
       ↓
 register_defect
       ↓
 correction_allowed?
   ├── NO ──> STOPPED
   └── YES
         ↓
      correct
         ↓
      revalidate
         ↓
   closure_review

Required state preservation

Verify durable preservation of:

• execution_id;
• original goal / task boundary;
• requirements and requirement IDs;
• defects and defect IDs;
• current iteration;
• validation evidence;
• source revision and freshness state;
• acceptance criteria;
• overall_delivery;
• qa_status;
• judge_verdict;
• authority_status;
• merge_status;
• production_status;
• rollback data;
• handoffs;
• continuation / resume stage;
• Closure Review state.

Do not create a competing authoritative state model.

Required failure / durability tests

At minimum test:

1. worker process crash mid-execution;
2. runtime / worker restart and resume of the same execution;
3. completed side-effecting step followed by crash — no unintended duplicate side effect;
4. transient tool / network failure eligible for bounded retry;
5. hard blocker not eligible for retry;
6. owner approval wait and durable resume;
7. stale validation evidence after relevant source change;
8. failed mandatory requirement cannot become overall_delivery: pass;
9. iteration / correction limit reached;
10. malformed / missing continuation state;
11. Restate unavailable — canonical AI-OS/AES path remains recoverable.

Required crash scenario:

run
-> persist
-> kill worker / runtime component
-> restart
-> resume same execution

After resume verify exact preservation of:

• execution ID;
• requirements;
• defects;
• iteration;
• source revision;
• evidence boundary;
• authority state;
• acceptance criteria.

Any material mismatch = pilot failure.

Phase 2 — one bounded real-case pilot

Only after Phase 1 passes.

Use one Codex-style AES execution:

task package
-> implementation
-> tests
-> FAILED
-> defect registration
-> one permitted correction
-> affected-scope rerun
-> regression check
-> Closure Review
-> owner review

Respect the stricter Codex one-fix policy where applicable.

Hard constraints

1. No AES semantic changes for Restate convenience.
2. No routing changes solely to accommodate Restate.
3. No authority-model changes.
4. No new authoritative status namespace.
5. No automatic owner approval.
6. No automatic PR creation, merge, deploy or production promotion.
7. No scope widening by the runtime.
8. No self-acceptance of residual risk.
9. No multi-agent implementation in this pilot.
10. No replacement or weakening of AES validators or Closure Review.
11. Retry behavior must obey the strictest applicable AES / project rule.
12. Restate must remain removable without changing canonical AI-OS semantics.
13. RIOS remains an evidence provider, not execution authority.
14. Local / cloud model routing must not alter governance or acceptance semantics.
15. Do not build a web UI / dashboard in this issue.

Operational KPI

The primary KPI is not framework throughput and not "Restate works".

Measure whether the runtime removes a real class of coordination burden that current AI-OS does not already solve.

Compare baseline vs pilot for:

• manual continuation / resume actions;
• repeated context reconstruction;
• handoff / state recovery work;
• manual deterministic next-step routing;
• retries / replay after transient failure;
• duplicate side-effect prevention;
• stale-evidence recovery;
• premature / invalid closure attempts;
• time spent locating failure state;
• owner interventions where no material owner decision was required.

Preferred decision metric:

How many owner interventions and recovery actions are eliminated without increasing semantic drift, operational complexity, authority risk or rollback cost?

Non-goals

Do not include:

• AI-OS-wide runtime migration;
• broad Phase 2 orchestration redesign;
• multi-agent architecture;
• LangGraph / Mastra / Dapr / Temporal implementation comparison;
• changes to AES schema solely for Restate convenience;
• new governance concepts;
• changes to Thinkers / Analytics / RIOS semantics;
• production automation;
• automatic approvals;
• automatic PR creation / merge / deploy;
• web UI / exception dashboard;
• model-routing redesign.

Future exception-inbox / operational UI work is permitted only after runtime value is demonstrated by real executions.

Stop conditions

Stop and mark revise / blocked if:

• no material durable-runtime gap can be demonstrated;
• existing Dual Surface solves the observed problem with a simpler bounded correction;
• Restate requires duplicating canonical AES semantics;
• runtime state can contradict AES without deterministic reconciliation;
• resume loses IDs, evidence, authority or acceptance boundary;
• side effects can be duplicated after crash / recovery without an acceptable guard;
• retry behavior violates AES limits;
• implementation requires broad routing / governance redesign;
• operational complexity added exceeds coordination removed;
• rollback is not local and clean.

Rollback

1. Disable / remove the isolated Restate adapter and runtime integration.
2. Restore the prior direct AES / Dual Surface execution path.
3. Preserve pilot evidence and execution records.
4. Rerun relevant existing validators / smoke checks.
5. Make no changes to canonical AES semantics or historical records.

Judge gates

After Phase 0

Review:

Is there sufficient observed evidence that AI-OS needs a durable runtime layer at all?

Outcomes:

• pass -> run isolated Restate fit test;
• revise -> collect or classify more evidence;
• blocked -> do not integrate Restate.

After Phase 1 / Phase 2

Review:

Does Restate materially reduce the proven execution / continuation gap while preserving AES semantics, evidence integrity, authority boundaries, failure safety and reversibility?

Outcomes:

• pass -> candidate durable runtime component for additional heterogeneous pilots;
• revise -> retain evidence and narrow design;
• blocked -> remove Restate integration and retain current AI-OS execution model.

Revisit triggers

Revisit if:

• new real execution evidence appears;
• measured coordination reduction is immaterial;
• state drift or duplicate-side-effect risk appears;
• authority / Closure Review boundaries weaken;
• Restate operational burden becomes material;
• a simpler mechanism solves the same observed gap;
• implementation feedback contradicts this architecture.

Handoff

Only after the applicable gate passes, hand off to [Codex] with:

• exact allowed files;
• explicit phase (gap instrumentation, isolated fit test, or bounded real pilot);
• observed gap evidence refs;
• Restate deployment mode for the pilot;
• minimal adapter contract;
• canonical AES mapping;
• deterministic tests;
• crash / restart / resume tests;
• side-effect idempotency tests;
• owner-wait / resume test;
• rollback procedure;
• acceptance criteria above.

This issue does not authorize production adoption or an AI-OS-wide runtime migration.

[sergstack]

Phase 0 evidence review is prepared in PR #343.

Verdict: blocked — the current evidence does not prove a material gap that requires durable runtime mechanics. Restate Phase 1/2 remains not authorized and the implementation path is not_planned. The observed rate-limit/access friction is retained with smaller-control alternatives and explicit revisit measurements.

The issue remains open pending owner review / PR merge. No Restate code, runtime, routing, AES semantic, authority, or production change was made.