Skip to content

feat(durable): add zeph durable cancel command and Canceled status#6439

Merged
bug-ops merged 1 commit into
mainfrom
feat/issue-6362/durable-run-cancel
Jul 18, 2026
Merged

feat(durable): add zeph durable cancel command and Canceled status#6439
bug-ops merged 1 commit into
mainfrom
feat/issue-6362/durable-run-cancel

Conversation

@bug-ops

@bug-ops bug-ops commented Jul 18, 2026

Copy link
Copy Markdown
Owner

Summary

Killing the process running a durable execution only delays its resume: the next crash-orphan sweep resumes it, since crash-resume is explicitly designed to survive process death. This adds a durable, intentional Canceled terminal status distinct from the crash-driven Aborted state, and a zeph durable cancel <id> command to set it.

  • New ExecutionStatus::Canceled variant, additive (SQLite CHECK constraint extended via migration 112 on both dialects).
  • LocalBackend::cancel_execution probes the execution's advisory lock (INV-15) before writing: no live owner → immediate cancel via the same race-safe UPDATE ... WHERE status = 'running' pattern finalize already uses; a live owner (or a concurrent maintenance sweep) → refused (CancelOutcome::LiveOwner), row untouched; a cross-process backend with no lock to probe (Postgres) → refused (CancelOutcome::LivenessUnverifiable) rather than blind-flipping a possibly-live row.
  • Canceled executions are never reopened or resumed: open_execution/open_execution_exclusive fail closed with DurableError::ExecutionCanceled (refines INV-16 → INV-16′ in specs/064-durable-execution/spec.md — operator intent now outranks the active-reopen heuristic it stands in for). zeph durable resume reports "intentionally canceled" distinctly from the generic adapters-not-wired message.
  • zeph durable list --status canceled and prune/prune --dry-run (TTL retention) both recognize the new status.
  • SQLite migration 112 rebuilds durable_executions' CHECK constraint under -- no-transaction + explicit BEGIN/COMMIT, required so PRAGMA foreign_keys = OFF actually takes effect for the parent-table rebuild (the PRAGMA is a no-op inside sqlx's default per-migration transaction, which would otherwise FK-violate in production against real durable_journal/durable_promises/durable_timers child rows — caught in review, verified by a dedicated FK-survival integration test). The matching Postgres migration is a plain constraint swap.

Cooperative cancellation of a genuinely live owner (a durable signal the owning process observes and stops on cooperatively) is explicitly out of scope here: there is no live-owner step loop to observe it yet, since the per-kind resume adapters (agent-turn/subagent) are unwired in this build. Tracked as a follow-up issue.

Closes #6362

Test plan

  • cargo +nightly fmt --check
  • cargo clippy --profile ci --workspace --all-targets --features "desktop,ide,server,chat,pdf,scheduler,testing" -- -D warnings
  • cargo nextest run --config-file .github/nextest.toml --workspace --features "desktop,ide,server,chat,pdf,scheduler" --lib --bins — 14304 passed, 0 failed
  • RUSTFLAGS="-D warnings" RUSTDOCFLAGS="--deny rustdoc::broken_intra_doc_links" cargo doc --no-deps --workspace --features "desktop,ide,server,chat,pdf,scheduler"
  • cargo test --doc --workspace --features "desktop,ide,server,chat,pdf,scheduler"
  • gitleaks protect --staged --no-banner --redact — no leaks
  • Dedicated FK-survival migration integration test (crates/zeph-db/tests/durable_canceled_migration.rs) — seeds all three FK-linked child tables, runs the real migration through a file-backed foreign_keys(true) pool, asserts row/FK survival
  • Real concurrent race tests (tokio::spawn + tokio::join!, 20 trials) for cancel-vs-finalize and cancel-vs-sweep_orphans
  • cargo clippy -p zeph-durable -p zeph-db --no-default-features --features postgres — dual-backend model respected
  • Live-session kill-and-cancel walkthrough — not yet performed (playbook scenarios written, flagged Partial in coverage-status.md, recommended as a follow-up live-tester pass)

@github-actions github-actions Bot added enhancement New feature or request size/XL Extra large PR (500+ lines) documentation Improvements or additions to documentation rust Rust code changes and removed size/XL Extra large PR (500+ lines) labels Jul 18, 2026
Killing the process running a durable execution only delays its
resume: the next crash-orphan sweep resumes it, since crash-resume
is explicitly designed to survive process death. This adds a
durable, intentional Canceled terminal status distinct from the
crash-driven Aborted state, and a `zeph durable cancel <id>`
command to set it.

cancel_execution probes the execution's advisory lock before
writing: no live owner cancels immediately via the same race-safe
conditional UPDATE finalize already uses; a live owner (or a
concurrent maintenance sweep) is refused rather than blind-flipped;
a cross-process backend with no lock to probe (Postgres) is refused
for the same reason. Canceled executions are never reopened or
resumed (refines INV-16 in spec-064, since operator intent now
outranks the reopen heuristic it stands in for).

Cooperative cancellation of a genuinely live owner is deferred to a
follow-up issue: there is no live-owner step loop to observe it yet,
since the per-kind resume adapters are unwired in this build.
@bug-ops
bug-ops force-pushed the feat/issue-6362/durable-run-cancel branch from afb313e to b76032b Compare July 18, 2026 02:17
@bug-ops
bug-ops enabled auto-merge (squash) July 18, 2026 02:17
@github-actions github-actions Bot added the size/XL Extra large PR (500+ lines) label Jul 18, 2026
@bug-ops
bug-ops merged commit 8c4ffe2 into main Jul 18, 2026
47 checks passed
@bug-ops
bug-ops deleted the feat/issue-6362/durable-run-cancel branch July 18, 2026 02:27
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

documentation Improvements or additions to documentation enhancement New feature or request rust Rust code changes size/XL Extra large PR (500+ lines)

Projects

None yet

Development

Successfully merging this pull request may close these issues.

research(core): zeph-durable has no external pause/cancel — killing the process only delays resume

1 participant