Skip to content

fix(azure): recover terminal worker claims - #291

Merged
ruby-dlee merged 1 commit into
mainfrom
codex/azure-wedge-recovery
Aug 21, 2026
Merged

fix(azure): recover terminal worker claims#291
ruby-dlee merged 1 commit into
mainfrom
codex/azure-wedge-recovery

Conversation

@ruby-dlee

Copy link
Copy Markdown
Owner

What this changes

This repairs terminal Azure worker execute claims that cannot be replayed or released under the old resource classification.

  • Reclassifies the task Run Command as per-execution transport in both provider and lifecycle identity checks while preserving its resource ID, attachment, tags, and all VM/disk/NIC identity fences.
  • Builds the execute script in one byte-stable helper and probes a stored Run Command only when it binds the exact script, or the exact request digest plus assignment-generation line.
  • Recovers a verified successful result without rerunning the guest. Failed or canceled exact commands return a deterministic terminal disposition that reconcile retains and an explicit operator abandon can record before clearing.
  • Adds a dedicated provider identity-refusal exit channel for permanently divergent recorded resource identities. Ordinary provider failures remain transient and cannot authorize abandonment.
  • Re-verifies the exact pending claim under lock before either refusal is recorded and before the claim is removed.
  • Documents the two new recordable abandonment dispositions and the deliberate fail-closed residuals.

Why

The live slot 2 claim recorded its task Run Command while its provisioning state was Succeeded. That same per-execution resource was rewritten and later ended Failed, but its fallback immutable identity is the provisioning-state string. Both provider replay and lifecycle classification therefore treated a status transition as a foreign resource identity, permanently blocking replay, release, and reset.

The terminal probe avoids a forced guest rerun while preserving exact execution identity. The structured identity-refusal channel is intentionally narrow: throttles, deallocated compute, landed-code failures, malformed results, tag mismatches, and other ordinary provider errors still retain the claim.

Verification

  • tests/run.sh tests/fm-worker-lifecycle.test.sh
  • tests/run.sh tests/fm-worker-outcome-transport.test.sh tests/fm-azure-pilot.test.sh
  • bin/fm-lint.sh
  • Python compilation and diff integrity checks
  • independent read-only review of the rebased exact commit; it found and closed ambiguous-source/nonterminal replay paths plus a foreign terminal-resource identity gap before push

The lifecycle fixture covers transient-refusal retention, terminal reconcile-retain then abandon-clear, exact identity-refusal abandonment, successful-result recovery, stale-script refusal, all-resource foreign identity matrices in provider and lifecycle, and the complete ordinary deallocate, delete-compute, reset release ladder. Mutation pins cover the five required safety boundaries.

No live worker claim or Azure resource is changed by this PR. Recovery begins only from merged main after exact-head review and CI are green.

@ruby-dlee
ruby-dlee merged commit d5d9044 into main Aug 21, 2026
13 checks passed
@ruby-dlee
ruby-dlee deleted the codex/azure-wedge-recovery branch August 21, 2026 20:18
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant