feat(worker): run provider mutations outside the fleet lock under per-slot leases (C2 3 of 3)

bin/fm-worker-lifecycle.sh

@@ -35,6 +35,7 @@
 #   fm-worker-lifecycle.sh proof-template --task <id> --task-generation <id>
 #   fm-worker-lifecycle.sh release --task <id> --task-generation <id> --proof-file <json>
 #   fm-worker-lifecycle.sh withdraw --task <id> --task-generation <id> --confirm-withdraw --confirm-subscription <uuid>
+#   fm-worker-lifecycle.sh abandon-claim --slot <n> --idempotency-key <sha256> --confirm-abandon --confirm-subscription <uuid>
 #   fm-worker-lifecycle.sh surrender --task <id> --task-generation <id> --reason <text> --output <json> --confirm-surrender --confirm-subscription <uuid>
 #   fm-worker-lifecycle.sh resume <exact recovery flags>
 #   fm-worker-lifecycle.sh steer <exact assignment flags>
@@ -49,7 +50,7 @@ SCRIPT_DIR=$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd -P)
 . "$SCRIPT_DIR/fm-cloud-state-lib.sh"
 
 case "${1:-}" in
-  request|release|resume|steer|execute|authority-receipt|capacity-reserve|capacity-reserve-shape|capacity-release)
+  request|release|resume|steer|execute|authority-receipt|capacity-reserve|capacity-reserve-shape|capacity-release|abandon-claim)
     fm_refuse_if_gate_agent
     exec python3 "$SCRIPT_DIR/fm-worker-lifecycle.py" "$@"
     ;;

docs/azure-requirements.md

@@ -324,9 +324,12 @@ Acceptance: a measured review completes in 20 to 30 minutes, with the breakdown
 
 Status: NOT DONE.
 
-The durable state now holds per-slot `pending_actions` with a load fence and a revision CAS
-(C2's second change), but every provider mutation still serializes behind the fleet lock.
-That lock is the remaining direct blocker on this requirement.
+All three C2 changes are landed: the transactional apply, the per-slot `pending_actions` map
+with its load fence and revision CAS, and the lock discipline that runs every provider mutation
+outside the fleet lock under a non-blocking per-slot lease, with the drain after convergence and
+`abandon-claim` as the evidence-preserving exit from a deterministically refused claim.
+What remains for DONE is the acceptance itself: many crewmates, no-mistakes runs, and
+crosschecks demonstrated running in parallel against live capacity without contention.
 
 The lock is the other half, and the harder one: `controller_lock` is held across provider calls
 and for an execute's whole guest run, and the code's own note records that fixing only the lock was

docs/azure-workers.md

@@ -236,7 +236,8 @@ bin/fm-worker-lifecycle.sh reconcile \
   --confirm-subscription "$FM_AZURE_SUBSCRIPTION_ID"
 ```
 
-One home lock serializes local state; unapplied provider actions are durable per slot in `pending_actions`, every load is fenced to the lock hold that commits it, and a save whose on-disk revision moved since its load refuses instead of overwriting another writer's document.
+The home lock now covers only short read-validate-claim and apply sections; every provider mutation runs outside it under a non-blocking per-slot lease, so mutations for different slots run concurrently while readers and unrelated mutations proceed. Unapplied provider actions are durable per slot in `pending_actions`, every load is fenced to the lock hold that commits it, and a save whose on-disk revision moved since its load refuses instead of overwriting another writer's document.
+Reconcile drains stranded claims AFTER convergence, skipping any slot whose claim a live process still owns, so a wedged or hours-long replay cannot stop the fleet; a claim whose provider result is final but whose apply deterministically refuses is retired only through `abandon-claim`, which replays the mutation itself under the lease, proves the result binds the exact idempotency key, and records the refusal verbatim before clearing the claim.
 Each reconcile refreshes Azure before selecting the next action and stops after 64 actions even if a provider never converges.
 A provider error preserves the slot's pending action and records a bounded cleanup refusal.
 The next controller process replays that exact action before considering new work.

tests/behavior-test-durations.tsv

@@ -112,7 +112,7 @@
 180000	tests/fm-watch-pause-absorb.test.sh
 95645	tests/fm-watch-triage.test.sh
 20435	tests/fm-watcher-lock.test.sh
-8000	tests/fm-worker-lifecycle.test.sh
+12000	tests/fm-worker-lifecycle.test.sh
 1000	tests/fm-worker-outcome-transport.test.sh
 1000	tests/fm-worker-supervisor.test.sh
 23744	tests/fm-x-mode.test.sh

tests/fm-azure-pilot.test.sh

@@ -1678,18 +1678,27 @@ env = {
     "provider_argv": ["/usr/bin/true"],
 }
 try:
-    lifecycle.provider_call(env, "mutate", {"type": "create"})
+    # provider_call refuses "mutate" outright under the lock discipline; the
+    # subprocess bound under test lives on the raw path every mutate reaches
+    # through provider_mutate.
+    banned = None
+    try:
+        lifecycle.provider_call(env, "mutate", {"type": "create"})
+    except lifecycle.LifecycleError as exc:
+        banned = exc
+    assert banned is not None and "slot lease" in str(banned), banned
+    lifecycle._provider_call_raw(env, "mutate", {"type": "create"})
     create_timeout = captured["timeout"]
-    lifecycle.provider_call(
+    lifecycle._provider_call_raw(
         env, "mutate", {"type": "execute", "request": {"wall_seconds": 3600}}
     )
     execute_timeout = captured["timeout"]
 finally:
     lifecycle.subprocess.run = _real_run
 
-assert create_timeout >= 900, ("provider_call did not bound a create by its action", create_timeout)
+assert create_timeout >= 900, ("the raw provider path did not bound a create by its action", create_timeout)
 assert execute_timeout >= 3600 + 1800, (
-    "provider_call did not bound an execute by its guest run", execute_timeout,
+    "the raw provider path did not bound an execute by its guest run", execute_timeout,
 )
 print("OK")
 PROVIDERBOUND