fix: heartbeat extend drift + actual_source audit marker (v0.5.1)

AUDIT.md

@@ -13,6 +13,58 @@
 
 ---
 
+## 2026-07-28 — Server-authoritative heartbeat via remaining_ttl_ms (v0.5.1, same PR)
+
+Spec review round 5 proved the fallback heuristic's regime detection
+undecidable (sticky band window [0.75×min(ttl/2,30s), 0.9×ttl)) and
+immediate priming schedule-dependent under maximum-lead clamping, so the
+protocol adopted remaining_ttl_ms on create+extend responses (spec
+v0.1.25.16; cycles-server v0.1.25.59 emits it, recomputed fresh on
+idempotent replays and excluded from evidence). The SDK implements the
+spec's hardened NORMATIVE algorithm when the field is present: per-attempt
+rtt, lead_floor = max(0, remaining − rtt), retry_reserve =
+2×max(request_timeout_budget, 1s, 2×rtt_max) + max(1s, 2×rtt_max) (the
+enforced httpx timeouts define the budget), next beat at lead_floor −
+retry_reserve from response receipt, recomputed per schema-valid 200
+(non-200 2xx is ambiguous → same-key recovery); recovery repeats while
+retry_window = lead_estimate − attempt_budget − safety_margin stays
+positive with a no-progress guard, 429 Retry-After honored only inside the
+window, other 4xx stop without key rotation; a lease that cannot hold the
+reserve gets one immediate fresh attempt then stop-and-surface; lead_min
+skip bypassed; no primed extension when the create carries the field.
+Bookkeeping keeps running so the v2.3+band heuristic (now explicitly
+best-effort fallback) resumes seamlessly if the field disappears. Final
+self-review also made the response contract uniform across both scheduling
+modes: only a complete schema-valid HTTP 200 create/extend response succeeds;
+ambiguous 2xx keeps the same key. The enforced timeout covers the whole
+attempt, first-delay setup time is deducted, and reliable pre-field RTT
+samples remain in the safety budget. 597 tests pass at 99.07% coverage.
+
+## 2026-07-27 — Heartbeat conservative-lead redesign + actual_source marker (v0.5.1)
+
+The heartbeat extended by full ttl_ms every ttl/2 beat while extend_by_ms
+is relative to current expiry — drifting expiry outward +ttl/2 per beat
+(zombie budget lockup on kill; max_extensions burned 2× too fast). Four
+adversarial review rounds refined the replacement; final (v2.3) design:
+conservative lead LOWER BOUND lead_min = Σ measured grants − monotonic
+elapsed (grants from successive returned expires_at_ms — same server
+frame only, no cross-clock arithmetic); FIRST extension fires immediately
+(any bounded first delay can outlive a tenant-policy-capped lease);
+cadence splits by regime — a grant tracking the lease drives
+clamp(grant/2, 500ms, ttl/2), while a grant merely mirroring elapsed time
+(maximum-lead clamping: grant ≤ 0, or grant < 0.9×ttl inside a
+[0.75, 1.25]×elapsed band — the lower edge keeps a post-skip small grant
+from sticking in the hold) carries no wire cadence signal, so the loop
+holds min(ttl/2, 30s) and warns once instead of burning max_extensions at
+the floor; a transient failure on the primed beat backs off to the held
+cadence (no hot loop); skip at lead_min ≥ 1.5×last_grant; extend
+idempotency key reused on retries; permanent stop on expired/finalized/
+max-extensions/tenant-closed/not-found (and raw 404/410). The HTTP Date
+header plays no heartbeat role (RFC 9110 §6.6.1; Redis TIME vs container
+clock). Commits whose actual was defaulted from the estimate now carry
+metadata.actual_source="estimate" for audit honesty. Spec guidance:
+cycles-protocol#148. 533 tests pass at 100% coverage.
+
 ## 2026-07-27 — Durable commit retries (journal + /v1/events fallback)
 
 Pending commits no longer exist only in memory: the retry engines journal

CHANGELOG.md

@@ -5,6 +5,20 @@ All notable changes to this project will be documented in this file.
 The format is based on [Keep a Changelog 1.1.0](https://keepachangelog.com/en/1.1.0/),
 and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0.html).
 
+## [0.5.1] - 2026-07-27
+
+### Fixed
+
+- **Final lease-response and timing conformance (supersedes the older fallback wording below).** Both server-authoritative and fieldless fallback scheduling count only a complete, schema-valid HTTP 200 create/extend response as success; malformed or non-200 2xx responses remain ambiguous and are recovered with the same idempotency key. Create performs one same-key recovery attempt, the enforced timeout covers the whole attempt, post-receipt setup time is deducted from the first delay, and reliable RTT samples gathered before a rolling-upgrade server starts sending `remaining_ttl_ms` remain part of the safety budget.
+- **Heartbeat redesign (conservative lead lower bound, immediate prime, regime-aware cadence)**: four adversarial review rounds refined the design. The heartbeat maintains `lead_min = Σ measured grants − monotonic elapsed` (grants = differences of successive returned `expires_at_ms` — the same server frame; no cross-clock arithmetic anywhere), starting at 0, so the FIRST extension fires immediately: any bounded first delay could outlive a tenant-policy-capped lease (a 24h request silently capped to seconds would expire before a delayed first beat), and priming costs one extension without changing total protected runtime. Cadence then splits by regime, detected from the measured grant: when the grant tracks the requested lease, cadence = `clamp(grant/2, 500ms, ttl/2)` — per-extend policy clamps automatically tighten the beat; when the grant merely mirrors elapsed time (maximum-lead clamping: `grant ≤ 0`, or `grant < 0.9×ttl` with `grant` inside `[0.75, 1.25]×elapsed-since-last-success`), no cadence signal exists on the wire — the loop holds `min(ttl/2, 30s)` and warns once that the extension budget will deplete, instead of collapsing to the floor and burning `max_extensions` in seconds. The band's lower edge makes misclassification non-sticky: a real per-extend grant seen across a skip-doubled gap (where grant ≈ elapsed exactly) lands in the hold once, but at the held cadence its grant/elapsed ratio falls below 0.75 and cadence re-tightens — with an upper bound alone the hold would stick and a clamped-grant lease would decay to a lapse. A transient failure on the primed (delay-0) beat backs off to the held cadence — never a hot loop. Skip when `lead_min ≥ 1.5×last_grant`. The HTTP `Date` header plays no role in the heartbeat (RFC 9110 §6.6.1: best-effort, whole-second, and possibly a different clock than the one stamping `expires_at_ms` — Redis TIME in the reference server); `CyclesResponse.server_date_ms` remains as a general accessor. Failed extends retry with the same idempotency key; permanent codes (`RESERVATION_EXPIRED`/`RESERVATION_FINALIZED`/`MAX_EXTENSIONS_EXCEEDED`/`TENANT_CLOSED`/`NOT_FOUND`, raw 404/410) stop the heartbeat; no interval floor. Spec guidance: cycles-protocol#148.
+- **Heartbeat (superseded intermediate designs, kept for history)**: the initial alternate-beat fix traded outward drift for inward-drift liveness hazards (single-failure lead-0 retry, guaranteed lapse for ttl < 2s under the old 1s interval floor, RTT slippage) found by adversarial self-review. The heartbeat now estimates its remaining lead from the authoritative `expires_at_ms` the server returns — compared clock-skew-free (server-frame differences + client-monotonic elapsed only) — and extends when lead < 1.5×ttl. It also: derives the **effective TTL** from `expires_at_ms − Date` header (tenant policy `max_reservation_ttl_ms`, default 1h, silently caps grants — a 24h request would otherwise heartbeat 12h late); reuses the same idempotency key when retrying a failed extend (a lost response cannot double-extend); stops permanently on `RESERVATION_EXPIRED`/`RESERVATION_FINALIZED`/`MAX_EXTENSIONS_EXCEEDED`/`TENANT_CLOSED`/`NOT_FOUND`; and drops the 1s interval floor. Spec guidance: cycles-protocol#148.
+- **Heartbeat extend drift** (superseded by the redesign above, kept for history): `extend_by_ms` is relative to the reservation's *current* `expires_at_ms` (spec), but the heartbeat extended by the full `ttl_ms` on every `ttl/2` beat — drifting expiry outward by `ttl/2` per beat. A killed process left its reserved budget locked until the drifted expiry (a zombie-reservation window scaling with runtime, bounded only by `max_extensions`), and long runs burned the `max_extensions` budget twice as fast as needed, losing heartbeat protection mid-flight. All four heartbeats (both lifecycles, both streaming context managers) now use alternate-beat extension: extend on the first beat and every second beat after a success, retrying immediately after a failure. Expiry lead stays within `[ttl/2, 1.5×ttl]`; extension consumption is halved. Fleet-wide fix (TS/Java/Rust ship the same change); spec guidance added in cycles-protocol#148.
+
+### Added
+
+- **Server-authoritative heartbeat scheduling via `remaining_ttl_ms`** (spec v0.1.25.16, cycles-protocol#148): when a create or extend response carries `remaining_ttl_ms` — the server's own statement of the live lease, measured on the clock that stamps `expires_at_ms` — the heartbeat runs the spec's NORMATIVE algorithm: per-attempt monotonic rtt (max tracked), `lead_floor = max(0, remaining − rtt)`, `attempt_budget = max(request_timeout_budget, 1s, 2×max rtt)` (the SDK's enforced httpx timeouts: connect + read + write), `safety_margin = max(1s, 2×max rtt)`, `retry_reserve = 2×attempt_budget + safety_margin`, next beat at `lead_floor − retry_reserve` after response receipt, recomputed on every field-bearing schema-valid HTTP 200 — never accumulated from expiry differences; the heuristic `lead_min` skip is bypassed and no primed first extension is spent (the create response's field drives the first delay). Only a schema-valid 200 counts as an observed success: an ambiguous non-200 2xx is recovered with the SAME idempotency key. Transient failures (timeout/connection/5xx/429/ambiguous 2xx) recover while the freshly recomputed `retry_window = lead_estimate − attempt_budget − safety_margin` stays positive (progress-guarded against zero-time loops; window < 0 stops and surfaces); a 429's `Retry-After` (delta-seconds) is honored only inside the window, never re-invented earlier; other 4xx stop without key rotation. A success whose lease cannot hold the retry reserve permits ONE immediate fresh attempt, then the heartbeat stops and surfaces that the lease is shorter than the retry-safety budget. Grants/lead bookkeeping keeps running in both modes, so the heuristic below takes over seamlessly if the field disappears; fieldless servers see unchanged fallback behavior. The heuristic (conservative lead lower bound + regime band) is now explicitly a **best-effort fallback**: spec review round 5 proved regime detection from `(grant, elapsed)` observables undecidable in general — the band's sticky window `grant ∈ [0.75×min(ttl/2, 30s), 0.9×ttl)` misclassifies permanently — which is precisely why the wire field exists.
+- `metadata.actual_source: "estimate"` is stamped on commits whose actual was structurally defaulted from the estimate (`@cycles` without an `actual` expression; streams with no recorded cost or a raised `cost_fn`), so audit evidence distinguishes measured spend from assumed spend. Defaults are unchanged; the marker flows into `/v1/events` recovery bodies via the shared metadata.
+
 ## [0.5.0] - 2026-07-27
 
 Durable commit retries. Previously a commit that failed transiently lived only in an in-memory daemon thread (or an unreferenced asyncio task): a process exit — even a clean one — dropped it, and once the reservation's grace period elapsed the server's expiry sweep returned the reserved budget to the pool, permanently under-counting spend that had already happened. Pending commits are now journaled to disk before retry, replayed on the next run, flushed (bounded) at interpreter exit, and — when the reservation has already expired — recovered via `POST /v1/events`, the spec's post-hoc direct-debit endpoint.

pyproject.toml

@@ -4,7 +4,7 @@ build-backend = "hatchling.build"
 
 [project]
 name = "runcycles"
-version = "0.5.0"
+version = "0.5.1"
 description = "Python AI agent budget control — enforce LLM cost limits, tool permissions, and multi-tenant policies before agent actions execute."
 readme = "README.md"
 license = "Apache-2.0"

runcycles/__init__.py

@@ -25,6 +25,7 @@
     CommitRequest,
     CommitResponse,
     CommitStatus,
+    CyclesEvidenceRef,
     CyclesMetrics,
     Decision,
     DecisionRequest,
@@ -92,6 +93,7 @@
     "Caps",
     "Decision",
     "CyclesMetrics",
+    "CyclesEvidenceRef",
     "Balance",
     "CommitOveragePolicy",
     "ErrorCode",

runcycles/client.py

@@ -41,7 +41,14 @@ def _extract_idempotency_key(body: dict[str, Any]) -> str | None:
     return body.get("idempotency_key")
 
 
-_RESPONSE_HEADERS = ("x-request-id", "x-ratelimit-remaining", "x-ratelimit-reset", "x-cycles-tenant", "retry-after")
+_RESPONSE_HEADERS = (
+    "x-request-id",
+    "x-ratelimit-remaining",
+    "x-ratelimit-reset",
+    "x-cycles-tenant",
+    "retry-after",
+    "date",
+)
 
 _BALANCE_FILTER_PARAMS = {"tenant", "workspace", "app", "workflow", "agent", "toolset"}