feat(spec): v0.1.25.16 — remaining_ttl_ms on reservation responses + normative heartbeat algorithm

[amavashev]

Summary

Add an optional remaining_ttl_ms field to reservation create/extend responses and define a deadline-safe heartbeat algorithm for extendReservation / extend_by_ms. The measured-grant scheme remains only as an explicitly non-normative fallback for older servers.

This is an additive wire-contract change in the still-unreleased runtime revision 0.1.25.16. semantic_base remains 0.1.25.

Why

extend_by_ms is relative to the reservation's current expires_at_ms, not request time. A client that extends by the full TTL every ttl/2 drifts expiry outward and burns the extension budget. Servers may also clamp the per-extension grant or clamp expiry to a maximum lead; those regimes cannot be identified safely from expiry deltas alone. A server-authoritative remaining-lifetime observation is therefore required for portable heartbeat scheduling.

Contract changes

• Add optional remaining_ttl_ms (int64, minimum 0) to ReservationCreateResponse and ReservationExtendResponse, computed from the same authoritative clock used for expiry decisions. It is absent on dry-run and DENY create responses.
• Treat remaining_ttl_ms as volatile transport metadata. On same-key CREATE or EXTEND replay, servers that emit it MUST recompute max(0, original expires_at_ms - current authoritative server time), never replay the stored value, and return 0 if the reservation is no longer ACTIVE. All other outcome fields replay verbatim.
• Exclude create-response remaining_ttl_ms from the signed reserve evidence payload, alongside cycles_evidence, so replay-time recomputation does not change the original evidence identity. The companion evidence spec is revised to 0.2.2 with no envelope wire-shape or schema_version change.
• Declare HTTP 429 on extendReservation, including non-negative delta-seconds Retry-After and X-RateLimit-Reset. HTTP-date Retry-After is intentionally invalid because safe conversion would require wall-clock assumptions.
• Keep HTTP Date out of lease arithmetic; it is documented only as standard HTTP metadata.

Normative heartbeat algorithm when remaining_ttl_ms is present

For every schema-valid HTTP 200 create or extend response:

• Measure per-attempt monotonic RTT.
• lead_floor = max(0, remaining_ttl_ms - rtt).
• attempt_budget = max(enforced request_timeout_budget, 1000 ms, 2 * maximum observed rtt).
• safety_margin = max(1000 ms, 2 * maximum observed rtt).
• retry_reserve = 2 * attempt_budget + safety_margin.
• next_delay = max(0, lead_floor - retry_reserve); schedule from response receipt.

A positive delay requires lead_floor > retry_reserve. Clients should configure timeout and expected-latency bounds so attempt_budget < (minimum expected lead_floor - safety_margin) / 2. With a 60000 ms returned lead, a 10000 ms timeout and 1500 ms maximum observed RTT produce a 23000 ms reserve and approximately 37000 ms cadence. A 30000 ms timeout produces at least a 61000 ms reserve and zero delay. An additive-delta server may establish more lead on the immediate extension; a 60000 ms maximum-lead clamp cannot.

All duration arithmetic uses milliseconds and is overflow-safe; positive overflow is treated as infinity, never wraparound. Where timer resolution requires rounding, clients round budgets/margins up and lead lower bounds/delays down so rounding cannot consume the reserved margin.

Unknown or unreliable attempt timing forces lead_floor = 0 and next_delay = 0. An unknown or unbounded request timeout makes attempt_budget infinite. A client without reliable monotonic attempt timing therefore cannot establish a safe primary-path cadence: after one immediate fresh extension, the two-consecutive-zero guard requires it to stop. It cannot switch to the fieldless fallback merely because local timing is unavailable.

Only schema-valid HTTP 200 responses count as observed success. Other or malformed 2xx responses are ambiguous and cannot drive scheduling. After a CREATE timeout, connection error, 5xx, or ambiguous 2xx, the client gets at most one immediate same-key recovery attempt.

For an EXTEND timeout, connection error, 5xx, 429, or ambiguous 2xx:

• Recompute current_lead_estimate from the last valid response and monotonic elapsed time.
• retry_window = current_lead_estimate - attempt_budget - safety_margin, without clamping.
• A negative retry window means no complete retry plus margin is provably safe: stop and surface the failure.
• Otherwise ordinary transient failures retry with the same idempotency key after min(30000 ms, current_lead_estimate / 4, retry_window).
• For 429, convert non-negative delta-seconds Retry-After to milliseconds with overflow checking and retry only when the converted delay fits within retry_window; clients do not invent an earlier retry that violates throttling.
• After every failed recovery, recompute from the same last valid response and current monotonic elapsed time. Further same-key retries may continue while the fresh retry_window is positive; elapsed time from prior attempts is already deducted and every decision reserves the next complete attempt plus margin.
• At exactly zero, permit only one immediate recovery retry. Stop if that fails or if neither elapsed time nor retry_window decreases between failures, preventing a zero-time loop.

A zero-delay success permits at most one immediate fresh-key extension attempt. If that success also produces zero delay, the client stops instead of burning a maximum-lead server's extension budget in a tight loop.

Fallback

When a server does not emit remaining_ttl_ms, the measured-grant/band guidance is NON-NORMATIVE and explicitly documents its limits. Clamp-regime detection is undecidable in general; fallback clients should prefer bounded over-beating and visible extension-budget depletion over risking a lease lapse.

The asymmetry is intentional: a field-bearing client with reliable timing can prove when no complete attempt plus margin fits and stops. A fieldless client cannot prove safety or lapse, so bounded over-beating is an explicitly best-effort choice that favors avoiding an unobservable lapse at the cost of extension-budget burn.

Files and versions

• Runtime spec/changelog: 0.1.25.16
• Evidence companion/changelog: 0.2.2
• Suite index: 0.1.45
• Runtime merged artifact regenerated

Validation

• All six canonical source specs and both merged artifacts: Spectral, 0 errors
• Changelog pointer/version validation: passed
• Spec-index consistency/publication inventory: passed
• CyclesEvidence golden fixtures: 13/13 passed
• Merge drift regeneration: byte-identical
• Dynamic retry-window scenarios: passed, including multiple recoveries, negative/zero windows, no-progress protection, and the 60s timeout configurations
• Retry-After seconds-to-milliseconds boundary and overflow scenarios: passed
• Conservative timer-resolution rounding rules reviewed against the retry-bound proof

Downstream implementation impact

• cycles-server: emit/recompute the volatile field, exclude it from reserve evidence, and expose the documented 429 response.
• Python, TypeScript, Rust, and Java SDKs: consume the optional field and adopt the normative timing/retry algorithm while retaining compatibility with older servers.
• Evidence consumers: envelope bytes remain compatible; emitters must continue omitting transport-only fields from ReservationCreateResponseMirror payloads.

Future option

An absolute-target extension such as set_expiry_to could make keep-alive idempotent with respect to drift, but it is outside this change.

[amavashev]

Guidance upgraded from alternate-beat to a lead-estimate pattern (commit c52b71b) after adversarial review found failure-path hazards in the original recommendation:

• Single-failure lead-0 race — with blind alternate-beat, steady-state extend attempts occur at exactly ttl/2 of remaining lead, so one failed extend puts the retry at the expiry instant.
• Success-path-only band — the [ttl/2, 1.5×ttl] target was silently breached by failed/missed beats and by the server-side clamping the same paragraph authorizes.
• Ignored authoritative signal — clients were never told to drive the loop from the returned expires_at_ms they already receive.

The rewritten HEARTBEAT GUIDANCE now recommends: maintain a clock-skew-free lead estimate from the authoritative returned expires_at_ms (server-frame differences + client-monotonic elapsed time only); beat at ~ttl/2, extend by ttl_ms when lead < 1.5×ttl_ms, skip otherwise — keeping lead in roughly [ttl, 2×ttl] on the success path, tolerating transient failures with ≥ ttl of retry margin, and self-correcting for slippage and clamping. It warns explicitly against unconditional extend-by-ttl (zombie windows) and blind alternate-beat without lead tracking, and notes heartbeat stop conditions (RESERVATION_EXPIRED / RESERVATION_FINALIZED / MAX_EXTENSIONS_EXCEEDED) plus idempotency_key reuse on retry.

Same unreleased 0.1.25.16 change — no additional version bump; description text only, no schema change. Changelog entry updated, merged artifacts regenerated, all validators + spectral clean.

[amavashev]

All three findings addressed in 02d1270:

1. [P1] Lead formula seeded from requested ttl_ms despite policy capping. Correct — max_reservation_ttl_ms (default 1h) silently caps the granted TTL and ReservationCreateResponse exposes neither the effective TTL nor a server-time anchor, so a 24h request capped to 1h would schedule the first beat ~12h after expiry. The guidance now states servers MAY cap the granted TTL by tenant policy, and clients SHOULD derive effective_ttl_ms ≈ expires_at_ms − HTTP Date response header (both server-frame, so the estimate stays clock-skew-free; the ±1s resolution of Date is negligible), clamped to [1000, requested ttl_ms], and use effective_ttl_ms wherever ttl_ms appeared — beat interval, lead seed, extension amount, and skip threshold. A first-class effective_ttl_ms (or server-time) response field is noted in the guidance and added to the PR description's discussion items alongside the existing set_expiry_to item.

2. [P2] "Retries occur with ≥ ttl of margin" overstated. Weakened as suggested: the margin guarantee now explicitly applies after the first successful extension; the first extend attempt inherently has only ~ttl/2 of margin.

3. [P2] Missing stop conditions. TENANT_CLOSED (with a note that closure is irreversible — the extend endpoint returns it) and NOT_FOUND added to the heartbeat stop list alongside RESERVATION_EXPIRED, RESERVATION_FINALIZED, and MAX_EXTENSIONS_EXCEEDED.

Also: the PR description was stale (still described the alternate-beat pattern and the old [ttl/2, 1.5×ttl] band) — rewritten to describe the lead-estimate guidance accurately. Changelog entry updated, merged artifacts regenerated; merge_specs.py, validate_changelogs.py, validate_spec_index_versions.py, and spectral (source + both merged) all clean at error severity. Same unreleased 0.1.25.16 — no additional version bump; description text only.

[amavashev]

Both round-3 findings confirmed and addressed in 00c7a7d. The guidance is rewritten around a conservative lead lower bound with no cross-clock arithmetic anywhere.

1. [P1] expires_at_ms − Date is not a same-frame measurement, and clamping it up to 1000ms fabricates lease. Correct on every count. RFC 9110 §6.6.1/§5.6.7 makes Date a whole-second, best-effort origination timestamp that may be generated at any point during message origination or replaced by intermediaries — and in the reference server the two values genuinely come from different clocks: expires_at_ms is computed from Redis TIME inside the Lua scripts, while Date is stamped by the servlet container. Calling the error "±1s negligible" was wrong, and the [1000, requested ttl_ms] clamp could round a stale-or-skewed estimate upward — inventing margin the reservation does not have. The derivation is gone as a correctness input. In its place:

• Clients maintain lead_min = Σ measured_grants − client_monotonic_elapsed_since_reserve_response, starting at 0, where each grant is the difference of successive returned expires_at_ms values (reserve response, then each extend response — the same server frame, so differencing them is legitimate). lead_min is a rigorous lower bound: it ignores the unknown initial effective TTL, and measuring elapsed time from receipt of each response makes request latency count against the bound, never for it.
• Because lead_min starts at 0, the first extension fires early after a small bounded delay — min(requested ttl/2, 30s, half the optional Date-derived estimate when available) — which both establishes real measured margin and reveals the actual per-extend grant.
• Date is reframed as an OPTIONAL cadence hint for sizing the first-beat delay only — never load-bearing for correctness, never clamped upward — with the RFC 9110 caveats and the different-clocks caveat stated inline. And since the guidance still references it, the header is now declared: a documented optional Date response header (components.headers.Date, with the same caveats in its description) on the createReservation and extendReservation 200 responses. Documentation of standard HTTP behavior — no schema or wire change.

2. [P2] "≥ ttl of margin after the first successful extension" is false under post-extension clamping and latency. Correct — if the server clamps an extension so the lead lands at 1.2×ttl, the next beat at ttl/2 arrives with only 0.7×ttl of lead, and request latency erodes whatever remains. All fixed-margin promises are removed. Replacements:

• Cadence now derives from the measured grant, not the requested ttl: beat at ~last_grant/2 (sensibly bounded), skip when lead_min ≥ 1.5×last_grant — so a clamped grant automatically tightens the beat instead of silently shrinking the margin under an unchanged schedule.
• The stated guarantees are now invariants that actually hold: extend attempts occur while lead_min ≥ the beat interval whenever grants keep pace with elapsed time; total protected runtime equals the initial effective TTL plus the sum of grants regardless of schedule; a failed attempt consumes lead_min and is retried at the current cadence with the SAME idempotency_key so a lost-response extend is not applied twice.

Also per the round-3 direction: the additive-field proposal is elevated into the guidance itself — a future revision SHOULD add an explicit effective-TTL (or server-time) field to ReservationCreateResponse; it is the only complete fix for the first-beat window, since no header-derived estimate can be made both safe and tight (set_expiry_to remains the second discussion item). The stop-condition list and idempotency-reuse text from the prior round are retained, and the PR description no longer presents the Date derivation as clock-skew-free.

Changelog entry rewritten, merged artifact regenerated; merge_specs.py, validate_changelogs.py, validate_spec_index_versions.py, and spectral (source + both merged) all clean at error severity. Same unreleased 0.1.25.16 — no additional version bump, no history rewrite.

[amavashev]

Both round-4 findings confirmed and addressed in b166911. I concede the central error explicitly: the previous revision conflated per-extend GRANT clamping with MAXIMUM-LEAD clamping and asserted "a server clamp automatically tightens the beat" as if it covered both. It covers only the former.

(a) [P1] A bounded first-beat delay can outlive a small capped initial lease. Correct. The round-3 scheme scheduled the first beat at min(requested ttl/2, 30s, half the Date-derived estimate) — but a 24h request silently capped to a 1s effective TTL is long expired when a 30s first beat arrives, and since the Date hint is non-load-bearing by construction it cannot be allowed to rescue the schedule (a hint that must not be trusted for correctness cannot be the thing correctness depends on). No field of the create response bounds the effective initial lease, so any bounded delay embeds an assumption the wire contract does not support. The guidance now says the first extension SHOULD be immediate — the only assumption-free schedule. Cost accounting is stated: it spends one extension from the budget, but total protected runtime (initial effective TTL + Σ grants) is schedule-invariant, so nothing is lost but the count. The bounded-first-delay recommendation and the Date hint's scheduling role are removed entirely; the Date header declaration remains purely as response documentation and the guidance no longer uses it for anything.

(b) [P1] Grant-derived cadence collapses under maximum-lead clamping. Correct, and this is the conflation conceded above. When the server holds expiry ≈ now + L, each successive expires_at_ms difference measures the elapsed time between calls, not the lease size — the "grant" merely echoes the client's own call spacing. Feeding it back as ~grant/2 cadence halves the interval every beat, collapses toward whatever cadence floor exists, and burns the extension budget (default 10) in ~10s; an immediate first extension can even measure a zero grant (expiry already at now + L). "Clamps self-correct" was written for grant clamping — where tightening is genuinely correct, because a smaller grant is a smaller lease — and silently extended to a regime where it is a destructive feedback loop.

The guidance now splits cadence by regime, distinguished by comparing each measured grant to the elapsed time since the previous successful extension:

• grant ≫ elapsed (normal, or per-extend grant clamp): beat at ~grant/2, sensibly bounded — tightening correct.
• grant ≈ elapsed or grant ≤ 0 (maximum-lead clamp): no cadence signal exists in the current wire contract. Hold a fixed bounded cadence (e.g. min(requested ttl/2, 30s)), MUST NOT tighten toward a floor in response to shrinking grants, SHOULD log/surface that the extension budget will deplete, and rely on MAX_EXTENSIONS_EXCEEDED as the terminal signal. A zero/tiny measured grant is documented as a regime signal, not an error.

The lead_min lower bound and the skip rule are retained (the skip rule annotated as meaningful only in the grant-measures-lease regime), as are the stop conditions and same-idempotency_key retry text.

Consequence, stated plainly in the spec: under maximum-lead clamping, correct heartbeat cadence is UNDECIDABLE from data currently on the wire — no combination of returned expires_at_ms values and client-side clocks distinguishes "small lease" from "re-anchored lead" until the budget is already burning. On that basis the additive proposal is elevated: a future revision SHOULD add an effective-lease field (effective TTL granted, or expires-relative lease info plus a server-time anchor) to both ReservationCreateResponse and ReservationExtendResponse, and the guidance now says this is REQUIRED for well-defined client keep-alive behavior under lead clamping — not an optimization. set_expiry_to remains the second discussion item.

Validation: merge_specs.py clean regenerate, validate_changelogs.py OK (6 specs), validate_spec_index_versions.py OK, spectral 0 errors on source + both merged artifacts. Changelog entry and PR body updated. Same unreleased 0.1.25.16; docs-only, no wire change.

[amavashev]

Follow-up amendment in 692e06f (same 0.1.25.16, docs-only): cross-SDK implementation review — the defect was found by the Rust implementation, and the fix is now shipped in all four SDKs — caught a hole in the regime-detection rule as written in b166911.

Defect: detecting the lead-clamp regime by an upper bound alone (grant ≤ ~1.25×elapsed-since-last-success) misclassifies a genuine per-extend-GRANT-clamped server after a lead_min skip: the next measured grant arrives across a doubled gap, so grant ≈ elapsed exactly for a server whose correct cadence is grant/2. The client flips to the held lead-clamp cadence, and the misclassification self-sustains — at the held cadence the ratio stays ≤ 1.25 forever — while the lease decays to a lapse, contradicting the guidance's own requirement that grant-clamped leases keep tightening.

Fix: regime detection SHOULD test that the grant sits inside a band around elapsed — e.g. 0.75×elapsed ≤ grant ≤ 1.25×elapsed — with grant ≤ 0 always indicating clamping and grant ≥ ~0.9×requested ttl_ms always indicating the normal regime. The band discriminates because a genuine maximum-lead clamp tracks ANY inter-success gap (grant/elapsed ≈ 1 at every cadence), while a real small grant seen across a skip-doubled gap matches the band at most once: at the held cadence its ratio falls below the lower edge and cadence re-tightens. Misclassification becomes transient, never sticky.

Guidance text and changelog wording updated in place; merge_specs.py / validate_changelogs.py / validate_spec_index_versions.py / spectral all clean (0 errors). No schema or wire change.

[amavashev]

Round-5 verdict accepted in full; both P1s conceded, and the reviewer's clean solution is adopted in a681e8c. The PR is no longer docs-only — title and body updated.

1. [P1] The band heuristic is formally undecidable — conceded, with your counterexample now in the spec text. Requested ttl 24s, per-extend grant cap 10s: after a lead_min skip the grant is measured across a doubled gap (ratio ≈ 1, inside the band), the client adopts the held cadence min(24s/2, 30s) = 12s, and at that cadence the ratio is 10/12 ≈ 0.833 — inside [0.75, 1.25] forever, while the lease erodes 2s per cycle to a lapse. My "matches the band at most once" claim was simply wrong: the entire window grant ∈ [0.75×min(ttl/2, 30s), 0.9×ttl) stays misclassified permanently once the held cadence is adopted, and grant ≥ 0.9×ttl does not prove the normal regime either (a maximum-lead clamp echoes any call spacing, including one approaching the requested ttl). No refinement of thresholds fixes this — regime detection from the observables (grant, elapsed) is undecidable in general, and the spec now says so plainly.

2. [P1] "Nothing is lost but the count" is false under maximum-lead clamping — conceded. Extension value is schedule-dependent: with a 60s maximum lead, an immediate extend measures a grant ≈ 0 where the same extend fired 30s later would have measured ≈ 30s. Total protected runtime and Σ grants are therefore NOT schedule-invariant, and immediate priming can spend one extension for zero added lifetime. The guidance now presents immediate priming (fallback path only) as exactly that tradeoff — minimized unknown-initial-lease lapse risk at the cost of one potentially zero-value extension — and explicitly retracts the earlier free-action claim in the text.

Adopted solution, per your recommendation: optional remaining_ttl_ms (int64, ≥ 0) on both ReservationCreateResponse and ReservationExtendResponse — remaining lifetime at the moment the response was evaluated, same authoritative clock snapshot as expires_at_ms, present on every successful live-reservation response (absent on dry-run and DENY). Servers SHOULD emit it; clients MUST treat it as optional. When present, the heartbeat algorithm is NORMATIVE: recompute from every successful response (never accumulate expiry differences); rtt unknown → 0; lead_floor = max(0, remaining_ttl_ms − rtt); retry_reserve = min(lead_floor/2, max(1000 ms, 2×max observed rtt)); next_delay = max(0, lead_floor − retry_reserve); first beat derives from the CREATE response the same way (no priming extension spent); transient failures retry with the same idempotency_key after clamp(current_lead_estimate/4, 1s, 30s); any 2xx counts as applied. The measured-grant/band scheme survives only as an explicitly NON-NORMATIVE fallback with its limits stated, including the conclusion that no portable, safe, extension-efficient heartbeat exists without remaining-lifetime data — which is why the field exists.

Correction to my earlier wording: the band fix is not "shipped in all four SDKs" — it is implemented on the four open SDK PR branches (py#90, ts#173, rust#75, java#110; none merged). Those branches will move to the remaining_ttl_ms primary algorithm. Server emission of remaining_ttl_ms is landing in cycles-server in parallel.

Versioning: the addition rides the same unreleased 0.1.25.16 (optional-field additive; semantic_base stays 0.1.25), but the changelog entry, spec-index comments, and PR title/body now describe it as a wire-contract addition, not documentation. Validation: merge_specs.py clean, validate_changelogs.py OK (6 specs), validate_spec_index_versions.py OK, spectral 0 errors on source + both merged artifacts.

[amavashev]

785c2f6 adds an idempotent-replay carve-out for remaining_ttl_ms, resolving a conflict between the replay normative ("replay MUST return the ORIGINAL successful response payload") and the field's intended semantics. On extend replays the field is volatile: a heartbeat retrying a lost extend with the same idempotency key schedules its next beat from the replayed body, and a cached remaining_ttl_ms would be stale by at least the retry delay — scheduling from it would overshoot the real lease — so servers that emit it MUST recompute it as max(0, original expires_at_ms − current authoritative server time) at replay-response construction time, while all other fields replay verbatim (this matches what cycles-server#260 deliberately implements). On create replays there is NO carve-out: the original body — including the original remaining_ttl_ms — is returned verbatim, because the emitted CyclesEvidence envelope references the original response body; the field descriptions now say clients SHOULD NOT treat a replayed create's value as current, and the HEARTBEAT GUIDANCE adds the matching client rule (a create response arriving after same-key retries may be a verbatim replay, so its rtt is measured from the first attempt's monotonic send time, bounding the staleness fed into lead_floor). The carve-out is stated at every replay-verbatim MUST (top-level IDEMPOTENCY, X-Idempotency-Key parameter, and both operations' IDEMPOTENCY blocks); changelog updated in place; merge_specs / validate_changelogs / validate_spec_index_versions / spectral all clean. Same unreleased 0.1.25.16.