Skip to content

v0.14.0 — Durable runtime hardening & idiom polish

Choose a tag to compare

@dberry37388 dberry37388 released this 24 Jun 05:13
73afb6b

Durable runtime hardening & idiom polish — batches the durable recovery sweeps off per-row hydration, investigates and guards the durable run-state write path, finishes the readonly class / Collection idiom adoption, and redacts exception messages on the audit event/log paths.

Added

  • Redact exception messages on the audit sink-failure / signing log paths through the capture authority (#275). Audit failure and signing paths previously embedded the raw $exception->getMessage() directly into audit log context (ConfiguredSinkFailureHandler's five policy branches, and SwarmAuditDispatcher::routeToOutbox's outbox-unavailable warning and outbox-enqueue-failure error). Provider, driver, and tool exception messages can carry prompt fragments or PII, so these embeds bypassed the runtime's capture/redaction authority. A new SwarmCapture::auditExceptionMessage(\Throwable $e): string helper now routes every such embed through the single capture chokepoint: the raw message passes through only when capture already permits failure free-text (capturesFailures() — i.e. both swarm.capture.inputs and swarm.capture.outputs are on) or the operator has explicitly opted out via the new config key (below); otherwise it collapses to SwarmCapture::REDACTED ([redacted]). The exception class/type is always preserved for diagnosability and is never gated — the logUnknownPolicyAndSwallow path, which previously logged only the message with no class sibling, now also records class. SwarmCapture is injected into both audit classes via the established container DI (autowired for ConfiguredSinkFailureHandler; passed explicitly in the SwarmAuditDispatcher singleton binding).

  • New swarm.audit.redact_exception_messages config key (SWARM_AUDIT_REDACT_EXCEPTION_MESSAGES), default true (#275). Governs the redaction above. The safe-production default redacts audit-path exception messages unless capture already permits failure free-text. Set to false to always emit the raw message regardless of capture posture (e.g. trusted internal-only deployments that need full diagnostic detail in audit logs).

Changed

  • Finish the final readonly class / Collection idiom adoption on the response DTOs (#274). Pure internal polish, no behavior change. The three response DTOs that are already finalDrainResult, SwarmStep, and AuditDrainResult — are promoted to final readonly class (the per-property readonly keyword is dropped in favour of the class-level modifier). The non-final / extensible response DTOs (SwarmResponse, StreamedSwarmResponse, DurableRunDetail, DurableChildRun, DurableRetryPolicy, DurableSignalResult, DurableWaitOutcome, SwarmArtifact, HierarchicalRoutePlan, and the HierarchicalRouteNode hierarchy) are intentionally left as per-property readonly, because readonly class is breaking for an external non-readonly subclass — completing the convention there is a separate public-surface decision. Alongside it, a handful of array_values(array_filter(...)) / array_map(...->toArray()) round-trips across the response, tool, memory, and persistence paths are rewritten as collect(...)->filter()/->map()/->unique()->values()->all() chains; every site still returns an identical array (each chain ends in ->all()), so no public output type changes.
  • Audit sink-failure / signing log paths now redact exception messages by default — operator-visible behavior change (#275). Before this release, the free-text message of a sink, signing, or outbox exception was always written verbatim into the audit failure log context, regardless of swarm.capture.*. As of v0.14.0 those messages are redacted to [redacted] unless capture permits failure free-text or swarm.audit.redact_exception_messages is false. The exception class is still always logged, so failure diagnosability by exception type is unchanged. Operator note: if your monitoring or incident tooling greps audit failure logs for raw exception text, either enable capture for those runs or set SWARM_AUDIT_REDACT_EXCEPTION_MESSAGES=false to restore the prior verbatim behavior.
  • The @internal audit classes ConfiguredSinkFailureHandler and SwarmAuditDispatcher gained a required SwarmCapture constructor dependency (driving the redaction above); both are container-resolved, so consumers using the service container need take no action.
  • Confirm — and document with a concurrency test — that the durable run_state read-modify-write cannot lose an update (#273). DatabaseDurableRunStore::upsertRunState() reads the swarm_durable_run_state row with first(), merges the patch in PHP, and writes it back with updateOrInsert() — no lockForUpdate() and no surrounding transaction at that method. An audit flagged this as a possible lost-update window: if two writers ever read-merge-write one run's run_state row concurrently (e.g. one patching route_plan, another patching failure), the later write could clobber the earlier. Per the "durable fixes, not patches" rule this was treated as investigate-then-guard, not a blind lock. The investigation (a new tests/ProcessConcurrency/DurableRunStateConcurrencyTest.php, in the real-DB skip-locked-real-db lane) drives the only genuinely-unguarded interleavings — cancel() (the sole upsertRunState caller not gated by the execution-token lease) racing a lease-holding markFailed(), plus a lease-expiry overlap where a stale-token advancer races a worker that re-acquires the expired lease — at 24 concurrent runs per scenario through the Laravel process driver. Across MySQL 8 ×10 and PostgreSQL 17 ×5 (360 cancel-vs-advance + 360 lease-expiry races) it observed zero torn run_state rows. The window is never opened: every caller is serialized on the main swarm_durable_runs row before it reaches upsertRunState() — advance callers hold the atomically-acquired execution lease (guardedUpdate() in the same transaction; only one token is ever valid per run), and cancel() nulls that lease under the main-row write lock so a racing advancer's guardedUpdate() matches 0 rows and rolls back (its run_state write with it), while an advancer that already reached a terminal status makes cancel()'s status guard a no-op. Outcome: no code-path change — the serializing invariant is now documented in a code comment at upsertRunState(), with an explicit note that any future caller writing run_state without first taking the main-row lease/status guard must wrap the read-merge-write in transaction() + lockForUpdate(). No behavior or API change for consumers.

Fixed

  • Batch-hydrate the durable recovery sweeps to kill a per-row find() N+1 (#272). DatabaseDurableRunStore::recoverableWaitTimeouts() and parentsWaitingOnTerminalChildren() previously hydrated each candidate row with a per-row find() — and find() itself issues three queries (main row + run_state + node_states) — so a bounded sweep of N candidates fanned out to 1 + 3N queries. Both now route their candidate collection through the existing batched hydrateSweepRecords() helper (the same single-source-of-truth assembler find() and the recoverable() / dueRetries() / recoverableWaitingJoins() sweeps already use), which loads run_state and node_states for all candidates in one whereIn pass each: a sweep is now a constant three queries (candidate select + two side-table loads), and an empty candidate set short-circuits to the single candidate-select query with zero side-table round-trips. The hydrated output shape is byte-for-byte identical to the old find()-per-row path (asserted by new equality tests). Two intentional, acceptable behavior deltas come with the shared path: (1) a candidate run concurrently deleted between the candidate-select and hydration is no longer silently dropped (the old .filter() removed runs that vanished mid-sweep) — this now matches the rest of the sweep family, which never re-fetches; and (2) parentsWaitingOnTerminalChildren() JOINs children→parents with select('parents.*'), so a parent with N terminal children still yields N identical rows (no dedup was added — preserving the prior behavior). Internal performance fix; no public-surface or contract change.