diff --git a/rfcs/0018-readiness-conditions-and-providers.md b/rfcs/0018-readiness-conditions-and-providers.md new file mode 100644 index 00000000..50448f7a --- /dev/null +++ b/rfcs/0018-readiness-conditions-and-providers.md @@ -0,0 +1,676 @@ +--- +title: Readiness Conditions and Providers +authors: + - Gio +created: 2026-07-09 +last_updated: 2026-07-26 +status: draft +issue: +rfc_pr: https://github.com/openclaw/rfcs/pull/33 +--- + +# Proposal: Readiness Conditions and Providers + +## Summary + +Add an opt-in canonical, structured condition result around OpenClaw's existing +Gateway readiness evaluator. Existing startup, drain, channel, and event-loop +observations become core conditions; activated plugins may register bounded, +observational readiness providers; every condition identifies the runtime +subject it observed; and configured `/ready`, `/readyz`, health, status, and an +optional CLI project the same result. Without readiness +configuration or another separately accepted activation contract, the existing +Gateway decision path remains authoritative. + +This RFC does not define hosting profiles. Standard Hosting Profiles are a +separate proposal that may compose conditions from this RFC into named support +contracts. + +## Motivation + +OpenClaw already exposes Gateway `/ready` and `/readyz`. Their fixed evaluator +answers important questions about startup, channel runtime health, drain state, +and event-loop health, but it is not an extensible readiness contract: + +- existing observations do not share an iterable condition shape; +- a result cannot identify whether a failed fact belongs to the Gateway, + configuration generation, plugin, node, storage backend, or another subject; +- plugins cannot contribute bounded readiness observations; +- operators cannot promote a known plugin dependency to required; +- health and status can omit facts or describe them differently; and +- hosts must interpret legacy fields and add private scripts for missing facts. + +The result is avoidable ambiguity. A process may respond while its workspace is +full, startup dependencies are pending, the Gateway is draining, or a required +plugin-owned backend is unavailable. These are readiness questions, not new +liveness endpoints and not reasons for each host to invent a private protocol. + +The proposed invariant is: + +```text +core Gateway observations ++ core runtime observations ++ activated plugin-provider observations +-> one canonical readiness result +-> /ready, /readyz, health, status, optional CLI +``` + +Required `False` or `Unknown` conditions fail readiness. Advisory conditions +remain visible without causing an outage. Existing installations preserve their +current readiness path unless an operator adds `gateway.readiness` or selects a +separately accepted Standard Hosting Profile. + +### Evidence from existing issues + +| Issue | Observed gap | Contract implication | +| --- | --- | --- | +| [openclaw#96084](https://github.com/openclaw/openclaw/issues/96084) | `/readyz` remains healthy when a PVC-backed workspace is full. | Workspace writability needs a bounded readiness condition. | +| [openclaw#78136](https://github.com/openclaw/openclaw/issues/78136) | Docker readiness remains healthy while the command queue is draining. | Admission state must remain a required readiness input. | +| [openclaw#73652](https://github.com/openclaw/openclaw/issues/73652) | The Gateway accepts connections before internal startup is ready. | Startup completion needs a stable required condition. | +| [openclaw#78954](https://github.com/openclaw/openclaw/issues/78954) | Channel/plugin sidecars can block a usable core Gateway. | Readiness needs explicit required versus advisory classification. | +| [openclaw#101083](https://github.com/openclaw/openclaw/issues/101083) | A channel can appear healthy while retrying a fatal login error. | Channel runtime truth should use the canonical result. | + +## Goals + +- Define one stable readiness-condition shape and aggregation rule. +- Define one bounded identity package so conditions can reference core- and + plugin-owned subjects without copying identity fields into every condition. +- Normalize existing Gateway readiness observations into core conditions. +- Preserve existing response fields as compatibility projections. +- Add activation-scoped plugin readiness providers through the plugin SDK. +- Keep provider execution bounded, observational, enumerable, and fail-closed. +- Let operators explicitly select additional registered core or plugin + criteria as required or advisory without selecting a hosting profile. +- Project one result through HTTP readiness, health, status, and an optional + `openclaw ready` command. +- Add missing runtime status facts alongside conditions rather than creating a + parallel evidence store. +- Keep detailed readiness output authenticated or local while preserving the + compact unauthenticated probe response. + +## Non-Goals + +- Define standard or custom hosting profiles. +- Add profile selection, profile inheritance, or release profile conformance. +- Replace `openclaw.json` or create a second configuration system. +- Define OCC resources, placement, tenants, or an AgentHarness protocol. +- Make readiness depend on Doctor, policy, telemetry, or another optional + subsystem. +- Permit arbitrary runtime code, shell commands, or remote probes in config. +- Make readiness prove checkpoint durability, safe shutdown, compatibility, or + safe destruction. +- Standardize Docker, Kubernetes, or systemd probe intervals and retries. + +## Proposal + +The implementer-facing v1 contracts are captured in +[`0018/readiness-v1-spec.md`](0018/readiness-v1-spec.md) and the focused +[`0018/readiness-subjects-v1-spec.md`](0018/readiness-subjects-v1-spec.md) +sidecar. The non-normative +[`0018/readiness-platform-comparison.md`](0018/readiness-platform-comparison.md) +compares the contract with Kubernetes, Docker, systemd, ASP.NET Core, Spring +Boot, and OpenTelemetry and records the v1 gap disposition. This RFC remains +the design rationale, compatibility argument, and rollout plan; the normative +sidecars are the concise schema, identity, provider-lifecycle, evaluation, +projection, and conformance references for OpenClaw runtime and plugin +implementations. + +### Canonical condition model + +Every readiness observation is represented as: + +```ts +type ReadinessConditionStatus = "True" | "False" | "Unknown"; +type ReadinessRequirement = "required" | "advisory"; + +type ReadinessCondition = { + type: string; + subjectRef: string; + relatedSubjectRefs?: string[]; + observedAtMs?: number; + status: ReadinessConditionStatus; + requirement: ReadinessRequirement; + reason: string; + message: string; +}; + +type ReadinessResult = { + contractVersion: 1; + evaluatedAtMs: number; + identity: ReadinessIdentity; + ready: boolean; + conditions: ReadinessCondition[]; + failures: string[]; + advisories: string[]; +}; +``` + +`type` identifies the condition contract. The stable comparison key is +`(subjectRef, type)`. `reason` is a stable machine-readable state or failure +reason. `message` is redacted operator guidance and is not part of machine +matching. + +Aggregation is deliberately simple: + +| Requirement | Effect of `False` or `Unknown` | +| --- | --- | +| `required` | `ready=false`; reason appears in `failures`. | +| `advisory` | Overall readiness is unchanged; reason appears in `advisories`. | + +An unobserved required fact is `Unknown`, never inferred as `True`. Duplicate +condition identities, invalid statuses, or malformed provider results are +converted to stable `Unknown` conditions or reject provider registration; they +must not disappear from the result. + +### Subject identity + +Readiness is an aggregate over independently owned runtime objects. A Gateway +serving incarnation, active configuration generation, plugin activation, +paired node, model route, storage backend, sandbox, and harness do not share one +useful lifetime. The canonical result therefore declares subjects once and lets +conditions reference them: + +```ts +type ReadinessSubject = { + ref: string; + kind: string; + id?: string; + generation?: string; + parentRef?: string; +}; + +type ReadinessIdentity = { + producerRef: string; + subjects: ReadinessSubject[]; +}; +``` + +`subjectRef` names the primary object whose state the condition describes. +`relatedSubjectRefs` names bounded dependencies without changing condition +ownership. An aggregate condition uses an explicit aggregate subject and lists +its members as related subjects. Atomic per-subject conditions remain preferred +when individual failures matter. + +Core reserves `openclaw/*` references. A plugin declares local `kind` and `key` +values; core derives `plugin.//`, validates every +reference, collapses identical declarations, and rejects conflicting +declarations. Subject `ref` is the stable role used for comparison; `id` and +`generation` identify what currently occupies that role. Identity follows +owner renewal boundaries: an optional host workload may contain an OS process, +which may create one or more Gateway serving lifecycles; each child receives a +new ID when that lifecycle is replaced. A generation changes only when the +same owner object is revised. Host-supplied correlation is a fingerprinted host +subject and never overrides the generated Gateway identity. Subjects and +conditions are deterministically ordered. The focused identity and +reconciliation contract is normative in +[`0018/readiness-subjects-v1-spec.md`](0018/readiness-subjects-v1-spec.md). + +### Core conditions + +The first implementation normalizes the observations already owned by the +Gateway and adds generally applicable runtime facts where source evidence +justifies them. + +| Condition | Requirement | True when | Stable non-ready reasons | +| --- | --- | --- | --- | +| `GatewayStartupComplete` | Required | Startup dependencies and startup sidecars are no longer pending. | `GatewayStartupPending` | +| `GatewayAcceptingWork` | Required | The Gateway is not draining and can admit new work. | `GatewayDraining` | +| `ChannelRuntimeReady` | Required | No selected channel has an unsuppressed runtime-health failure under existing channel policy. | `ChannelRuntimeUnavailable` | +| `ChannelRuntimeSuppressed` | Advisory when present | A channel runtime failure is intentionally suppressed by existing autostart/crash-loop policy. | `ChannelRuntimeSuppressed` | +| `EventLoopHealthy` | Advisory by default; selectable | Existing event-loop health is within its healthy threshold. | `EventLoopDegraded`, `EventLoopStatusUnavailable`, `CriterionEvaluationUnavailable` | +| `ReadinessEvaluationComplete` | Required when emitted | The bounded canonical evaluation completed. This failure-only guard condition is emitted when the evaluator cannot produce its normal condition set. | `ReadinessEvaluationTimedOut`, `ReadinessEvaluationFailed` | +| `GatewayResponding` | Required when observed remotely | The current operation successfully reached the live Gateway. | `GatewayUnavailable`, `GatewayNotChecked` | +| `ConfigLoaded` | Required | The validated effective runtime config snapshot is installed. | `ConfigNotLoaded`, `ConfigInvalid`, `EffectiveConfigUnavailable` | +| `WorkspaceWritable` | Required or advisory when selected | The effective workspace exists and passes a bounded write, flush, and cleanup probe. It is not a new universal blocker by default. | `WorkspaceMissing`, `WorkspaceStorageFull`, `WorkspaceNotWritable`, `WorkspaceProbeFailed`, `WorkspaceProbeTimedOut`, `WorkspaceNotChecked` | +| `PluginsLoaded` | Advisory by default; selectable | The activation-pinned plugin registry is available and selected plugins have no activation errors. | `PluginLoadFailures`, `PluginStatusUnavailable`, `CriterionEvaluationUnavailable` | +| `ConfigCurrent` | Required or advisory when selected | The accepted effective config does not require restart. | `ConfigRestartRequired` | +| `ModelRouteReady` | Required or advisory when selected | The selected model route has usable provider authentication. | `ModelRouteUnavailable`, `ModelAuthUnavailable` | +| `SecretsReady` | Required or advisory when selected | Required secret owners resolved during activation. | `SecretOwnersUnavailable` | +| `SessionStorageReady` | Required or advisory when selected | Session storage passes its bounded write, flush, and cleanup probe. | `SessionStorageMissing`, `SessionStorageFull`, `SessionStorageNotWritable`, `SessionStorageProbeFailed`, `SessionStorageProbeTimedOut` | +| `ContextEngineReady`, `ToolCatalogReady`, `McpRuntimeReady`, `SandboxReady`, `HarnessReady` | Required or advisory when selected | Each configured execution capability has activation evidence from its owning subsystem. An intentionally unconfigured optional capability is satisfied. | Owner-specific unavailable, failed, or unobserved reasons. | +| `StateReady`, `DeliveryRuntimeReady`, `SchedulerReady` | Required or advisory when selected | Existing state and background-service snapshots report their current observed status. | Owner-specific unavailable, degraded, or unobserved reasons. | + +Changing an advisory core condition to required is a compatibility-sensitive +behavior change. It requires focused review and release notes because it can +change `/ready` from `200` to `503` for an existing deployment. + +These selectable core conditions consume bounded, redacted observations from +their owning subsystems. Readiness polling must not reload plugins, reacquire +secrets, issue a model request, connect MCP servers, start a sandbox or harness, +open the state database, or start a scheduler. + +### Readiness providers + +Only an activated OpenClaw plugin can register executable provider code. Core +conditions continue to use internal evaluators. Operators and control planes +may reference provider IDs; they cannot inject callbacks through config. + +```ts +type PluginReadinessResult = { + subjectRef?: string; + relatedSubjectRefs?: string[]; + observedAtMs?: number; + status: "True" | "False" | "Unknown"; + reason: string; + message: string; +}; + +type PluginReadinessProvider = { + id: string; + description: string; + check(context: { + config: OpenClawConfig; + pluginConfig: unknown; + signal: AbortSignal; + subjects: PluginReadinessSubjectCollector; + }): Promise | PluginReadinessResult; +}; + +type RegisterReadinessCriterion = ( + provider: PluginReadinessProvider, +) => void; +``` + +Example: + +```ts +api.registerReadinessCriterion({ + id: "backend", + description: "Reports whether the plugin backend can accept work.", + async check({ pluginConfig, signal, subjects }) { + const backend = subjects.declare({ + kind: "backend", + key: "primary", + identity: { id: "configured-backend" }, + }); + return (await probeBackend(pluginConfig, { signal })) + ? { + subjectRef: backend, + status: "True", + reason: "BackendReady", + message: "Backend is reachable.", + } + : { + subjectRef: backend, + status: "False", + reason: "BackendUnavailable", + message: "Backend is unreachable.", + }; + }, +}); +``` + +Core publishes the condition as `plugin..backend`. Registration is +bound to the activated plugin registry snapshot. Reload replaces the complete +provider set atomically with the next activation; stale callbacks do not remain +registered. + +Each invocation receives a fresh subject collector bound to the activated +plugin namespace. `declare` returns a canonical subject reference and records a +bounded declaration for reconciliation. A provider may reference documented +core subjects but cannot declare or replace them. If a provider omits +`subjectRef`, core assigns the criterion's default plugin-owned subject. Invalid, +unresolved, conflicting, or excessive declarations become +`CriterionInvalidResult=Unknown`; partial raw identity output is never +projected. + +The bundled Policy plugin is an optional example of this contract. Readiness +does not depend on Policy being installed or enabled. When activated, Policy +registers `plugin.policy.conformant`, reuses its existing policy evaluator, and +reports `True` when evaluation has no findings, `False` with a bounded finding +count when findings exist, and `Unknown` when policy checks are disabled. The +provider remains inert until an operator selects it. Selecting it as advisory +exposes drift without changing `/readyz` from `200`; selecting it as required +makes nonconformance block readiness. + +Provider descriptors are enumerable without invoking callbacks. The active +registry exposes provider identity, description, owning plugin, and source; the +registry snapshot itself is the activation-generation boundary. A read-only +catalog projection may expose the selectable core and plugin descriptors, +current required/advisory selection, and selected-but-unregistered IDs without +exposing source paths or executing providers. + +Providers must be: + +- read-only and observational; +- idempotent under repeated invocation; +- safe under concurrent invocation or protected by core coalescing; +- cancellation-aware; +- free of blocking synchronous I/O; and +- redacted by construction. + +Core owns namespacing, validation, invocation, deadlines, cancellation, +coalescing, caching, error conversion, and result ordering. A provider cannot +alter another provider's condition or any core condition. + +Provider `reason` values use a bounded machine-readable token grammar. Public +messages must be non-empty, contain no NUL bytes, and fit within 512 UTF-8 +bytes after core redaction. Invalid output becomes +`CriterionInvalidResult=Unknown`; raw provider output never bypasses these +checks. + +### Operator-selected readiness conditions + +OpenClaw's universal Gateway lifecycle conditions always apply and cannot be +removed. Beyond that baseline, an operator may explicitly select registered +core or plugin criteria through Gateway readiness config without selecting a +hosting profile: + +```json5 +{ + gateway: { + readiness: { + requiredCriteria: [ + "openclaw.workspace-writable", + "plugin.storage.backend", + ], + advisoryCriteria: [ + "openclaw.event-loop-healthy", + "plugin.metrics.exporter", + "plugin.policy.conformant", + ], + }, + }, +} +``` + +Provider criteria are advisory unless selected as required. Selector syntax is +validated in config. A syntactically valid ID that is not present in the active +registry produces `CriterionNotRegistered=Unknown` with its selected +requirement; it cannot be silently ignored. Configuration changes are applied +through the normal validated config lifecycle. This RFC does not add a policy +language or a way to redefine criteria semantics. + +This explicit list is a complete standalone use of RFC 0018. An operator can +say exactly which additional observations must pass for its deployment without +creating or selecting a profile. RFC 0023 adds reusable, named, release-tested +presets over the same selection mechanism. + +### Bounded evaluation + +Readiness is a hot operational endpoint and must return within code-owned +limits even when a provider is slow or broken. + +The initial implementation uses layered bounds: + +- a one-second deadline per plugin provider; +- a one-second deadline for the workspace probe; +- concurrent evaluation of independent observations; +- cancellation signals for cooperative providers; and +- an independent two-second outer watchdog for the complete result. + +A timeout becomes `Unknown` with a stable reason. A required timeout returns +`503`; an advisory timeout remains visible without blocking. The outer watchdog +fails closed with a required `ReadinessEvaluationComplete=Unknown` condition +rather than allowing `/ready`, `/readyz`, health, or status to hang or reject. +Unexpected error details are not copied into the public result. + +Core retains ownership of a provider invocation after its deadline. If a +provider ignores cancellation and remains pending, later readiness polls reuse +the stable timeout result and do not start another invocation. A new invocation +may begin only after the original callback settles and the result cache expires. +Publishing a replacement plugin registry or effective config snapshot aborts +the prior generation, clears its cache, and prevents late settlement from +entering the active result. + +Workspace probes use the same generation fence but permit one replacement +probe when the effective workspace changes, so a blocked retired filesystem +does not pin the new workspace. At most two workspace probes may remain in +flight; further generations fail closed until capacity returns. + +In-process timers cannot interrupt synchronous JavaScript that blocks the event +loop. Providers therefore may not perform blocking synchronous I/O. Process or +worker isolation for malicious plugins is outside this RFC. + +### Canonical projections + +`/ready` and `/readyz` remain the authoritative host probes. Authenticated or +local callers receive the structured result; unauthenticated remote probes +retain a redacted boolean response. `HEAD` behavior remains unchanged. + +The same result is projected through Gateway health and status. A surface that +did not observe a required live fact reports `Unknown`; it does not synthesize +success. `/health` and `/healthz` remain shallow liveness and do not acquire +readiness semantics. + +An optional `openclaw ready` command may be a thin client of the live Gateway +result: + +- human output lists all conditions and identifies the existing lifetime and + generation of primary and related subjects behind non-`True` conditions; +- `--json` preserves the canonical result; and +- exit status is nonzero for required failure, required unknown, transport + failure, or a missing readiness contract. + +The CLI must not implement a second evaluator. + +### Compatibility + +Canonical readiness evaluation is opt-in. When `gateway.readiness` and any +separately accepted activation contract are both absent, the Gateway invokes +only its legacy lifecycle and channel checker; provider or runtime-evaluator +failure cannot create a new `503`. Presence of `gateway.readiness`, including +an empty object, activates bounded canonical evaluation. A separately accepted +Standard Hosting Profile may also activate canonical evaluation as part of +profile selection. + +Existing `ready`, `failing`, `suppressed`, `eventLoop`, and `uptimeMs` fields +remain compatibility projections during migration. New consumers use +`conditions`, `failures`, and `advisories`. + +The migration order is: + +1. preserve the legacy decision path when canonical readiness is not activated; +2. emit canonical conditions beside legacy fields after activation; +3. make every activated projection consume the same canonical evaluator; +4. migrate internal and external consumers; and +5. consider legacy-field removal only through a separate compatibility review. + +Activated installations gain structured output but do not gain new required +workspace or plugin dependencies. Registering or implementing an additional +criterion does not make it required. Only explicit operator configuration or a +separately selected standard profile changes the additional readiness gate. + +### Readiness transitions + +A later implementation may emit bounded, redacted events when overall +readiness or a condition changes. Event names, initial observation, +deduplication, restart behavior, and behavior without active polling require a +separate review. Telemetry sinks, dashboards, alerting, and retention remain +host responsibilities. + +### Relationship to Standard Hosting Profiles + +Standard Hosting Profiles depend on this RFC rather than sharing its acceptance +decision. A profile is a named preset of required and advisory criteria plus a +support promise. It cannot change condition evaluation, provider lifecycle, or +aggregation. Operators that do not need that support contract use +`gateway.readiness` directly. + +This separation allows OpenClaw to accept structured, subject-aware readiness +and provider extensibility without committing to profile names, profile +selection config, or a release support matrix. Profiles reuse the identity +package rather than adding a profile-only activation envelope. + +### Implementation plan + +The RFC has two direct upstream implementation PRs, in landing order: + +1. [openclaw/openclaw#104018](https://github.com/openclaw/openclaw/pull/104018) + at exact head `2f131c6e220` adds the opt-in canonical evaluator, bounded + providers, subject identity, shared HTTP/RPC/status/health projections, and + the `openclaw ready` CLI. Unconfigured probes remain on the legacy checker. +2. [openclaw/openclaw#113421](https://github.com/openclaw/openclaw/pull/113421) + at exact head `c1d7f394f86` adds selectable core-owner observations for + runtime activation, execution capability, session storage, state, delivery, + and scheduling. None becomes required by being implemented. + +Focused readiness, live Gateway, HTTP/RPC, health, selector, registry, subject, +CLI, and owner-adoption suites pass; type-aware lint, formatting, and diff +checks pass. Timed-out plugin checks remain single-flight until the original +callback settles, including across registry replacement when a plugin ignores +cancellation. Provider output is bounded, validated, and redacted. Config +publication fences provider and workspace evidence by runtime generation, +including recovery when a retired filesystem probe never settles while +retaining a strict two-probe ceiling across repeated generation changes. + +The +[exact-head package-installed Docker matrix](https://github.com/giodl73-repo/openclaw/actions/runs/30289122192) +proves the RFC 0018 surfaces as part of the dependent profile stack: `/ready` +and `/readyz` transition `200 -> 503 -> 200`, `/healthz` remains live, +`openclaw ready --json` exits `0 -> 1 -> 0`, repeated polls retain identity, +and container restart preserves host identity while renewing process and +Gateway lifetimes. Standalone published-upgrade proof remains separate. + +#### Operator facilities roadmap + +The canonical result should replace private polling, JSON-diff, and startup +scripts with a small set of ordinary operator facilities. These are follow-on +consumers of this RFC, not additions to the evaluation model: + +1. **Observe transitions.** Add `openclaw ready --watch` over the existing RPC. + Poll sequentially with a timeout per call, emit an initial snapshot followed + by semantic changes, suppress timestamp-only churn, identify condition + changes by `(subjectRef, type)`, and report producer or subject-lifetime + replacement. Continue through not-ready and unavailable states so recovery + remains observable. `--json` emits versioned JSON Lines. +2. **Inspect and wait.** Add read-only criterion/provider enumeration and a + bounded startup wait convenience over the same canonical result. Neither + facility may create a second evaluator. The catalog should use + `readiness.catalog` plus `openclaw ready criteria list|inspect`, reading one + activation-pinned config/registry snapshot without invoking a provider. + Startup waiting should use `openclaw ready --wait [duration]`. It polls + sequentially, bounds each call by the remaining total budget, independently + aborts slow connection setup at the deadline, and emits only the final + canonical observation. Because waiting belongs to `ready`, Docker, + Kubernetes, systemd, OCC, and local launchers can use the same facility. + The human diagnostic projection should show the existing kind, ID, + generation, and parent reference for primary and related subjects behind a + non-`True` condition. Healthy human output and canonical JSON remain + unchanged. +3. **Diagnose and operate.** Let existing Doctor, telemetry, support-bundle, + and update owners consume snapshots and transitions for remediation, + secret-safe evidence capture, transition telemetry, and pre/post-upgrade + comparison. Readiness remains observational and does not absorb those + subsystems. + +Profile discovery, validation, conformance artifacts, and release gates belong +to the separate Standard Hosting Profiles RFC. + +Profile selection, node-mode composition, profile subject attribution, and packaged +profile release conformance move to the Standard Hosting Profiles RFC and its +separate implementation stack. + +The earlier consolidated draft remains useful behavior evidence, but it is not +the proposed landing shape. The slices above can land without accepting profile +names, profile selection, profile subject attribution, or release conformance. + +#### Follow-on condition adoption + +The framework should not grow through one PR per condition. Follow-on adoption +is grouped by the OpenClaw owners that already maintain the underlying runtime +facts. Each bucket is one reviewable implementation unit; individual conditions +may be deferred when their owner does not yet expose a bounded activation +snapshot. + +| Bucket | Candidate conditions | Existing owner evidence | Proposed PR scope | +| --- | --- | --- | --- | +| Runtime activation integrity | `SecretsReady`, `ModelRouteReady`, `ConfigCurrent`, selectable `PluginsLoaded`, selectable `EventLoopHealthy` | Degraded secret-owner snapshots, model-auth and route resolution state, runtime config/reloader state, plugin activation and configured-unavailable state, Gateway event-loop snapshot | Project activation-pinned facts and existing canonical Gateway facts into selectable or advisory conditions. Do not resolve secrets, contact a model, reload plugins, or create a second event-loop probe during readiness evaluation. | +| Agent execution capabilities | `ContextEngineReady`, `ToolCatalogReady`, `McpRuntimeReady`, `SandboxReady` or `HarnessReady`; defer `SkillsReady` until bounded owner evidence exists | Context-engine and tool-schema quarantine records, MCP runtime ownership, sandbox and harness runtime state | Let each selected execution owner publish a bounded condition. Optional capabilities remain advisory unless explicitly selected as required. Do not rebuild the skill inventory in the readiness path. | +| State and background services | `StateReady`, `SessionStorageReady`, `DeliveryRuntimeReady`, `SchedulerReady`; defer `RestoreComplete` until a restore fence exists | State/store activation, bounded persistence-location probes, delivery runtime ownership, cron lifecycle and startup recovery state | Report whether configured stateful services can accept new work. A selected storage probe may perform a capped write/fsync/unlink check, but historical dead letters and individual job failures remain diagnostics or advisories rather than universal blockers. Do not infer restore completion from database-open, fleet-restore, or config-migration state. | +| Hosted dependencies | `HostBindingsReady`; defer `EgressReady` and `ManagedConfigApplied` until their owners publish authoritative facts | RFC 0020 host-integration bundle and owner-generation evidence | Project required host bindings through the ordinary readiness selector. Keep network probing, config inference, Lobster, OCC, tenant, and deployment policy out of the condition evaluator. | + +Validated prototypes cover all four buckets. The first three depend only on +the primary readiness implementation. The fourth also depends on RFC 0020 host +integration package 3. They remain optional follow-on work and are not +prerequisites for accepting this RFC: + +- Runtime activation integrity adds selectable `ConfigCurrent`, + `ModelRouteReady`, and `SecretsReady` + conditions plus quarantine-aware plugin activation evidence. +- Agent execution capabilities adds selectable context-engine, tool-catalog, + MCP, sandbox, and harness + conditions. MCP discovery is captured for every configured agent when the + Gateway accepts a runtime configuration; readiness evaluation does not + connect to MCP servers or start any execution surface. `SkillsReady` remains + deferred because the current skill inventory path is not a bounded readiness + source; the skills owner must first publish an activation snapshot or index. +- State and background services adds selectable state-database, + session-storage, durable session-delivery, + and scheduler lifecycle conditions. Owners publish synchronous process-local + snapshots. The selected storage condition uses a one-second, target-capped, + concurrency-capped, generation-safe write/fsync/unlink probe over resolved + persistence parents; failures are redacted. Readiness does not open SQLite, + install delivery recovery, import or start cron, scan queues, or run recovery. + `RestoreComplete` remains deferred until OpenClaw owns a generic restore + generation and admission fence. +- Hosted dependencies adds selectable `openclaw.host-bindings-ready` and + connects RFC 0020 bundle + declarations to the active core/plugin criterion catalog. Selecting the + aggregate includes referenced criteria as advisory detail without requiring + operators to repeat the bundle's selectors. A required contribution fails + the aggregate for missing, incompatible, unresolved, stale, degraded, or + unavailable owner state, and when a declared criterion is absent, `False`, + or `Unknown`; optional contributions remain visible without blocking it. + Status and Doctor name unresolved selectors, recursive aggregate references + are prohibited, and manifests are capped at 64 unique selectors. Evaluation + uses one bounded in-memory bundle/evidence snapshot. Real endpoint coverage + proves `/ready` returns `200 -> 503 -> 200` across owner-generation failure + and recovery. Generic `EgressReady`, `ManagedConfigApplied`, and + `RestoreComplete` remain deferred until their owners publish authoritative + facts. + +The capability slice passes 14 focused assertions, the state/background slice +passes 145 focused assertions, and the hosted-binding composition passes 142 +focused assertions including the real Gateway endpoint transition. The first +two pass production and source-test typechecks; all three pass lint, formatting, +diff checks, and independent review. Capability review +found and fixed an initial default-agent-only MCP snapshot so the final evidence +covers all configured agent workspaces. State/background review verified +disabled, starting, recovery-pending, active, stopped, failed, repaired, and +stale-generation transitions without adding probe-side I/O. + +These buckets are adoption work over the v1 framework, not prerequisites for +accepting it. They do not change the v1 aggregation algorithm or make a newly +implemented condition required by default. A condition becomes required only +through explicit `gateway.readiness` selection or a separately accepted Hosting +Profile composition. + +Every adoption PR must preserve the owner boundary: + +1. expose or reuse a redacted, activation-scoped status fact; +2. map that fact to `True`, `False`, or `Unknown` without mutation; +3. bound evaluation through the v1 provider and outer-evaluator deadlines; +4. test ready, failed, unknown, timeout, recovery, and generation replacement; +5. prove that HTTP, RPC, health, status, and CLI project the same result. + +Readiness evaluation must not perform model inference, credential acquisition, +unbounded network discovery, migration, repair, restore, or queue draining. +Those operations remain owner workflows; readiness only observes their current +activation state. + +## Rationale + +This extends a surface OpenClaw already owns. It does not add a new health +service, policy engine, or hosted envelope. Conditions make the current +evaluator explainable; providers give plugin-owned dependencies a bounded home; +and canonical projections prevent HTTP, status, health, and CLI from drifting. + +Keeping providers advisory by default is the critical compatibility rule. A +plugin can improve diagnostics without gaining the power to take down the +Gateway. Operators retain the explicit decision to make a dependency required. + +Separating profiles keeps this RFC independently useful and easier to accept. +OpenClaw can modernize readiness first, then decide separately whether named +runtime postures should become release-tested support promises. + +## Unresolved questions + +- Is `gateway.readiness` the correct config home for operator-required provider + IDs, or should requirement selection use another existing Gateway surface? +- Which existing advisory observations, if any, should become required in the + first compatibility-reviewed implementation? +- Should any non-plugin core owner need a public registration API, or should + core conditions remain internal evaluators? +- What cache lifetime best balances probe cost and freshness for plugin + providers? diff --git a/rfcs/0018/readiness-platform-comparison.md b/rfcs/0018/readiness-platform-comparison.md new file mode 100644 index 00000000..0196e2a9 --- /dev/null +++ b/rfcs/0018/readiness-platform-comparison.md @@ -0,0 +1,190 @@ +# Readiness Identity and Hosting Platform Comparison + +Status: non-normative design evidence for RFC 0018 +Last updated: 2026-07-26 + +## Purpose + +This sidecar compares RFC 0018 with established readiness and runtime identity +contracts. It tests whether OpenClaw is introducing unnecessary concepts and +records which adjacent-system features belong in readiness v1, in a host, or +in a later transport contract. + +The comparison is semantic rather than lexical. RFC 0018 is an in-process +hostee contract, not a workload orchestrator, telemetry backend, or control +plane. + +## Summary + +RFC 0018 combines two patterns that mature systems usually expose separately: + +1. a compact decision suitable for a container or service manager; and +2. structured current-state facts with enough identity to explain what was + evaluated and whether that object was revised or replaced. + +Kubernetes is the closest identity analogue. ASP.NET Core and Spring Boot are +the closest provider-composition analogues. Docker and systemd demonstrate why +the host-facing decision must remain small and bounded. OpenTelemetry confirms +the value of opaque instance identity and the danger of using dynamic identity +as unbounded metric dimensions. + +No comparison justifies another required v1 field. The main actionable result +is an atomic-observation invariant: one readiness result must not combine a +condition from one owner generation with identity from another. + +## Comparison Matrix + +| System | Composition | Identity and lifetime | Host-facing decision | RFC 0018 disposition | +| --- | --- | --- | --- | --- | +| Kubernetes | Container probes, Pod conditions, and readiness gates contribute to readiness. | Object name is a stable role; UID identifies one historical occurrence; generation and observed generation distinguish desired-state revision from observed status. | Readiness removes a Pod from service endpoints; liveness may restart it; startup delays both. | Reuse the condition and replacement/revision distinctions. Do not copy Kubernetes API storage, watch, or controller semantics. | +| Docker | One `HEALTHCHECK` command produces one container health state; the engine supplies interval, timeout, start period, and retries. | Container identity and restart history are engine-owned, outside the health payload. | `starting`, `healthy`, or `unhealthy`, plus bounded recent probe output. | Keep `/readyz` compact. Let Docker own scheduling and retries while OpenClaw explains the internal decision. | +| systemd | A service can notify readiness, status, process identity, reload, stop, and watchdog progress. | The service manager owns unit and process lifetime. | `READY=1` is a concise transition consumed by the manager. | Keep supervision host-owned. OpenClaw conditions add detail behind the service-level decision rather than replacing the manager. | +| ASP.NET Core Health Checks | Registered checks can be selected by tags and grouped into readiness and liveness endpoints. | Check registration names components, but the standard result does not model replacement identity or parent lifetime. | Aggregated `Healthy`, `Degraded`, or `Unhealthy` results map to endpoints. | Reuse selectable providers and required/advisory aggregation; retain stronger execution bounds and subject identity. | +| Spring Boot Actuator | Health contributors form nested components and configurable health groups, including readiness and liveness groups. | Component paths are stable names; incarnation and semantic generation are not first-class health fields. | Group status is exposed through probe endpoints such as `/readyz`. | Reuse owner-contributed components and named compositions. Avoid pulling external dependencies into liveness or making every contributor required. | +| OpenTelemetry | Resources and entities attach identity to telemetry from services, processes, containers, and orchestrators. | `service.instance.id` distinguishes concurrently running service instances and should be opaque and unambiguous. | OpenTelemetry reports observations; it does not decide traffic admission. | Allow bounded diagnostic correlation. Do not turn subject IDs, generations, or node references into metric labels or authority identities. | + +## Kubernetes Mapping + +Kubernetes provides the strongest precedent for separating a logical role from +the object currently occupying it: + +| RFC 0018 | Closest Kubernetes concept | Important difference | +| --- | --- | --- | +| Subject `ref` | Resource name and scope | A readiness reference is a diagnostic role within one result, not an API resource URL. | +| Subject `id` | Object UID | Both change when an object is replaced. RFC 0018 IDs are owner-scoped opaque values, not cluster-global resource identifiers. | +| Subject `generation` | `metadata.generation` plus status `observedGeneration` | RFC 0018 generation is an owner-defined revision of the active object, not a desired-state counter maintained by an API server. | +| Subject `parentRef` | Owner/dependent or workload hierarchy | RFC 0018 hierarchy expresses diagnostic lifetime and correlation only. It does not imply garbage collection, placement, or authorization. | +| `(subjectRef, condition.type)` | Condition type on one object | RFC 0018 makes the subject explicit because one Gateway result aggregates independently owned runtime objects. | +| `status`, `reason`, `message` | Kubernetes condition fields | RFC 0018 adds `requirement` so advisory degradation remains visible without removing traffic. | + +Kubernetes `resourceVersion` is intentionally not an analogue for RFC 0018 +generation. It is an opaque API-server concurrency and watch token. RFC 0018 +has no object store or watch protocol in v1. + +Kubernetes also validates the separation among startup, readiness, and +liveness. OpenClaw's shallow health/liveness surface answers whether the +process can respond. Canonical readiness answers whether the selected runtime +facts permit new work. Restart, retry, grace-period, and traffic-routing policy +remain host responsibilities. + +## Application Framework Mapping + +ASP.NET Core and Spring Boot show that reusable, owner-registered checks are a +normal application framework facility. Their filtering, tags, health groups, +contributors, and nested components are close to RFC 0018 criteria, selectors, +providers, and subjects. + +RFC 0018 is intentionally stricter at the plugin boundary: + +- registration is activation-scoped and enumerable; +- configured unknown criteria fail closed; +- provider work is observational and cancellation-aware; +- per-provider and outer deadlines bound response time even when a callback + ignores cancellation; +- cardinality, strings, relationships, and output are bounded; +- malformed or incomplete output becomes structured `Unknown`; and +- one reconciled identity package makes object replacement visible. + +Those constraints are necessary because OpenClaw plugins are a broader and more +dynamic extension surface than ordinary application-local dependency injection. + +## Container and Service-Manager Mapping + +Docker and systemd consume a small answer and own the reaction. That is the +correct boundary for OpenClaw too: + +```text +OpenClaw owners and plugins -> canonical current-state conditions + -> one bounded ready decision +Docker / Kubernetes / systemd -> polling, retries, routing, restart, rollout +``` + +The host does not need to understand every OpenClaw subsystem to decide whether +to send work. OpenClaw does not need to own container IDs, restart policy, probe +cadence, or deployment history. Detailed authenticated or local output explains +the same decision used by the compact probe; it must not become a second result. + +## Telemetry Mapping + +OpenTelemetry's `service.instance.id` supports the RFC's choice of opaque +incarnation IDs. It also highlights an integration rule: identity is useful +only when its producer and lifetime are unambiguous. A host correlation value +may add an optional fingerprinted host subject, but it cannot replace the +Gateway-generated serving-incarnation identity. + +Signals, logs, and support bundles may preserve bounded subject IDs and +generations for diagnostic joins. Metrics may use bounded dimensions such as +condition type, status, requirement, reason, and subject kind. Dynamic IDs, +generations, node references, messages, and related-subject lists must not +become metric labels. + +## Gap Assessment + +### Required for v1 + +- **Atomic observation binding.** Every emitted condition and related subject + must resolve against one reconciled evaluation snapshot. If an owner or + plugin activation changes during evaluation, generation fencing must discard + the late result or fail the evaluation closed rather than mix lifetimes. +- **Lifecycle replacement proof.** Conformance must prove stable identity + across repeated polls and changed IDs across process, Gateway, plugin, node, + and owner-resource replacement at their documented renewal boundaries. +- **Incomplete-result diff safety.** Consumers must not infer success, + deselection, or deletion from an absent condition when + `ReadinessEvaluationComplete` is `False` or `Unknown`. +- **Projection consistency.** `/ready`, `/readyz`, authenticated health/status, + and the optional CLI must expose the same observed decision and identity. + +### Deliberately host-owned + +- probe cadence, timeout, retry threshold, startup grace period, and restart; +- traffic removal, draining orchestration, rollout, and rollback; +- retained history, fleet indexing, alerting, and tenant routing; and +- authorization identity, workload placement, and control-plane resource IDs. + +### Deferred until a transport requires them + +- `lastTransitionTime`, because OpenClaw does not retain readiness history; +- a `resourceVersion`, ETag, monotonic sequence, or watch cursor, because v1 is + a current polling contract rather than a list/watch API; +- a condition freshness TTL, because synchronous evaluation is bounded and an + incomplete evaluation already fails closed; stored projections must apply + their own staleness policy; and +- a generic lifecycle phase, because explicit conditions describe startup, + serving, degradation, and drain without creating an ambiguous state machine. + +If OpenClaw later adds cached remote reads, a readiness watch, or retained +transition events, that transport should define cursor, freshness, and +transition semantics without changing the meaning of the v1 current result. + +## Validation Consequences + +The implementation and release proof should include this matrix: + +| Transition | Expected identity behavior | Expected decision behavior | +| --- | --- | --- | +| Repeated poll, no owner change | Same refs, IDs, and generations | Same condition keys; timestamps may advance. | +| Condition failure and recovery | Same subject ID and generation unless the owner revised | Required `200 -> 503 -> 200`; advisory remains `200`. | +| Config revision in one Gateway lifecycle | Same Gateway ID; changed config generation | Conditions bind to the new reconciled generation only. | +| Gateway dispose and restart in one process | New Gateway ID; stable process ID | No late condition from the prior Gateway appears. | +| Process or container replacement | New process ID; host identity follows its documented boundary | Host decides restart and routing behavior. | +| Plugin reload or node replacement | New owner ID or generation according to that owner's contract | Late provider output is discarded or represented as incomplete. | + +The +[exact packaged container matrix](https://github.com/giodl73-repo/openclaw/actions/runs/30289122192) +proves this restart boundary on dependent profile head `18c42a7f26a`: +the same host workload retains its fingerprinted host identity while process +and Gateway identities renew. Unit tests additionally establish schema and +generation fences. + +## Primary Sources + +- [Kubernetes object names and UIDs](https://kubernetes.io/docs/concepts/overview/working-with-objects/names/) +- [Kubernetes API concepts and `resourceVersion`](https://kubernetes.io/docs/reference/using-api/api-concepts/) +- [Kubernetes API conventions for conditions and generations](https://github.com/kubernetes/community/blob/main/contributors/devel/sig-architecture/api-conventions.md) +- [Kubernetes liveness, readiness, and startup probes](https://kubernetes.io/docs/concepts/workloads/pods/probes/) +- [Dockerfile `HEALTHCHECK`](https://docs.docker.com/reference/dockerfile/#healthcheck) +- [systemd service notification protocol](https://www.freedesktop.org/software/systemd/man/latest/sd_notify.html) +- [ASP.NET Core health checks](https://learn.microsoft.com/en-us/aspnet/core/host-and-deploy/health-checks) +- [Spring Boot Actuator endpoints and Kubernetes probes](https://docs.spring.io/spring-boot/reference/actuator/endpoints.html#actuator.endpoints.kubernetes-probes) +- [OpenTelemetry service resource conventions](https://opentelemetry.io/docs/specs/semconv/resource/service/) diff --git a/rfcs/0018/readiness-subjects-v1-spec.md b/rfcs/0018/readiness-subjects-v1-spec.md new file mode 100644 index 00000000..1671fddd --- /dev/null +++ b/rfcs/0018/readiness-subjects-v1-spec.md @@ -0,0 +1,292 @@ +# Readiness Subjects v1 Specification + +This document is the focused identity and reconciliation specification for RFC +0018. The parent readiness specification defines condition evaluation, +aggregation, and projections. This sidecar defines how one canonical readiness +result identifies the producer and the independently owned subjects observed by +its conditions. + +Status: draft, tied to RFC 0018. + +## Goals + +- Make repeated readiness results safely diffable. +- Identify which runtime object owns each condition. +- Let multiple conditions reference one subject without copying identity data. +- Let a condition identify bounded related dependencies. +- Let core and plugins contribute subjects without global-name collisions. +- Preserve owner-defined identity and generation semantics. + +This specification does not create a general OpenClaw resource store, require a +hosting profile, define tenant identity, retain readiness history, or authorize +providers to discover arbitrary remote resources. + +## Data Model + +```ts +type ReadinessSubject = { + ref: string; + kind: string; + id?: string; + generation?: string; + parentRef?: string; +}; + +type ReadinessIdentity = { + producerRef: string; + subjects: ReadinessSubject[]; +}; + +type ReadinessCondition = { + type: string; + subjectRef: string; + relatedSubjectRefs?: string[]; + observedAtMs?: number; + status: "True" | "False" | "Unknown"; + requirement: "required" | "advisory"; + reason: string; + message: string; +}; + +type ReadinessResult = { + contractVersion: 1; + evaluatedAtMs: number; + identity: ReadinessIdentity; + ready: boolean; + conditions: ReadinessCondition[]; + failures: string[]; + advisories: string[]; +}; +``` + +`evaluatedAtMs` is the Unix epoch time when core assembled the canonical result. +`observedAtMs` is optional and records when the owning subsystem captured the +underlying fact. An omitted `observedAtMs` means the observation was made during +the current evaluation or its capture time is unavailable; it must not be +invented from cache insertion time. + +## References And Identity + +`ref` is a canonical stable role within readiness results. `id` identifies the +current object occupying that role. `generation` distinguishes owner-defined +revisions of the same object. Consumers compare subjects by `ref`, then inspect +`id` and `generation` to detect replacement or revision. + +Examples: + +```text +openclaw/process/current +openclaw/gateway/current +openclaw/config/active +openclaw/node/desktop-7 +plugin.storage/backend/primary +``` + +Core references use the reserved `openclaw/` prefix. Plugin references use: + +```text +plugin.// +``` + +Plugin `kind` and `key` values are trimmed, lowercased, and must match +`^[a-z0-9][a-z0-9._-]{0,63}$`. Core constructs the reference; providers do not +submit a global reference for declarations. Subject `kind` is the canonical +owner-qualified kind, for example `plugin.storage.backend`. + +`id` and `generation`, when present, are opaque bounded values. Consumers must +not parse them or infer security authority from them. An ID is tied to the +renewal boundary of the object named by `kind`: it stays stable while that +object exists and changes when its owner replaces the object. A generation is +an owner-defined revision of that same object and changes without implying +replacement. Owners must omit either field when they cannot publish an +authoritative value. IDs are not credentials and must not contain secrets, +tenant content, credential-bearing paths, or raw connection strings. + +## Producer + +`producerRef` identifies the subject that assembled and owns the readiness +decision. A live Gateway result uses its Gateway serving-incarnation subject. +That subject begins when one Gateway serving lifecycle initializes and ends when +that lifecycle is disposed. A subsequent Gateway start, including one in the +same OS process, receives a new ID. Config reload, condition transition, drain, +and repeated evaluation do not change the Gateway subject ID. + +The standard lifecycle hierarchy is: + +```text +openclaw/host-instance/current (optional host workload lifecycle) + -> openclaw/process/current (OS process lifecycle) + -> openclaw/gateway/current (Gateway serving lifecycle) +``` + +OpenClaw generates opaque random process and Gateway IDs. A host may provide a +stable workload-correlation input for the optional host subject, but core +publishes only a one-way fingerprint of that value. It does not replace the +process or Gateway identity. Reusing one host workload across process or +Gateway restarts therefore retains the parent host ID while renewing the child +IDs at their actual lifecycle boundaries. + +A non-live diagnostic projection may use its own producer subject and may +declare an unresolved Gateway subject without an `id`. It must report live +Gateway facts as `Unknown`, not borrow a stale Gateway identity. + +## Condition Subjects + +Every condition has exactly one `subjectRef`. The pair +`(subjectRef, condition.type)` is unique within one canonical result and is the +stable comparison key across results. + +`relatedSubjectRefs` is an optional bounded list of other subjects involved in +the observation. The primary subject remains accountable for the condition. +For example, `ModelRouteReady` may target a route and relate its selected model +and credential owner. + +An observation over a collection uses an explicit aggregate subject: + +```json +{ + "type": "NodeModeCapacityReady", + "subjectRef": "openclaw/node-set/scout-desktops", + "relatedSubjectRefs": [ + "openclaw/node/desktop-7", + "openclaw/node/desktop-9" + ] +} +``` + +When individual failures matter, an owner registers separate criteria or emits +bounded owner-native core conditions. One plugin readiness provider emits one +condition. Providers that observe an unbounded collection must use an aggregate +subject and may include only a deterministic bounded subset of related subjects. + +## Plugin Subject Collector + +Each selected plugin-provider invocation receives a fresh collector bound to +the activated plugin ID: + +```ts +type PluginReadinessSubjectCollector = { + declare(input: { + kind: string; + key: string; + identity?: { + id?: string; + generation?: string; + }; + parentRef?: string; + }): string; +}; +``` + +`declare` validates the local name, constructs and returns the canonical `ref`, +and records the declaration for the current invocation. Providers use that +returned value as `subjectRef` or `relatedSubjectRefs`. A provider may reference +documented core subjects and may parent its subjects to its own or a documented +core subject. It may not declare, replace, mutate, or parent into another +plugin's namespace. Declarations are order-independent: a plugin-local parent +may be declared before or after its child, and the completed graph is validated +after the provider returns. Externally supplied identity values are published +only as one-way fingerprints. + +If a provider emits no explicit primary subject, core assigns: + +```text +plugin./criterion/ +``` + +This preserves a stable subject for simple providers while allowing richer +providers to identify their actual backend, route, or integration. + +## Reconciliation + +Core reconciles declarations into one map keyed by canonical `ref` before +projection: + +1. Validate all references, kinds, IDs, generations, and parent references. +2. Reject declarations outside the caller's namespace. +3. Merge compatible declarations for the same `ref` when one declaration only + fills an absent `id`, `generation`, or `parentRef`; this lets an owner enrich + a core placeholder without changing its canonical reference. +4. Reject different kinds or conflicting non-empty identity fields for the + same `ref`; never use last-writer wins. +5. Require every primary, related, parent, and producer reference to resolve. +6. Omit unreferenced subjects except the producer and its parent chain. +7. Sort subjects by `ref`, related references lexicographically, and conditions + by the ordering in the parent readiness specification with `subjectRef` as a + deterministic tie-breaker. + +The complete identity package and all condition references must come from one +reconciled evaluation snapshot. If an owner identity, owner generation, or +plugin activation changes while a condition is in flight, the evaluator must +discard that late result or fail the evaluation closed. It must not attach an +observation from one owner generation to the identity of another. + +Invalid plugin declarations or references convert that provider's condition to +`CriterionInvalidResult=Unknown` on its default subject. Invalid core-owned +identity state fails the outer evaluation closed with +`ReadinessEvaluationComplete=Unknown`. No partial invalid identity package may +be projected or cached. + +## Bounds + +V1 enforces these maximums per canonical result: + +- 256 conditions, failures, and advisories; +- 128 subjects; +- 64 plugin-declared subjects per provider invocation; +- 16 related subjects per condition; +- 192 characters per canonical reference; +- 128 characters each for `kind`, `id`, and `generation`; and +- one parent reference per subject with no cycles. + +Provider deadlines and outer readiness deadlines include declaration and +reconciliation work. Subject declaration must perform no I/O. The wire schema +enforces these collection and string bounds before a result is accepted. + +## Diff Semantics + +Consumers may compare canonical results as follows: + +- changed `producerRef` target ID means a different readiness authority; +- unchanged `ref` with changed `id` means the role has a different object; +- unchanged `id` with changed `generation` means the owner advanced that + object's revision; +- unchanged `(subjectRef, type)` with changed status or reason means the same + subject's condition transitioned; and +- when `ReadinessEvaluationComplete` is absent or `True`, disappearance of a + condition means its criterion is no longer selected or its owner is no longer + active, not that it became `True`; and +- when `ReadinessEvaluationComplete` is `False` or `Unknown`, consumers must + treat missing conditions as unobserved and must not infer deselection, + deactivation, or success. + +OpenClaw need not retain result history. Hosts and telemetry consumers may store +and diff bounded canonical results. Metrics may label stable condition type, +status, requirement, reason, subject kind, and selected profile. Dynamic `id`, +`generation`, node-specific `ref`, `message`, and related-subject values belong +only in bounded diagnostic events and must not become metric labels. + +## Conformance + +An implementation conforms when it proves: + +- one Gateway serving lifecycle retains one producer subject across repeated + evaluations and receives a new ID after disposal and restart; +- host, process, Gateway, and owner-resource IDs renew at their documented + lifecycle boundaries while generations revise only the same object; +- all condition and related references resolve; +- `(subjectRef, type)` is unique; +- plugin namespaces cannot collide with core or another plugin; +- compatible declarations merge and conflicting non-empty declarations fail + closed; +- missing, malformed, cyclic, or excessive declarations return bounded + structured failure; +- deterministic ordering is stable across equivalent provider completion + orders; +- every condition is bound to the same reconciled identity snapshot and late + results from a replaced owner or activation generation are discarded or fail + closed; +- provider timeout and cancellation behavior remains bounded while subjects are + collected; +- an incomplete evaluation cannot make consumers infer condition removal; and +- public subject fields contain no secrets or unredacted owner data. diff --git a/rfcs/0018/readiness-v1-spec.md b/rfcs/0018/readiness-v1-spec.md new file mode 100644 index 00000000..490afe34 --- /dev/null +++ b/rfcs/0018/readiness-v1-spec.md @@ -0,0 +1,543 @@ +# Readiness v1 Specification + +This document is the implementer-facing specification for RFC 0018, Readiness +Conditions and Providers. The RFC explains the motivation, compatibility +strategy, and rollout plan. This file defines the v1 condition model, provider +lifecycle, operator selection, bounded evaluation, aggregation, and projection +contract that OpenClaw runtimes, plugins, and hosts can build against. The +focused subject identity and reconciliation contract is defined in +[`readiness-subjects-v1-spec.md`](readiness-subjects-v1-spec.md). + +Status: draft, tied to RFC 0018. + +## Scope + +This specification defines: + +- the canonical readiness condition and result shapes; +- stable condition, subject identity, and reason requirements; +- required and advisory aggregation; +- the core condition namespace and baseline lifecycle conditions; +- plugin readiness-provider registration and activation lifecycle; +- operator selection of additional required or advisory criteria; +- deadlines, cancellation, coalescing, caching, and error conversion; +- HTTP, health, status, and CLI projections; and +- compatibility and conformance requirements. + +This specification does not define: + +- standard or custom hosting profiles; +- checkpoint durability, safe shutdown, or restore compatibility; +- arbitrary shell, network, or script probes configured by operators; +- orchestration retry intervals or restart policy; or +- telemetry sinks, dashboards, alerting, or retention. + +## Terminology + +- **Condition**: one bounded observation about the current runtime activation. +- **Criterion**: an enumerable condition definition that can be selected for + evaluation. +- **Provider**: activated plugin code that implements one plugin-owned + criterion. +- **Requirement**: whether a non-true condition blocks readiness. +- **Universal condition**: a core lifecycle condition that always participates + once canonical readiness is activated and cannot be removed by operator + configuration. +- **Canonical result**: the single aggregated result consumed by all readiness + projections. +- **Subject**: one core- or plugin-owned runtime object observed by conditions. +- **Producer**: the subject that assembled and owns one readiness decision. +- **Projection**: an HTTP, health, status, or CLI representation of the + canonical result; a projection is not a second evaluator. + +## Compatibility And Evolution + +Readiness v1 uses these compatibility rules: + +- absence of both `gateway.readiness` and another separately accepted + activation contract preserves the legacy Gateway readiness decision path and + does not execute the canonical runtime evaluator; +- presence of `gateway.readiness`, including an empty object, activates v1; +- a separately accepted Standard Hosting Profile may activate v1 through its + profile-selection contract; +- condition `type` and `reason` values are stable machine contracts; +- `message` is redacted operator guidance and is not stable machine identity; +- adding an advisory criterion is backward compatible; +- making a criterion required, or adding a new universal required condition, + is compatibility-sensitive because it can change `200` to `503`; +- unknown optional result fields must be ignored by v1 consumers; +- legacy readiness fields remain projections during migration; and +- implementing or registering a criterion does not select it or make it + required. + +## Canonical Data Model + +The canonical v1 shapes are: + +```ts +type ReadinessConditionStatus = "True" | "False" | "Unknown"; +type ReadinessRequirement = "required" | "advisory"; + +type ReadinessCondition = { + type: string; + subjectRef: string; + relatedSubjectRefs?: string[]; + observedAtMs?: number; + status: ReadinessConditionStatus; + requirement: ReadinessRequirement; + reason: string; + message: string; +}; + +type ReadinessResult = { + contractVersion: 1; + evaluatedAtMs: number; + identity: ReadinessIdentity; + ready: boolean; + conditions: ReadinessCondition[]; + failures: string[]; + advisories: string[]; +}; +``` + +`ReadinessIdentity` and subject reconciliation are normative in the focused +subject sidecar. Implementations may add optional metadata, but the fields above +retain their v1 meaning. + +### Identity + +The pair `(subjectRef, type)` must be stable and unique within one result. Core uses +the public condition types defined below, such as `WorkspaceWritable`. A plugin +condition uses this namespace: + +- `plugin..` for plugin criteria. + +Configuration selects enumerable criterion IDs. Selectable public core +criteria use `openclaw.` and map to a core condition type. Not +every universal or built-in advisory condition is operator-selectable. Plugin +selectors are the same namespaced ID used by their emitted condition. Core and +plugin criterion IDs must not collide after normalization. A plugin-local ID is +trimmed, converted to lowercase, and must match +`^[a-z0-9][a-z0-9._-]*$`. Core prefixes the canonical activated plugin ID and +must reject an invalid or duplicate resulting ID. Configured selectors use the +canonical stored ID; consumers must not perform a second case-folding rule. + +`reason` must describe the observed state, not repeat the criterion identity. +Provider reasons must match `^[A-Za-z][A-Za-z0-9._-]{0,127}$`. Provider messages +must be valid text of at most 512 UTF-8 bytes after redaction. Reason and message +values must not contain secrets, paths with credentials, raw exception text, or +tenant content. Core validates these bounds before caching or projection; +invalid output becomes `CriterionInvalidResult`. + +Every primary, related, parent, and producer reference must resolve in the +result identity package. Subjects are declared once and deterministically +ordered. Core owns the `openclaw/*` namespace; plugins receive +`plugin./*`. Identity declarations, conflict behavior, bounds, +Gateway serving-incarnation lifetime, and diff semantics are defined by +[`readiness-subjects-v1-spec.md`](readiness-subjects-v1-spec.md). + +### Status + +- `True` means the criterion was observed and satisfied. +- `False` means the criterion was observed and not satisfied. +- `Unknown` means the criterion was selected but could not be established + within the contract, including timeout, unavailable source state, or an + evaluator failure. + +An absent observation must never be inferred as `True`. A selected required +criterion that cannot be evaluated must be emitted as `Unknown`. + +### Aggregation + +Aggregation is deterministic: + +| Requirement | `True` | `False` or `Unknown` | +| --- | --- | --- | +| `required` | No failure. | `ready=false`; append the stable reason to `failures`. | +| `advisory` | No advisory. | Preserve readiness; append the stable reason to `advisories`. | + +The condition array must use this v1 order when present: + +1. `ReadinessEvaluationComplete` when the normal evaluation cannot complete; +2. `GatewayStartupComplete`, `GatewayAcceptingWork`, `ChannelRuntimeReady`, + `ChannelRuntimeSuppressed`, and `EventLoopHealthy`; +3. `ConfigLoaded` and `WorkspaceWritable`; +4. `GatewayResponding` and `PluginsLoaded`; and +5. remaining selected core, profile-contributed, and plugin conditions sorted + by condition type and then `subjectRef`. + +`failures` and `advisories` follow condition order and contain no duplicates. + +Malformed evaluator output must become a stable `Unknown` result or cause +registration/config validation to fail. It must not disappear from aggregation. + +## Core Criteria + +The initial public core criteria are: + +| Selector ID | Condition type | Default requirement | Stable non-true reasons | +| --- | --- | --- | --- | +| Not selectable | `GatewayStartupComplete` | Universal required | `GatewayStartupPending`, `GatewayStartupNotChecked` | +| Not selectable | `GatewayAcceptingWork` | Universal required | `GatewayDraining`, `GatewayAdmissionNotChecked` | +| Not selectable | `ChannelRuntimeReady` | Universal required | `ChannelRuntimeUnavailable`, `ChannelRuntimeNotChecked` | +| Not selectable | `ChannelRuntimeSuppressed` | Advisory when present | `ChannelRuntimeSuppressed` | +| `openclaw.event-loop-healthy` | `EventLoopHealthy` | Advisory unless selected as required | `EventLoopDegraded`, `EventLoopStatusUnavailable`, `CriterionEvaluationUnavailable` | +| Not selectable | `ConfigLoaded` | Universal required | `ConfigNotLoaded`, `ConfigInvalid`, `EffectiveConfigUnavailable` | +| `openclaw.workspace-writable` | `WorkspaceWritable` | Selectable | `WorkspaceMissing`, `WorkspaceStorageFull`, `WorkspaceNotWritable`, `WorkspaceProbeFailed`, `WorkspaceProbeTimedOut`, `WorkspaceNotChecked` | +| `openclaw.config-current` | `ConfigCurrent` | Selectable | `ConfigRestartRequired` | +| `openclaw.model-route-ready` | `ModelRouteReady` | Selectable | `ModelRouteUnavailable`, `ModelAuthUnavailable` | +| `openclaw.secrets-ready` | `SecretsReady` | Selectable | `SecretOwnersUnavailable` | +| `openclaw.session-storage-ready` | `SessionStorageReady` | Selectable | `SessionStorageMissing`, `SessionStorageFull`, `SessionStorageNotWritable`, `SessionStorageProbeFailed`, `SessionStorageProbeTimedOut`, `SessionStorageNotChecked` | +| `openclaw.context-engine-ready` | `ContextEngineReady` | Selectable | Owner-defined bounded activation reasons. | +| `openclaw.tool-catalog-ready` | `ToolCatalogReady` | Selectable | Owner-defined bounded activation reasons. | +| `openclaw.mcp-runtime-ready` | `McpRuntimeReady` | Selectable | Owner-defined bounded activation reasons. | +| `openclaw.sandbox-ready` | `SandboxReady` | Selectable | Owner-defined bounded activation reasons. | +| `openclaw.harness-ready` | `HarnessReady` | Selectable | Owner-defined bounded activation reasons. | +| `openclaw.state-ready` | `StateReady` | Selectable | Owner-defined bounded state reasons. | +| `openclaw.delivery-runtime-ready` | `DeliveryRuntimeReady` | Selectable | Owner-defined bounded delivery reasons. | +| `openclaw.scheduler-ready` | `SchedulerReady` | Selectable | Owner-defined bounded scheduler reasons. | +| `openclaw.plugins-loaded` | `PluginsLoaded` | Advisory unless selected as required | `PluginLoadFailures`, `PluginStatusUnavailable`, `CriterionEvaluationUnavailable` | + +Each condition is `True` only when the runtime observes the corresponding +startup, admission, channel, event-loop, config, activation, workspace, +execution-capability, state, background-service, or plugin predicate described +by RFC 0018. A successful condition uses a stable success reason that normally +matches its condition type. Selectable owner criteria must observe existing +snapshots and must not initiate the subsystem work they report. + +The evaluator may emit these failure-only guard conditions: + +| Condition type | Requirement | Predicate | +| --- | --- | --- | +| `ReadinessEvaluationComplete` | Required | The bounded canonical evaluation completed; otherwise `ReadinessEvaluationTimedOut` or `ReadinessEvaluationFailed`. | +| `GatewayResponding` | Required when the operation is remote | The caller reached the live Gateway; otherwise `GatewayUnavailable` or `GatewayNotChecked`. | + +Implementations must preserve existing unconfigured Gateway readiness behavior +while these observations are normalized. `workspace-writable` remains +unselected unless an operator or a separately accepted profile selects it. + +## Plugin Readiness Providers + +Only activated OpenClaw plugins may register executable provider code. Runtime +configuration may select a provider ID but must not contain callbacks, scripts, +commands, or arbitrary probe definitions. + +The v1 plugin SDK contract is: + +```ts +type PluginReadinessResult = { + subjectRef?: string; + relatedSubjectRefs?: string[]; + observedAtMs?: number; + status: "True" | "False" | "Unknown"; + reason: string; + message: string; +}; + +type PluginReadinessProvider = { + id: string; + description: string; + check(context: { + config: OpenClawConfig; + pluginConfig: unknown; + signal: AbortSignal; + subjects: PluginReadinessSubjectCollector; + }): Promise | PluginReadinessResult; +}; + +type RegisterReadinessCriterion = ( + provider: PluginReadinessProvider, +) => void; +``` + +Registration through `api.registerReadinessCriterion` is scoped to the current +plugin activation. Core publishes the resulting ID as +`plugin..`. + +Each provider invocation emits exactly one condition. A plugin registers +separate criteria for independently selectable resources or states; an +unbounded collection is summarized by one aggregate subject and a bounded, +deterministic set of related subjects. + +The bundled Policy plugin is an optional demonstration of this contract; core +readiness does not depend on Policy being installed or enabled. When activated, +it registers `plugin.policy.conformant`. It reuses the plugin's existing policy +evaluation, +returns only bounded summary text, and is not evaluated until selected through +`gateway.readiness`. Advisory selection does not affect the overall readiness +decision; required selection fails closed for findings, disabled evaluation, +provider failure, or timeout. + +### Provider Requirements + +A provider must be: + +- observational and read-only; +- idempotent under repeated invocation; +- cancellation-aware; +- safe under concurrent invocation or core coalescing; +- free of blocking synchronous I/O; and +- redacted by construction. + +A provider must not mutate config, reload plugins, acquire or rotate secrets, +send model requests, change admission state, invoke tools, or alter another +condition. + +Provider `reason` must match `^[A-Za-z][A-Za-z0-9._-]{0,127}$`. Provider +`message` must be non-empty after trimming, contain no NUL bytes, and be at +most 512 UTF-8 bytes after core redaction. Output that violates these rules +becomes `CriterionInvalidResult=Unknown`; no raw provider exception or +unvalidated message is projected. + +### Activation Lifecycle + +The registered provider set is bound to one activation-pinned plugin registry +and effective config snapshot. Plugin or config reload atomically publishes one +replacement config-and-registry snapshot identity. Publication aborts in-flight callbacks from the prior +generation, invalidates their cached results, and prevents late settlement from +entering the active result. Core must keep retained state bounded when a +callback ignores cancellation; an abandoned callback must not retain an active +registry or become selectable after replacement. + +Provider descriptors must be enumerable without invoking callbacks. At minimum +the active registry entry contains the namespaced ID, description, owning +plugin, and source. The registry snapshot identity is the activation-generation +boundary. + +### Optional Criterion Catalog Projection + +An implementation may expose an authenticated read-only projection that +enumerates configurable criteria from one activation-pinned config and registry +snapshot. When implemented, the v1 catalog has this shape: + +```ts +type ReadinessCriterionCatalog = { + catalogVersion: 1; + criteria: Array<{ + id: string; + description?: string; + owner: + | { kind: "core" } + | { kind: "plugin"; pluginId: string; pluginName?: string } + | { kind: "unresolved" }; + registered: boolean; + selection: "required" | "advisory" | "unselected"; + }>; +}; +``` + +The projection lists selectable core criteria and all descriptors in the active +plugin registry. A selected ID absent from that registry remains visible with +`registered: false` and unresolved ownership. Results are sorted by criterion +ID. Registration source paths, plugin configuration, callback references, and +provider results are not projected. + +Catalog enumeration is deterministic for one config/registry snapshot and must +not invoke readiness evaluation, provider callbacks, probes, reload, repair, or +network I/O. It remains available when canonical readiness is not selected so +an operator can discover criteria before changing configuration. + +## Operator Selection + +Universal conditions always apply. Operators may select additional registered +core or plugin criteria without selecting a hosting profile: + +```json5 +{ + gateway: { + readiness: { + requiredCriteria: [ + "openclaw.workspace-writable", + "plugin.storage.backend", + ], + advisoryCriteria: ["plugin.metrics.exporter"], + }, + }, +} +``` + +Selection uses namespaced criterion IDs. The same ID must not appear in both +lists. Configuration validates selector syntax. A syntactically valid selected +ID absent from the active registry produces `CriterionNotRegistered=Unknown` +with the requirement of its containing list. Unknown IDs must never be silently +ignored. + +Registration alone does not activate a plugin criterion. Only explicitly +selected criteria participate: `advisoryCriteria` selects them as advisory and +`requiredCriteria` selects them as required. Operator selection may strengthen +a selectable criterion from advisory to required but must not weaken universal +required conditions. + +## Bounded Evaluation + +Readiness is a hot operational path and must complete within code-owned bounds. +The initial v1 limits are: + +- one second per plugin provider; +- one second for the workspace observation; +- concurrent evaluation of independent observations; and +- an independent two-second outer deadline for the complete result. + +Core owns deadlines, cancellation, coalescing, caching, error conversion, and +subject reconciliation, and result ordering. A timed-out criterion becomes `Unknown` with a stable timeout +reason. Unexpected exceptions become `Unknown`; raw exception text is not +copied into public output. + +Universal core observations consume already-owned runtime snapshots and must +not perform request-time network I/O. Plugin providers should report from +asynchronously refreshed local state when an observation can exceed the normal +probe path. Blocking synchronous work is invalid even though the outer watchdog +bounds cooperative asynchronous work. + +Plugin-provider failures use these core-owned reasons: + +- `CriterionNotRegistered` when a selected provider is unavailable; +- `CriterionInvalidResult` when output violates the provider result shape; +- `CriterionTimedOut` when the provider exceeds its deadline; and +- `CriterionCheckFailed` when invocation throws or rejects. + +If the outer deadline expires, the evaluator must return a required +`ReadinessEvaluationComplete=Unknown` condition with reason +`ReadinessEvaluationTimedOut`. The HTTP request must not hang or reject. + +Core retains ownership of an invocation after its deadline. If a provider +ignores cancellation and remains pending, later polls reuse the stable timeout +result and must not start another invocation, including after config or plugin +registry replacement. The callback is quarantined by canonical criterion ID +until it settles. A new invocation may start only after settlement and the +applicable cache or replacement-generation rules permit it. + +Successful and failed provider or workspace observations may be cached for at +most five seconds. Replacement effective-config or plugin-registry snapshots, +effective workspace identity, or selected-criterion changes invalidate affected +entries immediately. Admission, drain, startup, channel, and event-loop +snapshots are not made stale by this cache. A late result from an invalidated +generation is discarded. + +A changed workspace identity may start one replacement probe while a retired +probe remains blocked. No more than two workspace probes may remain in flight; +additional generations fail closed with `WorkspaceProbeTimedOut` until probe +capacity returns. + +## Projections + +Once v1 is activated, all projections consume the same canonical result. When +neither direct readiness configuration nor another separately accepted +activation contract is present, `/ready` and `/readyz` retain the legacy Gateway +decision path and must not invoke providers or the canonical runtime evaluator. + +### HTTP + +- `/ready` and `/readyz` are authoritative readiness probes. +- A ready result returns `200`; a non-ready result returns `503`. +- `HEAD` compatibility remains unchanged. +- Authenticated or local callers may receive the structured result. +- Unauthenticated remote callers receive only a compact redacted response. +- `/health` and `/healthz` remain shallow liveness probes and do not acquire + readiness semantics. + +### Health And Status + +Gateway health and status project the canonical result or a bounded summary of +it. A surface that did not observe a required live fact reports `Unknown`; it +must not synthesize success. Descriptor enumeration must not execute providers. + +### CLI + +An `openclaw ready` command, when implemented, is a client of the live Gateway +result: + +- human output lists all conditions with pass, fail, or warning classification; +- when identity is available, human output may identify the affected subject; +- `--json` preserves the canonical result; and +- exit status is nonzero for required failure, required unknown, transport + failure, or absence of a supported readiness contract. + +The CLI must not implement condition evaluation independently. + +### Optional CLI Facilities + +The following facilities are compatible extensions over the same live result. +They are not required for core readiness v1 conformance. When implemented, they +must follow the contracts below. + +Human output for non-`True` conditions may identify the existing kind, ID, +generation, and parent reference of each affected primary and related subject. +It must derive that explanation only from the canonical identity package. + +Criterion list and inspect commands are clients of the live catalog projection. +They may filter descriptors for presentation but must not execute a criterion +or infer registration from a readiness result. + +A bounded CLI wait mode is a client of repeated live Gateway results. It must: + +- use one explicit total deadline that covers config, credential, TLS, + connection, and RPC setup; +- run attempts sequentially and cap each per-call timeout by the remaining + total budget; +- retry both unavailable and not-ready observations without overlapping calls; +- emit only the final observation; +- preserve the canonical result unchanged for successful JSON output and for a + final observed not-ready result; and +- exit nonzero at timeout or transport failure. + +The wait facility must not evaluate conditions locally. Launcher-specific +commands may delegate to it, but they must not define another readiness or +deadline model. + +## Legacy Projection + +Existing `ready`, `failing`, `suppressed`, `eventLoop`, and `uptimeMs` fields +remain compatibility projections during migration. Implementations preserve the +legacy path before activation, emit canonical conditions beside legacy fields +after activation, then move every activated surface to the canonical evaluator. +Legacy removal requires a separate compatibility review. + +## Conformance Checklist + +An implementation conforms to readiness v1 when it proves: + +- existing unconfigured readiness behavior is unchanged; +- unconfigured probes do not execute providers or the canonical runtime + evaluator; +- an explicit empty `gateway.readiness` object activates canonical evaluation + without selecting additional criteria; +- implementations that support Standard Hosting Profiles prove that profile + selection activates canonical evaluation without requiring a separate + `gateway.readiness` section; +- universal startup, admission, and selected-channel failures return `503`; +- additional criteria do not become required without explicit selection; +- required `False` and `Unknown` return `503`; +- advisory `False` and `Unknown` remain visible while returning `200` when all + required conditions are true; +- provider timeout, exception, malformed output, and ignored cancellation all + return within the outer deadline; +- cache entries expire within five seconds and invalidate on relevant config, + workspace, selection, and plugin-generation changes; +- reload atomically replaces provider descriptors and callbacks, discards late + results, and remains bounded across repeated never-settling providers; +- unknown selected criteria fail closed; +- every condition and relationship resolves to one reconciled subject; +- every canonical result has `contractVersion: 1` and satisfies the wire + cardinality and string bounds; +- plugin subject declarations are namespaced, bounded, deterministic, and + conflict-safe; +- one live Gateway serving lifecycle retains one producer identity across + repeated evaluations and receives a new identity after restart; +- process and optional host-workload parents renew independently at their own + lifecycle boundaries; +- `/ready`, `/readyz`, health, status, and CLI do not disagree about the same + observed activation; and +- public output contains no raw exceptions, secrets, credentials, or tenant + content. + +### Optional Facility Conformance + +An implementation that adds the optional operator facilities must also prove: + +- catalog enumeration reflects one active config/registry snapshot, reports + selected missing IDs, and never invokes provider callbacks; +- bounded CLI waiting aborts slow pre-request setup at its total deadline, + never overlaps evaluations, and emits only its final observation; and +- CLI subject-lifetime explanation is derived only from the canonical identity + package and does not alter canonical JSON or evaluate conditions locally.