Config forward-compat: unknown fields from a newer CP whole-row-reject resources on older DPs — lenient parse + tri-state compat status

[membphis]

Description

The DP parses every etcd resource document strictly: each resource struct carries #[serde(deny_unknown_fields)] and the runtime JSON Schema generated from the same struct carries additionalProperties: false — both the runtime validator and schemas/resources/*.json build from struct_root_schema (crates/aisix-core/src/models/schema.rs:172), so serde strictness and schema strictness are one artifact. A document containing any field this DP version does not know is rejected whole-row at the single decode chokepoint validate_and_parse<T> (crates/aisix-etcd/src/loader.rs:312-352).

That turns every additive schema change into a breaking change under the normal upgrade order (CP first, DPs rolling behind):

1. The upgraded CP adds a field to a resource; the admin edits any document of that kind.
2. An older DP receives the watch Put, fails schema validation, and keeps serving the stale pre-edit value — Supervisor::apply_put returns before mutating the snapshot (crates/aisix-etcd/src/supervisor.rs:423-438). The admin's change silently does not apply.
3. On the next resync or restart the snapshot is rebuilt wholesale from accepted rows only (apply_resync, supervisor.rs:628-664) and the row vanishes. For an api_key the credential stops authenticating, with a 401 byte-identical to "no such key", logged only at DEBUG (crates/aisix-proxy/src/auth.rs:118-136).

The failure is therefore delayed and decoupled from the action that caused it: everything looks green after the CP upgrade, then a routine DP restart days later takes the resource out. The behavior is pinned — knowingly — by the provider_key_*_payload_currently_rejected tests (loader.rs:521-575), whose names anticipate this change.

This affects all resource kinds dispatched by build_snapshot (models, api_keys, provider_keys, guardrails, cache_policies, ...), not one resource. CachePolicy is the one existing exception: it already parses leniently, without incident.

Proposed design

Uniform tri-state compatibility, derived mechanically at the existing chokepoint — no per-resource rules:

• RED — incompatible. Lenient-schema or serde deserialization fails: field type conflict, missing required field, constraint violation (the generated schemas already carry required / minimum / maximum), unknown enum value. Whole-row reject as today, but: log at ERROR (today warn, loader.rs:324,342), report incompatible in the heartbeat, and retain the last known good typed value across resync/restart for as long as the etcd key still exists (xDS-NACK-style; removes the step-3 cliff above), with the staleness age reported every cycle.
• YELLOW — partially compatible. Struct parses and constraints pass but unknown fields are present: load and run with the extra fields ignored, collect their exact paths (serde_ignored wrapped around the existing from_value — one wrapper covers all kinds, nesting included), log WARN deduplicated per (kind, field-set), report partially_compatible aggregated per kind + field with counts, in a buffer separate from RejectedEntry so YELLOW volume cannot evict RED entries.
• GREEN — fully compatible. Exact match; unchanged.

Constraints that are part of the design, not optional hardening:

• Strict write / lenient read. Only the etcd loader becomes lenient. The self-hosted Admin API write path keeps rejecting unknown fields with 400 (apply an additionalProperties: false-injecting transform to the write-path validators only).
• Deliberate closures stay closed. The hand-injected additionalProperties: false on the observability_exporter oneOf branches (schema.rs:816-866) guards against plaintext-secret smuggling and must survive the migration.
• config_hash must not lie. A YELLOW row must not produce a "fully synced" hash match while enforcement differs — either exclude YELLOW rows from the hash or ship an explicit partially_compatible[] list next to rejected[] in the status surface.
• Same-version drift stays a test failure. CP-DP e2e at equal versions asserts zero YELLOW after convergence, preserving the typo-catching deny_unknown_fields performs today (provider_key.rs:654).
• No silent window. The YELLOW reporting machinery lands in the same PR as the leniency flip; there must be no interim state where unknown fields are ignored without a signal.
• Known-load-bearing unknowns get case-by-case decisions. The three pinned *_currently_rejected payloads (aws_region / gcp_project_id / azure_resource_name) become YELLOW loads whose runtime behavior is not "the old behavior" — flipping each test is a per-case decision in the same PR (prefer adding the field to the model where it is load-bearing for a provider family).

Paired CP work — non-blocking follow-up, filed as api7/AISIX-Cloud#1227. The CP needs no code change for this issue to ship: its heartbeat handler uses plain json.Unmarshal (no DisallowUnknownFields), so the new tri-state fields from a newer DP are tolerated as-is. The follow-up covers consuming/surfacing them:

• heartbeat consumption of the tri-state + dashboard surface + i18n;
• save-time fleet-version warnings — the heartbeat already carries BUILD_VERSION, so cp-api can warn "this environment has DPs at vX which will not enforce field Y" at the moment the admin saves. This is the only mitigation that acts before an admin relies on an unenforced restriction-type field. This class is real in-repo: ApiKey.disabled / expires_at were restriction fields added after first ship; under lenient parsing alone, an old DP would keep authenticating a key the dashboard shows as disabled;
• a written policy for new enum values. A new routing strategy / adapter / guardrail kind is not an unknown field — it stays RED (correctly: there is no old behavior to fall back to for a value the DP cannot interpret), and enum extension is routine roadmap work here, not a rare break. Each new value needs an explicit decision: version-gate it at the CP, or ship a #[serde(other)]-degradable variant where a fallback is semantically safe;
• the CP's heartbeat endpoint must itself accept unknown fields from newer DPs (the same skew problem in reverse).

Delivery plan

Two PRs (sized against this repo's history — feature PRs here routinely land at +500~1600 lines with e2e included, e.g. the outbound-TLS and OpenAPI-to-MCP work):

PR 1 — lenient parse + tri-state + full reporting (one atomic change).
The leniency flip and its signal must land together (the no-silent-window constraint), and the repo norm is to ship a feature with its e2e in one PR:

• schema.rs: strict-write/lenient-read split (loader validates the opened schema; Admin API keeps 400 via an additionalProperties: false-injecting transform; the observability_exporter closed branches stay closed);
• drop #[serde(deny_unknown_fields)] (~30 sites, mechanical) + regenerate schemas/resources/;
• validate_and_parse: serde_ignored collects unknown-field paths -> GREEN/YELLOW/RED; YELLOW in its own buffer (never sharing the RejectedEntry cap); RED logs at ERROR, YELLOW at WARN deduped per (kind, field-set);
• reporting in the same PR: /status/config partially_compatible[] (aggregated per kind + field + count), heartbeat fields (the CP tolerates unknown fields today — verified), Prometheus counter, config_hash companion list;
• tests: characterization flips, the three *_currently_rejected payloads decided case by case, e2e for unknown-field-goes-YELLOW plus the zero-YELLOW-at-equal-versions invariant;
• rider: the written enum-addition policy (version-gate at CP vs #[serde(other)] vs accept RED) goes into the contributor guidelines.

Estimated +1200~1800 — top of the normal range, but one coherent reviewable concept; the attribute sweep inflates the count mechanically.

PR 2 — RED last-known-good across resync/restart.
Separable on purpose: it changes serving semantics (supervisor resync/cache/delete interplay), has its own rollback risk, and fixes a cliff that already exists today independent of the leniency flip. e2e: rejected update -> stale value keeps serving -> survives resync -> survives restart -> disappears on delete, with staleness age reported.

User-facing docs go to api7/docs (separate repo, not a step here). CP-side surfacing is api7/AISIX-Cloud#1227, non-blocking.

Why

CP-before-DP is the supported rolling-upgrade order, and additive fields are the most common schema change — the current strict reader makes the common case a breaking one, with the worst failure surfacing days later at an unrelated restart. Whole-row rejection remains correct for genuine contract breaks (type changes, removed required fields), which the tri-state keeps as RED.

Priority

High. Triggered by every CP-first upgrade that adds any resource field, across every resource kind; the failure mode is delayed (post-restart cliff), and at request time indistinguishable from an invalid credential.

Prior art

System	Reader behavior on unknown config fields	Reference
Envoy	Lenient by default for xDS-delivered config — ignore + one deduped WARN + server.dynamic_unknown_fields counter; strict is opt-in via --reject-unknown-dynamic-fields ("This allows newer xDS configurations to be delivered to older Envoys"). Hard-invalid updates are NACKed with error_detail and the last valid config keeps serving.	CLI flags, xDS ACK/NACK
Kubernetes	Readers lenient by design: kubelet may run up to 3 minors behind the API server and decodes newer objects non-strictly; strictness is write-side only (fieldValidation=Strict|Warn|Ignore, CRD pruning at admission). API rule: adding a field is compatible; changing a type requires a new API version.	version skew, field validation, API change rules
Kong (hybrid)	The counter-example: CP-side per-DP-version field stripping — removed_fields.lua carries 9 version keys / 103 field removals and grows every minor; viable only because the CP holds a websocket to each DP and knows its version. The DP itself stays strict with last-known-good fallback and reports a per-DP sync status enum (Compatible / Compatible with limitations / Incompatible).	removed_fields.lua, version compatibility
protobuf	proto3 restored unknown-field preservation as the default in v3.5.0 specifically for uncoordinated rolling upgrades and intermediaries; ignoring unknown fields is the documented forward-compat mechanism. (ProtoJSON, by contrast, is strict by default with an opt-in ignore.)	updating message types, v3.5.0 release
LiteLLM	Fully lenient: router config models set ConfigDict(extra="allow"); invalid deployment rows are logged and skipped.	types/router.py
Apache APISIX	Row-strict at the DP (additionalProperties: false) but with per-key last-known-good retention; unknown plugin names arriving from etcd are warn-and-skip.	schema_def.lua
Tyk	Fully lenient: plain encoding/json decode ignores unknown fields; a definition that fails decoding is logged and skipped.	tyk

Current behavior (whole-row reject, no post-restart fallback) is stricter than every surveyed system.

_{Surveyed 2026-08-04; claims verified against the linked docs/source.}

[coderabbitai]

🔗 Related PRs

#816 - feat(retries): move the retry budget onto the model and apply it everywhere [merged]
#822 - feat(mcp): layered MCP access policies — env default, team entitlement, key narrowing [merged]
#825 - feat(auth): inbound OIDC/JWT authentication — trust providers + jwt_subject key binding [merged]
#835 - feat(headers): request-context variables in default_headers + client-header forwarding allowlist [merged]
#853 - feat(config): request_body_limit_bytes defaults to 0 = no cap; uniform 413 surface [merged]

---

📝 Issue Planner

_{Check the box below or use the @coderabbitai plan command to generate an implementation plan and prompts that you can use with your favorite coding assistant.}

• Create Plan

---

🧪 Issue enrichment is currently in open beta.

You can configure auto-planning by selecting labels in the issue_enrichment configuration.

To disable automatic issue enrichment, add the following to your .coderabbit.yaml:

issue_enrichment:
  auto_enrich:
    enabled: false

💬 Have feedback or questions? Drop into our discord!

[membphis]

Reopening: the "Closes" reference in #872 auto-closed this on merge, but only PR 1 of the delivery plan (lenient parse + tri-state + full reporting) has landed. PR 2 — RED last-known-good retention across resync/restart, with staleness age reporting — is still outstanding, tracked by the Delivery plan section above. The CP-side surfacing remains api7/AISIX-Cloud#1227 (non-blocking).