Skip to content

fix: the ops platform declared an AgentFleet the API server refuses - #200

Merged
stxkxs merged 1 commit into
mainfrom
fix/ops-fleet-rejected-at-admission
Aug 9, 2026
Merged

fix: the ops platform declared an AgentFleet the API server refuses#200
stxkxs merged 1 commit into
mainfrom
fix/ops-fleet-rejected-at-admission

Conversation

@stxkxs

@stxkxs stxkxs commented Aug 9, 2026

Copy link
Copy Markdown
Member

addons/ai-platform/agent-platform/base/platform.yaml ships an AgentFleet whose single agent carries name, systemPrompt and modelRoute. The AgentFleet CRD requires spec.agents[].image, and it has no default, so every cluster syncing this addon gets

AgentFleet.agents.nanohype.dev "ops-fleet" is invalid:
  spec.agents[0].image: Required value

and that Application never reaches Healthy. This manifest is in the base — it is applied to every cluster in the fleet.

image is the container the agent actually runs, and it is the tenant's own build. This catalog has nothing truthful to put there: nothing in the org publishes a generic agent, and agentctl's own scaffold writes the literal REPLACE_WITH_YOUR_AGENT_IMAGE for a human to fill in. So the fleet is gone rather than patched.

The EvalSuite went with it, deliberately rather than incidentally: its agentFleetRef had nothing left to point at, and an EvalSuite over an absent fleet is the same shape of problem one layer up. The eval alerts and the eval-quality dashboard therefore start empty, which is the honest state for a cluster running no agents — AgentEvalNeverRan carries noDataState: OK, so an empty catalog is silent rather than alarming.

Why nothing caught it

Platform, Tenant, ModelGateway, BudgetPolicy, AgentFleet and EvalSuite are on kubeconform-scan.sh's skip list. That skip is honest about why — their schemas are published to no public catalog — and says these kinds are validated "out-of-band" with kubectl apply --dry-run=server. For this manifest that never happened. A skip that records a gap beats a green tick that pretends there is none, but a gap nobody closes is still a gap.

scripts/check-platform-crs.py closes it without a cluster. It resolves the CRDs from the operator chart at the version this catalog pins — read out of addons-agent-operator's targetRevision, not latest, because the question is whether the manifest is valid against the CRDs this catalog installs — then walks every CR of those kinds: required present at every level, nothing excess, arrays transparent.

Against the manifest as it shipped:

addons/ai-platform/agent-platform/base/platform.yaml (ops-fleet): AgentFleet
spec.agents[0].image is REQUIRED by the CRD, carries no default, and this manifest
does not set it — the API server rejects it with `spec.agents[0].image: Required
value`, and the Application never reaches Healthy

One thing the gate had to be taught

Its first pass also reported the ops Tenant as refused over spec.primaryPersona. That was wrong, and the correction is the interesting part: structural-schema defaulting runs before validation, so a required property carrying a default is filled in by the API server and admitted. primaryPersona defaults to generic; image has no default. Reading required alone cannot tell those apart, and a gate that cried wolf about a manifest running on every cluster today would not survive its first week. There is a self-test case pinning it.

Found while tracing every rackctl phase that has never executed live. rackctl's GitOps phase waits for --for=jsonpath={.status.health.status}=Healthy applications --all, which is stricter than the e2e harness's catalog gate — so this would have failed the first real install.

`addons/ai-platform/agent-platform/base/platform.yaml` shipped an AgentFleet
whose single agent carried name, systemPrompt and modelRoute. The AgentFleet
CRD requires `spec.agents[].image`, and it has no default, so every cluster
syncing this addon got

    AgentFleet.agents.nanohype.dev "ops-fleet" is invalid:
      spec.agents[0].image: Required value

and the Application never reached Healthy. This manifest is in the base — it is
applied to every cluster in the fleet.

`image` is the container the agent actually runs, and it is the tenant's own
build. This catalog has nothing truthful to put there: nothing in the org
publishes a generic agent, and agentctl's own scaffold writes the literal
REPLACE_WITH_YOUR_AGENT_IMAGE for a human to fill in. So the fleet is gone
rather than patched.

The EvalSuite went with it, deliberately rather than incidentally: its
`agentFleetRef` had nothing left to point at, and an EvalSuite over an absent
fleet is the same shape of problem one layer up. The eval alerts and the
eval-quality dashboard therefore start empty, which is the honest state for a
cluster running no agents. AgentEvalNeverRan carries noDataState: OK, so an
empty catalog is silent rather than alarming.

─── why nothing caught it ───

Platform, Tenant, ModelGateway, BudgetPolicy, AgentFleet and EvalSuite are on
`kubeconform-scan.sh`'s skip list. That skip is honest about why — their schemas
are published to no public catalog — and says these kinds are validated
"out-of-band" with `kubectl apply --dry-run=server`. For this manifest that
never happened. A skip that records a gap beats a green tick that pretends there
is none, but a gap nobody closes is still a gap.

`scripts/check-platform-crs.py` closes it without a cluster. It resolves the
CRDs from the operator chart at the version this catalog PINS — read out of
addons-agent-operator's targetRevision, not `latest`, because the question is
whether the manifest is valid against the CRDs this catalog installs — and walks
every CR of those kinds: required present at every level, nothing excess, arrays
transparent.

Run against the manifest as it shipped, it fails naming
`spec.agents[0].image`.

─── one thing the gate had to be taught ───

Its first pass also reported the ops Tenant as refused over
`spec.primaryPersona`. That was wrong, and the correction is the interesting
part: structural-schema defaulting runs BEFORE validation, so a required
property carrying a `default` is filled in by the API server and admitted.
`primaryPersona` defaults to `generic`; `image` has no default. Reading
`required` alone cannot tell those apart, and a gate that cries wolf about a
manifest running on every cluster today would not survive its first week.
@github-actions

github-actions Bot commented Aug 9, 2026

Copy link
Copy Markdown

CI Results

Check Status
Zero-placeholder gate ✅ success
YAML Lint ✅ success
Dashboard gate (live grafana.com ids + AMG-saveable) ✅ success
Kyverno policy tests (+ verify-images contract) ✅ success
Fork-safety gate (no hardcoded org, blocking) ✅ success
Helm-render (every addon × every env) ✅ success
Policy-admission (Enforce-tier Kyverno vs the fleet) ✅ success
ApplicationSet schema + sync waves ✅ success
Appset render (Karpenter subnet selector) ✅ success
Secret scan (gitleaks) ✅ success
Render + assert + schema + misconfig (all environments) ✅ success

All checks passed.

@stxkxs
stxkxs merged commit 1199c78 into main Aug 9, 2026
22 checks passed
@stxkxs
stxkxs deleted the fix/ops-fleet-rejected-at-admission branch August 11, 2026 17:18
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant