feat(skills): gate the skills against the API the platform actually ships - #82
Merged
Merged
Conversation
…hips
The skills go into the preamble of every agent session fab runs, so a kind named
there is a kind agents will confidently author.
When the batch tier was deleted from eks-agent-platform, `BatchJob` survived in
eks-agent-platform-curation.md, and every session kept being told the platform
had a batch API. Nothing could catch it: fab vendors no CRDs, so there was no
local list to compare against. It was found by reading, and reading is not a
thing that keeps finding it.
`scripts/check-skill-resources.mjs` compares the kinds the skills enumerate
against the CRDs the operator chart ships, sparse-checked-out in CI.
Both directions, and the second is why this is not a one-way check:
* A kind the skills name that no CRD defines — the batch case. Agents are
taught an API that does not exist and find out at admission.
* A kind the operator ships that no skill mentions. Quieter, and it is how
SLOPolicy came to have a controller, a CRD, a kube-state-metrics projection
and three Grafana alerts while nothing in the org ever authored one.
Only API-group enumerations and definition bullets are read, not free prose — a
skill may say "platform" or "tenant" as English, and what has to be accurate is
where it enumerates the API.
Both empty sets fail rather than passing vacuously: a CRD directory that parses
to nothing, and a skills tree where no enumeration matches.
Negative-tested, each failing as intended: adding BatchJob back to a group line,
removing SLOPolicy from both its group line and its bullet, and pointing the
check at an empty CRD directory.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
The skills go into the preamble of every agent session fab runs, so a kind named there is a kind agents will confidently author.
When the batch tier was deleted from eks-agent-platform,
BatchJobsurvived ineks-agent-platform-curation.md— and every session kept being told the platform had a batch API. Nothing could catch it: fab vendors no CRDs, so there was no local list to compare against. It was found by reading, and reading is not a thing that keeps finding it.What it checks
scripts/check-skill-resources.mjscompares the kinds the skills enumerate against the CRDs the operator chart actually ships, sparse-checked-out in CI — the same mechanicnanohype/scripts/check-slo-constants.mjsalready uses to reach this repo.Both directions, and the second is why this isn't a one-way check:
SLOPolicycame to have a controller, a CRD, a kube-state-metrics projection and three Grafana alerts while nothing in the org ever authored one.Only API-group enumerations and definition bullets are read, not free prose. A skill may legitimately say "platform" or "tenant" as English; what has to be accurate is where it enumerates the API.
Vacuity
Both empty sets fail rather than passing quietly — a CRD directory that parses to nothing, and a skills tree where no enumeration matches. A gate whose healthy output matches its did-nothing output is the failure it exists to catch.
BatchJobback to a group lineSLOPolicyfrom its group line and bulletCurrently green: the skills name exactly the nine kinds the operator ships.