feat(core): containment is traversable without being drawn - #1452
Merged
Conversation
lex00
added a commit
to INTENTIUS/chant-bench
that referenced
this pull request
Aug 3, 2026
The middle of a replicate set rather than a single run, the same rule the board
uses — and here it matters more, not less. Six trials move further than
twenty-four do: on one build these questions returned 3, 4 and 6 of 6 with
nothing changed between the runs.
before the kind fix 3/6, 4/6, 6/6 median 4/6
after 6/6, 6/6, 6/6 median 6/6, no spread at all
That is a real shift rather than another draw from the same distribution, and
the variance collapsing is the more convincing half.
## What was making it noisy
Not the traversal — the counting. `kind:` is a substring, and a substring of a
CloudFormation type crosses kind boundaries: `kind:EC2::VPC` also matched
`AWS::EC2::VPCGatewayAttachment` and `kind:EC2::Subnet` also matched
`AWS::EC2::SubnetRouteTableAssociation`. An estate holding 6 VPCs answered 9.
Agents were naming exactly the right VPCs and reporting "9 VPCs in the estate"
beside them, and the grader failed the answer for contradicting the estate —
correctly, because it did. The finding was right and the census next to it was
of a set nobody asked for. Fixed in INTENTIUS/chant#1451.
The traversal work that made the questions answerable at all is
INTENTIUS/chant#1452, which replaced a per-rule opt-in with containment being
traversable by construction.
## A provenance error, found and corrected
`chant-neg-3/4/5` were published claiming workspace `30c405b9a8f5`. They ran
against `787ab0fd5bd0`; the emitter read the fingerprint file at emit time and
that file had been overwritten by a later export. Corrected here from the build
logs, and made impossible in lex00/aws-bench#11 — the runner now records the
fingerprint into the job.
Worth stating plainly because the correction changes what the table says: the
three runs either side of the kind fix are NOT the same experiment, and the
mis-stamp had made them look like they were.
Refs #27
Claude-Session: https://claude.ai/code/session_014KoduiMJRyLqjFYBUbMskE
Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Supersedes #1437, which fixed this by hand. The question that prompted the rewrite: shouldn't this be expanded rather than hand-maintained? It should.
What was wrong
reconstructEdgesmade a containment rule traversable only if the rule setviaAttr. That one field decided two unrelated things — "draw this as a line" (a renderer's concern) and "let a query walk this" (a caller's concern) — so making a relationship queryable meant drawing it, and the list of which relationships had been remembered was the list of which questions could be asked.AWS::EC2::Instance -> Subnethad been remembered.AWS::EC2::NetworkInterface -> Subnethad not. Sokind:EC2::Subnet !<-kind:EC2::NetworkInterface— the query the grammar exists to express — matched 23 of 23 subnets on an estate where 8 of 13 are empty. An inert negation is indistinguishable from one that searched and found nothing.Three changes, all general
1.
containmentEdges. The same pairs, as edges a query can walk, with the traversal name derived from the attribute path the containment was read through. A containment rule is now traversable because it is a containment rule, not because someone remembered.edgesis untouched, soreference-catalog.test.ts'sexpect(edges.some(e => e.to === "vpc")).toBe(false)still holds and renderers gain no lines.2.
Instance -> VPC. Subnet, security group and route table all had an "in VPC" rule; the instance did not. So the one kind every estate question is about reached its VPC only through its subnet — two hops, which a one-hop<-cannot cross.describe-instancesreturnsVpcIddirectly; nothing is inferred.3. Containment survives the overlay.
sourceOverlayGraphsadmits a live edge only when one endpoint is foreign, because "declared edges already cover every declared relationship". True of the references it was written for, false of containment — no source declares "this instance is in this VPC"; the instance declares a subnet and the VPC is implied by the account. So an instance and its VPC, both declared, had their containment dropped between them, while an instance in the account's default VPC kept it because that VPC is foreign.That asymmetry is what the measurement showed: "which VPCs have no instances" answered correctly for the one VPC nobody declared and wrongly for every VPC somebody did. Containment now merges into the queried IR after the overlay, so it is never subject to a filter written for a different kind of edge.
Measured against a live estate
Ground truth 13 subnets (8 empty) and 6 VPCs (2 empty):
Zero hand-added
viaAttrflags — the AWS catalog's count is unchanged from main.Full suite 10498 passed, 5 skipped. Typecheck and lint clean.
Closes #1432
🤖 Generated with Claude Code
https://claude.ai/code/session_014KoduiMJRyLqjFYBUbMskE