fix(aws): a subnet's occupants are traversable - #1437
Closed
lex00 wants to merge 1 commit into
Closed
Conversation
`reconstructEdges` treats a containment rule as a boundary hint and emits no edge, unless the rule declares a traversal name — the carve-out #1275 added because "a fold's first hop is sometimes exactly a containment". `AWS::EC2::Instance -> Subnet` was given a name. The network interface, which is the same relationship for the same reason, was not. Neither was the NAT gateway. So `kind:EC2::Subnet !<-kind:EC2::NetworkInterface` — the query the grammar exists to express, and the one an agent reaches for first — matched 23 of 23 subnets on an estate where 8 of 13 are empty. The negation was inert, and an inert negation is indistinguishable from one that searched and found nothing. `--explain` reported "23 of 23 matched", which is the tell, and nothing else said anything was wrong. Measured on the aws-bench negative question set: two of three trials asked this query, believed it, and answered with every subnet including the occupied ones. The one that passed did not believe it — it cross-checked with `kind:EC2::NetworkInterface --show SubnetId`, dumped `graph --format ir` and did the join in Python. A tool whose query language is only right when the caller does not trust it is worse than one that cannot answer at all. Scoped to "occupies a subnet", matching the exception that already existed. Containment INTO a VPC stays a boundary hint: `reference-catalog.test.ts` asserts that explicitly ("containment is never an edge"), and widening it is a separate decision from fixing an omission. That does leave `vpcs-with-no-running-instances` unanswerable by the same route, which is worth settling deliberately rather than as a side effect of this. The same omission exists on the ELB, ECS and RDS containments. No estate here exercises them, so they are left rather than changed untested. Closes #1432 Claude-Session: https://claude.ai/code/session_014KoduiMJRyLqjFYBUbMskE Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
lex00
added a commit
to INTENTIUS/chant-bench
that referenced
this pull request
Aug 3, 2026
5/6, up from 3/6. The subnet question went 1/3 to 3/3; the VPC question is unchanged at 2/3, which was predicted rather than hoped for. Two more chant defects, both pre-existing and both found by measuring rather than reading: - **INTENTIUS/chant#1437** — `AWS::EC2::NetworkInterface -> Subnet` was declared as containment with no traversal name, so `reconstructEdges` recorded a boundary hint and emitted no edge. `kind:EC2::Subnet !<-kind:EC2::NetworkInterface` — the query the grammar exists to express — matched 23 of 23 subnets. An inert negation is indistinguishable from one that searched and found nothing. - **INTENTIUS/chant#1440** — a subnet its own stack declares is also recorded as a dependency, so 13 subnets replayed as 16. A duplicate always reads as empty, because the edges resolve to the other copy: the three false positives were the three OCCUPIED subnets. Each of the four fixes was necessary and none was sufficient. Measured against the estate at each stage, the query returned 5 (one region visible), 7 (regions merged), 11 (negation inert), 11 (duplicates read as empty), and finally 8 — the correct 8, keeping us-east-1's occupied default-c out. The VPC question stays at 2/3 because containment INTO a VPC is deliberately not traversable — `reference-catalog.test.ts` asserts it — and flipping that to improve this number is a decision for its own change, not a side effect of this one. Left on #1437. ## The page now shows every run A second run of an arm is usually a different build, and which build produced a number is the whole question this repo exists to keep answerable. Showing only the newest quietly dropped the evidence that the number moved, so both runs are listed with their workspace fingerprints — which is what says they are not the same experiment. Refs #27 Claude-Session: https://claude.ai/code/session_014KoduiMJRyLqjFYBUbMskE Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
Contributor
Author
|
Superseded by the branch above — containment is now traversable by construction rather than by a per-rule |
Contributor
Author
|
Superseded by #1452, now merged — containment is traversable by construction rather than by a per-rule |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
reconstructEdgestreats a containment rule as a boundary hint and emits no edge — unless the rule declares a traversal name. That carve-out is #1275's, added because "a fold's first hop is sometimes exactly a containment".AWS::EC2::Instance -> Subnetwas given a name. The network interface, which is the same relationship for the same reason, was not:So
kind:EC2::Subnet !<-kind:EC2::NetworkInterface— the query the grammar exists to express, and the first thing an agent reaches for — matched 23 of 23 subnets on an estate where 8 of 13 are empty. The negation was inert, and an inert negation is indistinguishable from one that searched and found nothing.How it presented
--explainprinted the tell and nothing read it:A negation term that excludes zero of everything is worth suspecting. (That line also shows
kind:EC2::Subnetsubstring-matchingSubnetRouteTableAssociation, which is documented behaviour and a separate conversation.)On the aws-bench negative question set, two of three trials asked this query, believed it, and answered with every subnet including the occupied ones. The trial that passed did not believe it — it cross-checked with
kind:EC2::NetworkInterface --show SubnetId, dumpedgraph --format irto a file, and did the join in Python. A query language that is only right when the caller distrusts it is worse than one that cannot answer at all.Scope, and what I deliberately did not do
Scoped to "occupies a subnet" — network interface and NAT gateway — matching the exception that already existed for the instance.
Containment into a VPC stays a boundary hint. My first patch made those traversable too and
reference-catalog.test.tscaught it:That is an explicit, deliberate invariant. Flipping it to improve a benchmark number is not a change I should make as a side effect, so I reverted it. The cost is that
vpcs-with-no-running-instancesstays unanswerable by the same route — worth settling on its own terms, and I have left it in the issue rather than quietly deciding it here.The same omission exists on the ELB, ECS and RDS containments. No estate here exercises them, so I left them rather than change what I cannot test.
Verification
Four tests: the ENI reaches its subnet, the NAT gateway does too, an empty subnet is still reached by nothing, and the containment boundary is still recorded — the traversal is added, not traded for.
Full suite: 10467 passed, 5 skipped, 0 failed. Lint clean.
Closes #1432
🤖 Generated with Claude Code
https://claude.ai/code/session_014KoduiMJRyLqjFYBUbMskE