feat(conformance): strict route detection via SUPPORTED list#696
Merged
vieiralucas merged 2 commits intomainfrom Apr 23, 2026
Merged
feat(conformance): strict route detection via SUPPORTED list#696vieiralucas merged 2 commits intomainfrom
vieiralucas merged 2 commits intomainfrom
Conversation
The lenient 2xx-4xx = Pass rule was treating every unrouted fakecloud response (generic NotFoundException, "Stage not found" from APIGWv2's execute-api fallback, Lambda's UnknownOperationException, etc.) as a routed success. Result: conformance reported 100% per service while several services implement a small fraction of their Smithy surface. Cross-reference Smithy operations against each fakecloud service's own `fn supported_actions()` list (the same source-of-truth used by the Level-2 audit). Operations not in SUPPORTED are classified as NotImplemented without a network probe — saves the round-trip and, more importantly, gives an honest per-service coverage number. Also widen the classifier's NotImplemented response-body patterns with `UnknownOperationException`, `Unknown path:`, and `Unknown operation:` as belt-and-suspenders for any service the audit mapping doesn't cover yet. Baseline refresh: 59,954/59,954 (100%) -> 42,889/59,954 (71.5%). The new numbers reflect reality: 11 of 23 services are at 100%, the rest have real implementation gaps that were previously masked. Per-service breakdown is in conformance-baseline.json. No service implementations changed. This commit only tightens the conformance classifier so it stops lying about coverage.
Codecov Report✅ All modified and coverable lines are covered by tests. 📢 Thoughts on this report? Let us know! |
![cubic-dev-ai[bot]](https://avatars.githubusercontent.com/in/1082092?s=60&v=4){kind=small}
Codecov patch check on PR #696 dropped because the three new NotImplemented response-body patterns (UnknownOperationException, Unknown path:, Unknown operation:) added to classify_response had no direct unit test — they were exercised only through the full conformance run. Add four unit tests that exercise each new pattern plus one negative case (ResourceNotFoundException must still classify as Pass).
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
The lenient `2xx-4xx = Pass` rule in the conformance classifier was hiding real implementation gaps: fakecloud's various "I didn't route this" responses (generic NotFoundException, APIGWv2's execute-api fallback, Lambda's UnknownOperationException, etc.) all looked like successful AWS-shaped errors from the probe's point of view. Result: conformance reported 100% per service while several services ship only a fraction of their Smithy op surface.
This PR cross-references every Smithy operation against the service's own `fn supported_actions()` list (the same source-of-truth used by the Level-2 audit) and classifies out-of-list ops as NotImplemented without even sending a probe. No service implementations change — the classifier just stops lying.
Honest baseline
```
total: 42,889 / 59,954 (71.5%)
100% services (11): bedrock, bedrock-runtime, cognito-idp, dynamodb,
events, kinesis, kms, logs, scheduler,
secretsmanager, ssm
partial services (12):
apigateway 28 / 103 (27%)
cloudformation 8 / 90 ( 9%)
elasticache 44 / 75 (59%)
iam 130 / 176 (74%)
lambda 13 / 85 (15%)
rds 23 / 163 (14%)
s3 74 / 107 (69%)
ses 97 / 110 (88%)
sns 34 / 42 (81%)
sqs 20 / 23 (87%)
states 14 / 37 (38%)
sts 8 / 11 (73%)
```
Additional belt-and-suspenders
Added `UnknownOperationException`, `Unknown path:`, and `Unknown operation:` to the NotImplemented response-body patterns in `classify_response` so any Smithy service not yet wired into audit still degrades gracefully.
Test plan
Follow-up work (not in this PR) is closing the 529-op gap across the 12 partial services. Handed off to a separate agent session.
Summary by cubic
Make conformance strict by checking each Smithy operation against the service’s
supported_actions()list, so unrouted paths no longer count as passes. Added unit tests for the new NotImplemented patterns; honest coverage is now 42,889/59,954 (71.5%) with no service code changes.supported_actions(); out-of-list ops areNotImplementedwithout a probe.classify_responsepatterns to treatUnknownOperationException,Unknown path:, andUnknown operation:asNotImplemented.ResourceNotFoundExceptionclassified as Pass.Written for commit d47e3d9. Summary will update on new commits.