Skip to content
This repository was archived by the owner on Oct 10, 2026. It is now read-only.

v0.4.0 - transcript-backed observations

Choose a tag to compare

@Dashtid Dashtid released this 31 Aug 20:09
· 53 commits to main since this release
9835eab

The AWS tranche is now transcript-backed, and AWS's own creation-time guardrail
turns out to be shallower than its error message implies.

Every observed vector now carries its raw record

observation.evidence is prose, and prose loses the detail you did not know
mattered. This release adds an optional observation.transcript to the schema
and a committed transcript for every observed vector, under
observations/ —
the exact request document, the verbatim response, the argv, the CLI version and
the derived verdict. A reader can now audit any observed vector without an AWS
account and without re-running anything.
Account IDs are scrubbed before
anything is written.

Provenance: 122 documented / 11 observed (was 127/6). github-aws at
suite version 0.4.0.

The finding: AWS's creation guardrail checks existence, not scope

iam:CreateRole, probed three ways on 2026-08-31 with the condition operator
recorded (all transcripts committed):

Trust policy sub condition Result
StringLike ["repo:octo-org/octo-repo:ref:refs/heads/main", "repo:octo-org/*"] accepted
StringLike ["repo:acme/x", "*"] accepted
no condition at all rejected — MalformedPolicyDocument

The rejection message says a GitHub-OIDC trust policy must evaluate
...:sub or ...:job_workflow_ref "which is not scoped to all". But under
StringLike a bare * value is scoped to all — and AWS accepted it as one
value of an OR-list. So the creation-time guardrail checks that a sub condition
exists, not that its values are scoped. The 2023 condition-less class is
closed at creation; the multi-value loose-value hole is open.

IAM evaluates multiple values of one condition key with a logical OR, so the
loosest value sets the trust boundary regardless of how tight its neighbours are.

Newly observed

Five vectors promoted from documented, all confirmed on the same run:
gh-aws-pull-request-subject, gh-aws-branch-prefix-collision,
gh-aws-single-char-wildcard, gh-aws-exact-branch-stringlike, and
gh-aws-stringequals-treats-star-literally — the cleanest operator-semantics
pair in the corpus, since repo:octo-org/* admits an entire organization under
StringLike and matches nothing at all under StringEquals.

The six already-observed vectors were re-run and re-evidenced against
aws-cli/2.36.33, so the whole AWS tranche rests on one dated, reproducible
session.

Harness

scripts/observe_aws.py
now writes a transcript for every call — success, disagreement or error — and
covers the role-creation probe. That probe refuses to run on an implied
condition operator
: it is either derived from the named vector or passed
explicitly, and either way it lands in the transcript. It confirms before
creating anything, always deletes the role, and records the get-role
verification. 25 offline tests cover everything but the network hop.

Install

pip install subvectors==0.4.0

The wheel ships the entire corpus. The vectors are CC0; everything else is
Apache-2.0.