feat(008): bundle-scoped impact reporting — one row per advisory (4.2.0) - #16
Merged
Merged
Conversation
`uv run` in a checkout writes a lockfile that nothing here reads. Every documented path is `uv tool install` (which resolves dependencies itself) or `python -m unittest`; no `uv sync` or `uv run` appears in CONTRIBUTING, README, or the workflows. Committing it would also work against the packaging intent: this package is both the `caia` command and an importable library, and pyproject pins loosely on purpose so consumers resolve their own versions. A committed lockfile would imply a pinned set neither install path honours. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Adds the requirements for scoping a run to a Cisco Event Response bundle and reporting one row per advisory in that bundle, rather than one row per device. Feasibility was verified against the live sources using ERP-75736 and the 70-row reference inventory, not read from documentation: - a bundle page yields its advisory identifiers from the links in its table; they are absent from the visible text, so reading the text alone fails - `advisoryId` and `firstPublished` are already in the per-device payload the tool retrieves, so filtering an assessment costs no extra requests - an advisory impacting nothing appears in no device's result, so its metadata needs an individual lookup (1 of 25 here; worst case the bundle size) - Cisco publishes no bundle endpoint (HTTP 404), hence reading the page - 24 of the 25 advisories touch at least one device, from 0 to 50 each Two decisions were escalated rather than defaulted, and are recorded with their reasoning in the spec's Clarifications: - the device is the unit of count, with a per-row rollup by affected release. A release-based denominator cannot close FR-025's accounting, because an undetermined device often has no release Cisco recognises. - rows are scoped to the families an advisory covers, with whole-run totals stated once. A single whole-inventory denominator would emit ~375 mostly meaningless undetermined markers across 25 rows, training readers to disregard the marker Principle IV depends on. The load-bearing constraint throughout: a device whose assessment reached no conclusion has no advisory list at all, so it can neither be asserted as impacted nor excluded as clean. FR-024 through FR-028 exist for that, and FR-027b fixes the direction of the guarantee — family scoping may only narrow which rows an undetermined device appears in, never suppress an impacted device or manufacture an unaffected one. Planning gate: fetching a bundle page adds sec.cloudapps.cisco.com as an endpoint the analyzer contacts, which Technology & Data Constraints designate a reviewed change requiring a constitution amendment before implementation. Docs only — no product code changes, so VERSION is untouched. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Phase 0 and Phase 1 output for bundle-scoped impact reporting. Fifteen research decisions, all three items the spec deferred to planning now resolved. What the research changed: - No new dependency. An earlier curl timeout suggested the bundle page might need a browser engine; it does not — urllib fetched it in 0.54s on first attempt. stdlib HTMLParser matching on href and a lexical scan both return exactly the 25 identifiers. The parser is chosen regardless: an advisory id mentioned in prose rather than linked would silently widen scope, and scope is the only thing the page is trusted for. - The publication-date cross-check floated in the spec is rejected. The date window returns 27 advisories for this bundle's date — the 25 plus two unrelated same-day publications — so it cannot distinguish a missed parse from a same-day publication the bundle legitimately excludes. A check that cannot fail cleanly is worse than none; the advisory count shown to the user before the run is the honest check. - Family derivation costs nothing: productNames is already in the per-device payload, so both derivation sources are free. Verified on two advisories. - Added cost is ~3s on a 72s run (0.54s page fetch plus one metadata lookup for the single advisory in this bundle that impacts nothing), inside SC-008. Module split mirrors the roles the package already has, so the Principle VI boundary needs no new reasoning: erp.py does I/O like psirt, bundle.py is pure like matching, bundle_report.py renders like html_report. erp and bundle join the shared surface, because the ERP parse and family derivation are exactly the fragile logic a second consumer would otherwise reimplement. The data model's worked example was verified arithmetically against the live assessment: the per-row invariant (impacted + unaffected + undetermined == population) closes on all four sampled rows. Constitution Check is a CONDITIONAL pass. Two gates depend on an amendment (3.1.0 -> 3.2.0, MINOR) carried as a blocking prerequisite of the first implementation task: admit sec.cloudapps.cisco.com; state the scope-selector boundary that keeps "no scraped or cached-as-authoritative alternatives" substantively true; and extend Principle VI's named module list from four to six. Complexity Tracking is empty — no violations. Product version will move to 4.2.0 (MINOR: the shared surface gains, nothing incompatible changes). Not bumped here — docs only. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Phase 2 output. Tasks organised by user story so each ships and tests independently; MVP is Phase 0-3 (US1). Three things differ from what the task template would produce by default: - Phase 0 is the constitution amendment, and it is the only phase that does not happen on this branch. Spec 006 recorded its amendment as already applied and explicitly not a task; ours is outstanding, so it becomes a blocking prerequisite on its own chore/ branch against main. Until it merges, an implementation that fetches a bundle page violates the very constraint that made the change reviewable. - Tests are marked not-optional. The template treats them as opt-in, but the Development Workflow gate makes them mandatory here on three counts: verdict-adjacent logic, analysis-flow changes needing credential-free coverage, and the shared surface's headless property. - FR-035 and FR-036 initially had no tasks — dropped because they are SHOULDs rather than MUSTs — and FR-019 was implemented but uncited. Added as T056/T057 and cited on T017. Coverage is now 46/46. Validated mechanically: 60 tasks, IDs sequential T001-T060, every task carrying a checkbox, id, optional labels and a concrete file path; all five user stories represented; 18 tasks parallelisable. The parallel win is Phase 2, where the erp.py and bundle.py chains share no files and can run concurrently, then the four test modules together. Each phase states its independent test criteria against a numbered quickstart scenario. T060 singles out Scenario 2 to re-run after any report change: a row reading "0 impacted" without its 11 undetermined devices is the failure mode the design exists to prevent. Docs only — VERSION untouched; its bump to 4.2.0 is T059. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
…pe-selector boundary Enacted as a prerequisite of spec 008. Cisco publishes no API equivalent of a bundle (a request for one returns HTTP 404), so a bundle's membership can only be learned from its published page. That made two existing constraints block the feature and left one principle's text stale. Four changes: - Networking admits sec.cloudapps.cisco.com, scoped to resolving a bundle to its advisory identifiers and its own identity. The request is explicitly unauthenticated and forbidden from carrying ANY inventory-derived data — a stricter bound than the advisory API's, which may carry the derived family/platform/release triple. - A new Scope selectors bullet, rather than a relaxation of "No scraped or cached-as-authoritative alternatives". Reading a published page could be argued to survive that prohibition on a technicality, which is the kind of reasoning a governance document should foreclose. So the prohibition stands and a narrow enumerated permission sits beside it: a bundle page may supply which advisories are in scope and the bundle's own identity, and may not supply any advisory fact — title, severity, score, CVE, affected product, fixing release, or an advisory's publication date, every one of which the page actually displays. Disagreement resolves in the API's favour without exception, and a page yielding no advisory is an error rather than an empty scope, because an empty result rendered as a report reads as an all-clear. This narrows the tool's latitude while unblocking the feature. - Principle VI's shared surface grows from four modules to six, adding erp (bundle resolution) and bundle (family derivation, filtering, row construction) — both fragile, safety-bearing logic a second consumer would otherwise reimplement, and a second divergent answer about which advisories are in scope is the harm the principle exists to prevent. bundle_report joins the terminal-side list beside html_report. - Consumers now enumerates all six modules of the released interface. A new constraint bullet plus materially expanded guidance, with no principle removed or redefined and no existing permission widened: MINOR. Applied on this feature branch rather than a standalone chore/ branch. Checking the history corrected an earlier assumption: both prior feature-driven amendments travelled in their feature's own PR — spec 006 in cb6938a (#9) and spec 007 in e043d41 (#12). The one standalone constitution commit, 73de536 (#15), was retroactive, documenting Principle VI after 4.1.0 had shipped, so it is not the precedent here. Keeping the amendment with the code it authorises also keeps its justification in front of the reviewer. Also realigns the spec-008 artifacts that asserted the amendment was still outstanding: the plan's Constitution Check moves from CONDITIONAL PASS to PASS, tasks Phase 0 is marked complete, and research Decision 9 records the applied date and the rejected standalone-branch alternative. The README and docs/index.html shared-surface updates remain tasks T051 and T052 — feature work due in this same PR, not amendment debt. Product VERSION untouched at 4.1.0; it moves to 4.2.0 as task T059. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Scope a run to a Cisco bundled publication by pasting its Event Response link, and get back one row per advisory in that bundle rather than one row per firewall. Each row carries the advisory as a link, its publication date, and the products it impacts with the release that fixes them. Adds no data source. The advisory id and publication date were already in the payload retrieved per device, so filtering costs zero extra requests; only advisories that impact nothing need a lookup (1 of 25 for the reference bundle). Cisco publishes no bundle endpoint, so membership comes from the published page — which contributes scope only, never an advisory fact. New modules, taking the three roles the package already has: erp.py I/O, like psirt: bundle page -> advisory identifiers bundle.py pure, like matching: covered families, filtering, rows bundle_report.py rendering, like html_report: the per-advisory layout erp and bundle join the shared surface, because the page parse and the family derivation are the fragile parts a second consumer would otherwise reimplement. The load-bearing property, and the reason for most of the design: a device whose assessment reached no conclusion has NO advisory list at all, so it can neither be asserted as impacted nor excluded as clean. Two mechanisms enforce that rather than leaving it to review — the per-row accounting invariant (impacted + unaffected + undetermined == population), which makes a dropped device fail arithmetic instead of rendering as clean; and fail-open family scoping, where a failed derivation widens to every family so its worst outcome is a noisier report, never a suppressed device. One bug found while verifying, worth calling out because it would have misdirected patching. first_fix() returned one fixing release per row, while FR-023 requires it per impacted product. Invisible on the FMC row, where both affected releases share 7.4.4 — but cisco-sa-asaftd-ikev2-dos-eBueGdEG spans 13 releases needing 6 different fixes (7.0.9, 7.4.3, 7.7.11, 9.16.4.85, 9.18.4.71, 9.20.4.10). The old code would have shown one target for all 50 devices. Fixes are now per device, rendered per release group. Verified twice end to end against ERP-75736 and the 70-row reference inventory: 25 rows, zero accounting failures, identical row order across runs, 33 requests in ~75s (~3s over the whole-fleet baseline). Every number matches the design's worked examples — FMC auth bypass 5/0/0 fixed in 7.4.4; ASA TCP flood 0 impacted, 17 not impacted, 11 undetermined; FTD snort3 SSL 11/22/4; IKEv2 50/0/15. Tests: 272 -> 392, all without credentials or network, including the headless property of both new shared-surface modules. Page fixtures are hand-authored builders mirroring the verified live shape, following psirt_responses.py rather than vendoring ~180 KB of Cisco's page. Declining the prompt is byte-equivalent to today's behaviour, and no bundle page is fetched on that path. --erp makes the capability scriptable, as Principle III requires. VERSION 4.1.0 -> 4.2.0 (MINOR: the shared surface gains erp and bundle; nothing incompatible changes). Tag is pushed only after this PR merges. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
The clean-room harness installed the new code but never touched the new capability, so a --shell session would not have tested the thing it was spun up to test. - automated checks now assert --erp is advertised and requires a value - the --shell banner walks both report shapes and the three failure paths, and shows the two filenames side by side so the layouts are easy to compare Also records an ordering quirk that is confusing to hit unexplained: the credential gate fires before bundle validation, so --erp with a bad value exits 1 (no credentials) rather than 2 (not a bundle address) until --config has run. That order is deliberate — the tool refuses before making any request — so the banner explains it instead of the behaviour changing. Default mode installs the working tree, so this tests unreleased work without pushing anything. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Scope a run to a Cisco bundled publication by pasting its Event Response link, and get back one row per advisory in that bundle rather than one row per firewall. Each row carries the advisory as a link, its publication date, and the products it impacts with the release that fixes them.
Closes the gap the ops team actually works in: Cisco publishes firewall vulnerabilities in bundles (ERP-75736 lists 25 advisories covering 48 CVEs), and the review cycle is triggered by one of those — but the tool had no idea a bundle existed, so matching a publication to a 70-row report was done by hand.
caia --erp ERP-75736 # or paste the link at the prompt; press Enter to skip and get today's reportApproach
Adds no new data source. The advisory identifier and publication date were already in the payload retrieved per device, so filtering an assessment to a bundle costs zero extra requests; only advisories that impact nothing need a lookup (1 of 25 for the reference bundle). Cisco publishes no bundle endpoint (
/eventresponse/…→ 404), so membership comes from the published page — which contributes scope only, never an advisory fact.Three new modules, taking the three roles the package already has:
erp.pypsirt— bundle page → advisory identifiersbundle.pymatching— covered families, filtering, rowsbundle_report.pyhtml_report— the per-advisory layouterpandbundlejoin the shared surface because the page parse and the family derivation are exactly the fragile logic a second consumer would otherwise reimplement, and a second divergent answer about which advisories are in scope is the harm Principle VI exists to prevent.The load-bearing constraint
A device whose assessment reached no conclusion has no advisory list at all — Cisco was never able to answer for it, so its status against any specific advisory is genuinely unknown. It can neither be asserted as impacted nor excluded as clean. Two mechanisms enforce that rather than leaving it to reviewer vigilance:
impacted + unaffected + undetermined == population, asserted for every row. A device silently dropped fails arithmetic instead of rendering as clean. This is also why the device is the unit of count: a release-based denominator cannot close, because an undetermined device often has no release Cisco recognises.A single whole-inventory denominator was the first candidate and was rejected on Principle IV grounds, not readability: with 15 undetermined devices across 25 rows it emits ~375 mostly-meaningless markers (an ASA appliance flagged against an FMC-only advisory it cannot run), training readers to skip the signal the principle depends on.
One bug worth calling out
first_fix()returned one fixing release per row, while FR-023 requires it per impacted product. Invisible on the FMC row, where both affected releases share 7.4.4 — butcisco-sa-asaftd-ikev2-dos-eBueGdEGspans 13 releases needing 6 different fixes (7.0.9, 7.4.3, 7.7.11, 9.16.4.85, 9.18.4.71, 9.20.4.10). The old code would have shown one target for all 50 devices, sending most engineers to the wrong version. Fixes are now computed per device and rendered per release group.Verification
Run twice end to end against ERP-75736 and the 70-row reference inventory: 25 rows, zero accounting failures, identical row order across runs, 33 requests in ~75s (~3s over the whole-fleet baseline, inside SC-008's 25% bound). Every number matches the design's worked examples:
The ASA row is the one to look at: 0 impacted, but 11 of 28 devices were never conclusively assessed. Rendered as "no impact" alone it would read as an all-clear for 11 devices nobody assessed.
Tests: 272 → 392, all running with no credentials and no network, including the headless property of both new shared-surface modules. Page fixtures are hand-authored builders mirroring the verified live shape, following
psirt_responses.pyrather than vendoring ~180 KB of Cisco's page.Constitution compliance
Amended 3.1.0 → 3.2.0 (MINOR) as a prerequisite, in
7b64654, following the feature-branch precedent of specs 006 (cb6938a) and 007 (e043d41):sec.cloudapps.cisco.com, unauthenticated and forbidden from carrying any inventory-derived data (a stricter bound than the advisory API's).bundle_reportjoins the terminal-side list.urllib,html.parser,html.escape,repathlibfor report paths--erpneeds no prompt; exit codes reuse the existing vocabulary (2bad argument,1run failure,130interrupt)Documentation gate:
README.mdshared-surface table updated to six modules and the bundle flow documented, mirrored indocs/index.html.Backward compatibility
Declining the prompt is byte-equivalent in content to today's report, and no bundle page is fetched on that path. A non-interactive run with no
--erpnever prompts. The whole-fleet layout, verdict rules, verdict vocabulary and reason kinds are untouched.Versioning
VERSION4.1.0 → 4.2.0 (MINOR: the shared surface gainserpandbundle; nothing incompatible changes). PerCONTRIBUTING.mdthe tag is pushed only after this merges.Reviewing it
Default mode installs the working tree into a clean-room
python:3.9-slimcontainer. Inside:caia --config, then comparecaia --inventory inventory.xlsx --no-update-checkagainst the same with--erp ERP-75736.5fbe88fextends the harness to cover the new flow and its failure paths.🤖 Generated with Claude Code