Scan precision: runtime-scope filtering, iOS lockfile support, EOL flagging (BomLens parity #15-#17)#482
Merged
Merged
Conversation
…nd matching cdxgen keeps every resolved node, so a deployed app's SBOM also carried its test/provided/dev toolchain (junit, lombok, devDependencies) as if it shipped — inflating CVE counts, license obligations and NOTICE content. Mirror the BomLens build-prep post-filters (#331/#335/#341) as a worker-side stage: - integrations/sbom_scope_filter.py: pure post-filter. Maven drops cdxgen scope optional/excluded behind a hasScopes guard (an unscoped SBOM is left whole so recall never regresses). Node diverges from BomLens deliberately: instead of an npm re-resolve subprocess, drop pkg:npm components the already-parsed package-lock.json positively classifies dev (hasDev guard, keep-if-unknown for purls the root lockfile does not cover). Shared tail prunes dependencies[] to kept refs and always preserves the root ref. - Pipeline wiring runs copy-then-commit BEFORE artifact persist, cosign signing, SCANOSS and Trivy so every consumer sees one consistent filtered document; a failed disk rewrite degrades to filter-skipped with memory and disk still identical. Telemetry lands in scan_metadata.scope_filter and a trusca:scope_filter SBOM property; no new pipeline stage (sub-second step). - Ingest path deliberately unfiltered: an uploaded SBOM is the supplier's declared truth (pinned by an AST regression test). - Config: SCAN_SCOPE_FILTER_ENABLED / _MAVEN_ENABLED / _NODE_ENABLED, all default ON; only exact falsy tokens disable, so a typo fails open to the correct-by-default behaviour (inverse of SCANOSS_ENABLED, no egress here). - Fixtures are real captured cdxgen 12.3.3 output (Spring Boot 3.2 Maven app: 108 components with required/optional/excluded; Express app: 487 components + its 502-package lockfile, 377 dev-flagged). - Docs EN/KO: runtime-scope filtering section, env vars, stage description; also refresh the stale WebSocket slug list (dt_upload/dt_findings era).
The scope filter (previous commit) drops dev/test components server-side — without a visible trace, a re-scan that loses 300 components reads as data loss. Surface the worker telemetry in the Components tab: - useScanScopeFilter reads scan_metadata.scope_filter off the ordinary scan detail response (ScanPublic.metadata already carries the JSONB verbatim — no backend change). Scan resolution mirrors the tab's anchoring: pinned ?scan= wins, else the latest succeeded scan from the overview. The parser treats the blob as untrusted at every level and collapses all nothing-to- show shapes to null. - ComponentsTab summary band gains an additive 'excluded N' note (data-testid components-scope-filter-note, per-ecosystem breakdown on the title attribute); EN/KO keys in the same PR. - Seeds (seed_demo + seed_e2e_user) stamp deterministic scope_filter telemetry on their succeeded scans so the e2e spec asserts without a live worker run.
With a Podfile present and no pod CLI (the worker ships no Ruby/CocoaPods toolchain), cdxgen's cocoapods cataloger throws a TypeError on the undefined pod stdout and aborts the whole SBOM stage — a terminal CdxgenFailed for any iOS repo. Mirror BomLens #347 in two halves: - Crash guard: pass --exclude-type cocoapods when a Podfile exists within depth 3 (Pods/ copies ignored; bounded glob, not BomLens' unbounded find). Applied in both the in-process adapter (integrations/cdxgen.py) and the sidecar script (build_prep_source.sh), cross-referenced to stay in sync. - Fill-in: integrations/cocoapods_lockfile.py hand-parses Podfile.lock's PODS: block (npm_lockfile discipline: never raises, MAX_PODS cap, name-to-ref guard so edges never point at phantom refs, self-edges dropped) and merges components + dependency graph into the SBOM before the scope filter, copy-then-commit like the filter so disk and memory never diverge. Deliberate divergence from BomLens: no syft — the PODS: block already carries names, pinned versions and edges. Subspecs (Moya/Core) percent-encode into the purl; dependency_scope stays NULL (Podfile.lock has no runtime/test signal). - SPM: repos committing only Package.resolved now detect as swift, and the sidecar skips swift package resolve over a committed lockfile (it is the resolved truth; re-resolving hits the network). Verified offline: cdxgen 12.3.3 without a swift toolchain parses Package.resolved directly (the captured real_cyclonedx_swift.json fixture is that verification run). - Fixtures: BomLens' real pod-install Podfile.lock (Moya -> Moya/Core -> Alamofire — subspecs, trailing-colon parents, constraint-only entries).
…e snapshot An EOL runtime or framework gets no upstream fixes — a later Critical has no patch. That is a supply-chain risk axis distinct from CVEs, and TRUSCA had no signal for it. Mirror BomLens #368 (enrich-eol.sh) as a catalog-layer enrichment: - EOL lives on component_versions (0038: six nullable columns + a partial index WHERE eol_state='eol') — the KEV shape: a fact about product+cycle shared across scans/projects, re-stampable when the snapshot moves. NULL = not a tracked product; the whitelist philosophy never guesses. - services/eol: the purl->product map vendored VERBATIM from BomLens (byte-identity contract test against a captured copy; the %40 npm-scope encoding divergence is absorbed in the matcher, keeping the map exact) + a compact endoflife.date snapshot vendored into the repo (16 KB, 10 products) so Docker builds stay network-free and air-gapped installs work out of the box. scripts/refresh_eol_snapshot.py (port of BomLens build-eol-index.py: 15s/request, 60s budget, failed products skipped) refreshes it per release; a map<->snapshot product-set contract test trips when the map changes without a refresh. - Evaluation is the BomLens semantics table-tested: cycle derivation (leading numeric segments, major vs major.minor), bool eol as-is, ISO date vs today, everything else unknown. express-session must not match pkg:npm/express@ (over-match guard test). - Stamping happens inside persist_sbom_components right after the catalog row upsert — one evaluator per persist call, changed-value-guarded (idempotent re-scan), covering the source-scan AND sbom-ingest paths for free. Never fatal: corrupt/missing dataset degrades to no stamping with one aggregate log line. - Config: EOL_ENABLED default ON (offline, zero egress — contrast SCANOSS), EOL_SNAPSHOT_PATH for air-gapped snapshot mounts. - Verified: alembic upgrade head applies 0038 on live PostgreSQL 17 (columns + partial index present); full unit suite green. API/FE surfacing and the refresh beat land in the next two changes.
Expose the catalog-layer EOL data (previous commit) end to end: - API: ComponentSummary carries eol_state/eol_date and the detail response the full verdict (product/cycle/date/source) — free columns on the existing ComponentVersion joins, no extra query. New ?eol=true|false facet rides the partial index; the overview gains eol_count (one count query gathered with the existing aggregation). OpenAPI snapshot regenerated for the new param. - FE: EolBadge (KevBadge structural mirror — High tone, renders only for state 'eol', absence is the signal), EOL column in the components table, an 'EOL only' toolbar toggle mirrored to ?eol=true, an always-rendered drawer row that distinguishes untracked (blank) from unknown (tracked, cycle undecidable), and an Overview chip deep-linking to the filtered list. EN/KO keys in the same change. - Contracts: the closed eol_state vocabulary now lives in three places (catalog tuple, wire Literals, FE mirror) — set-equality tests on both sides (test_catalog_contracts.py + catalogMirrors.test.ts) plus EN/KO label coverage. - e2e: components_eol.spec.ts (badge / filter+URL / drawer / overview chip); the seed stamps a far-past EOL verdict on exactly the first seeded component so assertions never depend on the run date or the vendored snapshot's live contents. - Docs EN/KO: end-of-life section in the components guide (blank-vs-unknown semantics), endoflife.date row in data-sources, EOL_* env vars.
Close out the EOL feature's ops layer (KEV subsystem structural mirror): - eol_sync_state (0039): the kev_sync_state single-row pattern plus one addition — the fetched dataset itself rides the row (JSONB, a few KB) so the re-stamp pass can prefer the freshest data without network. - tasks/eol_catalog_refresh: two-half tick. The fetch half is env-gated (EOL_REFRESH_ENABLED, DEFAULT OFF — new third-party egress follows the SCANOSS fail-closed posture, not the KEV one; EOL dates churn quarterly and the per-release vendored snapshot bounds staleness) with a proportional sanity floor (at least half the mapped products) so a gutted sweep never displaces a good dataset. The re-stamp half ALWAYS runs — pure local: effective dataset = newer of (vendored, fetched), bounded whitelist-prefix SELECT, changed-value-guarded stamps, and a clear pass for rows the whitelist no longer covers (the per-scan hook enriches but never reconciles). Weekly beat Sunday 02:15 UTC, between the KEV and Trivy ticks. - integrations/eol_feed: kev_feed discipline scaled to many small documents — per-product byte ceiling + skip-and-count, 60s whole-run budget, EolFeedUnavailable only when NOTHING fetched, host-only logging. - Admin surface: GET /v1/admin/eol/health (super-admin, existence-hide) assembled by eol_health_service (kev_health mirror + effective-snapshot resolution); FE EolPanel with a 180-day stale escalation, stamped/cleared KPIs and an origin/refresh footer; EN/KO keys; panel unit tests. - Docs EN/KO: admin health guide section (docs-uat marker on the endpoint), EOL_REFRESH_* env rows. OpenAPI snapshot regenerated. - Also: mypy-strict cleanups in the Phase K/L test files surfaced by the full 'mypy .' run.
…to 5a62094 7th review cycle over BomLens 9cc37e0..5a62094 (v1.7.0 + 27 commits, 80 total): the runtime-scope filter series, EOL flagging and the iOS lockfile work were judged as gaps #15-#17 and implemented in this same branch — all closed. Baseline, local-path correction (~/projects/bomlens) and the review history row updated. Follow-up backlog recorded: Android release-classpath scope filter (needs an Android SDK in the worker image), surfacing Trivy's image-level eosl flag for container scans. Also carries two doc changes that were finished earlier but stranded (docs-only PRs are BLOCKED by the ci.yml paths-ignore required checks): the reachability-analysis coordination row and docs/commercial-parity-candidates.md.
Producer-Reviewer findings (Medium 3 / Low 3 / Info 2), all addressed: - CocoaPods ReDoS (M): the PODS-block regexes backtrack quadratically on a crafted line (measured 16s at 80k chars). Guards, in order: 2 MiB file cap before reading, 4 KiB per-line cap before the regex, and the work bound now counts EXAMINED lines (a never-matching block cannot bypass MAX_PODS). Regression test pins the pathological line under 1s. - eol_cycle VARCHAR(32) overflow (M): a 33+-digit crafted version segment used to overflow the column at flush time — OUTSIDE the per-component best-effort guard — aborting the whole persist. derive_cycle now treats over-long numeric segments (>15 chars) as underivable -> 'unknown'. - eol_sync_state.snapshot bloat (M): per-field 256-char bound in the feed compactor (drop, never truncate — a truncated version would match the wrong cycle) plus a 256 KiB size gate on the assembled dataset before the JSONB UPSERT (skip as 'snapshot_too_large', last-good survives). - EOL feed redirects (L): max_redirects=3 with the trust-extension rationale documented (endoflife.date fronts a redirecting CDN, so redirects can't be off entirely). - Scope-filter audit trail (L): the purls of dropped components are now recorded (bounded sample of 200 + exact totals) in ScopeFilterResult and scan_metadata.scope_filter.dropped_refs — counts alone are not reviewable when the npm predicate trusts an attacker-controlled lockfile classification. - Symlink guards (L): Podfile.lock and the Podfile-detection glob reject symlinks (cosign-module precedent). - LIKE metacharacter escape + assert->explicit guard (Info).
- Cap each recorded dropped ref at 512 chars so 200 hostile mega-purls cannot inflate the scan_metadata JSONB row. - Correct the _podfile_present comment: the symlink guard covers the leaf only (glob traverses directory symlinks) — acceptable because the flag decides a CLI argument, not a read path; the read guard lives in read_podfile_lock.
![github-code-quality[bot]](https://avatars.githubusercontent.com/u/9919?s=60&v=4){kind=small}
| """A refactor that moves ``_persist_artifact`` or ``_sign_sbom`` ahead of | ||
| the scope filter would persist/sign the UNfiltered document while Trivy | ||
| and the component rows see the filtered one. Pin the order.""" | ||
| import tasks.scan_source as scan_source_module |
|
|
||
| from alembic import op | ||
|
|
||
| revision: str = "0038" |
| from alembic import op | ||
|
|
||
| revision: str = "0038" | ||
| down_revision: str | None = "0037" |
|
|
||
| revision: str = "0038" | ||
| down_revision: str | None = "0037" | ||
| branch_labels: str | Sequence[str] | None = None |
| revision: str = "0038" | ||
| down_revision: str | None = "0037" | ||
| branch_labels: str | Sequence[str] | None = None | ||
| depends_on: str | Sequence[str] | None = None |
|
|
||
| from alembic import op | ||
|
|
||
| revision: str = "0039" |
| from alembic import op | ||
|
|
||
| revision: str = "0039" | ||
| down_revision: str | None = "0038" |
|
|
||
| revision: str = "0039" | ||
| down_revision: str | None = "0038" | ||
| branch_labels: str | Sequence[str] | None = None |
| revision: str = "0039" | ||
| down_revision: str | None = "0038" | ||
| branch_labels: str | Sequence[str] | None = None | ||
| depends_on: str | Sequence[str] | None = None |
| from services.eol_health_service import get_eol_health | ||
|
|
||
| router = APIRouter(prefix="/eol", tags=["admin"]) | ||
| log = structlog.get_logger("admin.eol.api") |
Live-stack verification (docker-compose dev + Playwright + real worker scans of public repos) surfaced three defects the unit suites structurally could not see: - Hand-built responses dropped the new fields (third instance of this class): the component drawer built ComponentDetailResponse without the eol_* block and the overview built ProjectOverviewResponse without eol_count — Pydantic silently defaulted both, so the DB and the list carried the verdict while the drawer/overview read null (caught by components_eol e2e E3/E4). Fixed, and a new AST completeness contract (test_handbuilt_response_completeness.py) requires a keyword for every model field in both constructions so the fourth instance fails in CI. - Root-keyed consumers missed the git-clone subdirectory: _fetch_source returns workspace/source while the git path clones into source/repo, so the prep lockfile generators, the scope filter's and CocoaPods fill-in's lockfile reads, persist's W4-D npm scope enrichment and language detection all silently no-opped on every git scan (zip extractions, which land at source/ itself, were fine — which is why the fixture e2e never saw it). _resolve_project_root now hands them the repo root. Verified live: a chalk.git scan now drops 648 lockfile-dev components (47 kept) with the audit refs recorded; before the fix the same scan persisted all 695 with no telemetry. - components_scope_filter.spec.ts seeded with the default component prefix, which collides with earlier runs' catalog purls (component-mode purls are not run-suffixed) — the seed failed on every run after the first. Now mints a per-run prefix like the sibling EOL spec. Also verified live: maven-simple (all test-scope, no required) is left whole by the hasScopes guard exactly as designed, with Usage=Optional visible; the iOS guard survives a real cdxgen 12.3.3 run against a Podfile tree in the worker and fills 3 pods + the subspec edge back in.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Ports the three scan-precision improvements from BomLens v1.7.0 (parity gaps #15–#17, 7th review cycle) and closes the gap table again. Nine commits, one per reviewable unit (K-1/K-2/L/M-1/M-2/M-3 + parity docs + two security-hardening passes).
Phase K — runtime-scope SBOM filtering (default ON)
cdxgen keeps every resolved node, so a deployed app's SBOM also carried its test/provided/dev toolchain (junit, lombok, devDependencies) — inflating CVE counts, license obligations and NOTICE content for artifacts that never ship.
integrations/sbom_scope_filter.py: Maven drops cdxgen scopeoptional/excludedbehind a hasScopes guard (a scope-less SBOM is left whole — recall never regresses). Node deliberately diverges from BomLens: nonpm installre-resolve subprocess — we droppkg:npmcomponents only when the already-parsedpackage-lock.jsonpositively classifies themdev(hasDev guard + keep-if-unknown for purls the root lockfile does not cover)._run_pipelinebefore artifact persist, cosign signing, SCANOSS and Trivy, so every consumer sees one consistent document; a failed disk rewrite degrades to filter-skipped with memory and disk still identical (pinned by a lifecycle-sequence AST test).scan_metadata.scope_filterrecords counts and the dropped purls (bounded sample of 200), atrusca:scope_filterproperty rides the signed SBOM, and the Components summary band shows “N dev/test components excluded”.SCAN_SCOPE_FILTER_ENABLED/_MAVEN_ENABLED/_NODE_ENABLED, all default ON; only exact falsy tokens disable (a typo fails open to the correct-by-default behaviour — inverse ofSCANOSS_ENABLED, no egress here).Phase L — iOS CocoaPods/SPM lockfile scanning
With a
Podfilepresent and nopodCLI, cdxgen's cocoapods cataloger throws and aborts the whole stage-1 scan (terminalCdxgenFailedfor any iOS repo).--exclude-type cocoapodswhen a Podfile exists within depth 3 (Pods/ copies and symlinks ignored), applied in both the in-process adapter and the sidecarbuild_prep_source.sh(cross-referenced headers).integrations/cocoapods_lockfile.pyhand-parsesPodfile.lock'sPODS:block (npm_lockfile discipline — never raises, byte/line/entry caps, name→ref guard, self-edges dropped; no syft needed, diverging from BomLens) and merges components + dependency graph before the scope filter, same copy-then-commit shape. Subspecs (Moya/Core) percent-encode into the purl;dependency_scopestays NULL (no runtime/test signal in the lockfile).Package.resolvednow detect as swift, and the sidecar skipsswift package resolveover a committed lockfile. Verified empirically: cdxgen 12.3.3 without a swift toolchain parsesPackage.resolvedoffline — the capturedreal_cyclonedx_swift.jsonfixture is that verification run.Phase M — end-of-life (EOL) component flagging
An EOL runtime gets no upstream fixes — a later Critical CVE has no patch. A supply-chain risk axis distinct from CVEs, previously invisible.
eol_*columns oncomponent_versions+ a partial indexWHERE eol_state='eol'— the KEV shape (a fact about product+cycle, shared across scans, re-stampable). The purl→product whitelist is vendored verbatim from BomLens (byte-identity contract test; the%40npm-scope encoding is absorbed in the matcher), and the endoflife.date snapshot is vendored into the repo (16 KB, 10 products;scripts/refresh_eol_snapshot.pyrefreshes per release) — zero network at scan time, air-gap safe. Stamping happens insidepersist_sbom_components, covering source scans and SBOM ingest.eol, absence is the signal),?eol=truefacet riding the partial index, an always-rendered drawer row distinguishing untracked (blank) from unknown, an Overview KPI chip deep-linking to the filtered list.EOL_STATESis contract-tested on both sides.EOL_REFRESH_ENABLED— new third-party egress follows the SCANOSS fail-closed posture) with a proportional sanity floor and a 256 KiB snapshot size gate. Admin/health gains an endoflife.date panel (dataset age with a 180-day stale escalation) atGET /v1/admin/eol/health.Security review
Producer-Reviewer loop completed: initial review returned 3 Medium / 3 Low / 2 Info (CocoaPods parser ReDoS — measured 16 s on a crafted 80 k-char line, now < 1 s;
eol_cycleVARCHAR(32) overflow aborting persist outside the best-effort guard; unboundedeol_sync_state.snapshotJSONB; redirect bounds; audit trail; symlink guards; LIKE escaping). All fixed with regression tests; re-review verdict APPROVE with two non-blocking residuals, also applied.Verification
ruff check .andmypy .clean; migrations 0038/0039 applied against live PostgreSQL 17 (columns + partial index verified); OpenAPI snapshot regenerated for the new query param and endpoint.tsc -bclean; eslint clean; EN/KO locales mirrored; KO docs pass the ko-style lint (S1/S2 = 0).pod installPodfile.lock.components_eol.spec.ts,components_scope_filter.spec.ts) and the docs-uat marker for the new admin endpoint ride the standard suites.Docs
EN + KO in the same PR: runtime-scope filtering and end-of-life sections in the components guide (blank-vs-unknown semantics), iOS section in the scans guide, endoflife.date row in data-sources, the admin health panel section, all nine new env vars, and a CHANGELOG that calls out the count-drop surprise with its rollback switch. The stale WebSocket slug list in the scans guide (dt_upload/dt_findings era) was refreshed on the way.
Also carries the parity-review baseline move to
5a62094(v1.7.0 + 27) with gaps #15–#17 recorded as closed, plus two earlier doc changes that were stranded by the docs-only-PR check blockade (reachability-analysis coordination row,commercial-parity-candidates.md).