Skip to content

spec: Proposal to wire Scorecard to ExecutionContext - #353

Open
justaugustus wants to merge 1 commit into
darnitdevorg:mainfrom
justaugustus:scorecard
Open

spec: Proposal to wire Scorecard to ExecutionContext#353
justaugustus wants to merge 1 commit into
darnitdevorg:mainfrom
justaugustus:scorecard

Conversation

@justaugustus

Copy link
Copy Markdown
Collaborator

Summary

Draft proposal for #194 — giving the shared ExecutionContext.get_or_run_tool infrastructure from #189 its first real consumer, using OpenSSF Scorecard as the motivating heavy tool.

This PR adds only the proposal doc (specs/021-scorecard-execution-context/proposal.md) — no code yet. It records two maintainer-confirmed design decisions so we have them written down before implementation:

  • Invocation backend: uwu-tools/scorecard-mcp (MCP over stdio) behind a swappable ScorecardBackend seam, so a live-scan CLI backend can be added later without touching caching or mapping logic.
  • Verdict policy: evidence-only. Scorecard never changes a control's status; it only attaches evidence/hints that later passes can use. This matches both darnit's conservative-by-default constitution and scorecard-mcp's own framing (cached / opt-in / partial data; "heuristic signals, not a verdict").

The proposal also documents the existing latent infra (with file:line anchors), the backend seam + handler design, candidate Scorecard-check → OSPS-control mappings, a phased implementation plan, and open questions — notably #194's acceptance criterion 2 (cache_key consumed by adapter dispatch), flagged because the live audit path is the sieve-handler path and CheckAdapter dispatch appears vestigial.

Type of Change

  • Documentation update

Framework Changes Checklist

If this PR modifies the darnit framework (packages/darnit/):

  • Updated framework spec (docs/architecture/framework-design.md) if behavior changed
  • Ran uv run python scripts/validate_sync.py --verbose and it passes

N/A — proposal doc only; no framework code changes.

Control/TOML Changes Checklist

If this PR modifies controls or TOML configuration:

  • Control metadata defined in TOML (not Python code)

N/A — no control/TOML changes in this PR.

Testing

  • Tests pass locally (uv run pytest tests/ -v)

N/A — documentation only.

Additional Notes

Requesting review from @mlieberman85.

Two things worth deciding before this graduates to a full spec.md/plan.md:

  1. Acceptance criterion 2 resolution — amend the bullet (handler path is the real consumer) vs. wire the currently-unused adapter cache_key path. Proposal recommends amend.
  2. The exact Scorecard-check → OSPS-control set for the first cut (minimum two from the mapping table).

Refs: #194 · Depends-on: #189

…itdevorg#194)

Draft proposal (specs/021-scorecard-execution-context/proposal.md) for
giving the shared ExecutionContext.get_or_run_tool infrastructure from
darnitdevorg#189 its first real consumer, using OpenSSF Scorecard as the motivating
heavy tool.

Captures two maintainer-confirmed design decisions:
- Invocation backend: uwu-tools/scorecard-mcp (MCP over stdio) behind a
  swappable ScorecardBackend seam, so a live-scan CLI backend can be
  added later without touching caching or mapping logic.
- Verdict policy: evidence-only. Scorecard never changes a control's
  status; it only attaches evidence/hints. This matches both darnit's
  conservative-by-default constitution and scorecard-mcp's own framing
  (cached/opt-in/partial data; 'heuristic signals, not a verdict').

Also documents:
- Existing latent infra (get_or_run_tool plumbing, normalize_scorecard_output,
  evidence-on-INCONCLUSIVE support) with file:line anchors.
- Backend seam + evidence-only handler design, per-repo cache key, and
  candidate Scorecard-check -> OSPS-control mappings.
- Mapping to darnitdevorg#194 acceptance criteria; criterion 2 (cache_key consumed by
  adapter dispatch) flagged as the one open item, since the live audit
  path is the sieve-handler path and CheckAdapter dispatch is vestigial.
- Phased implementation plan, open questions, and non-goals.

Refs: darnitdevorg#194
Depends-on: darnitdevorg#189

Co-Authored-By: Claude <noreply@anthropic.com>
Signed-off-by: Stephen Augustus <foo@auggie.dev>
@justaugustus justaugustus changed the title docs(scorecard): proposal to wire Scorecard to ExecutionContext (#194) docs(scorecard): proposal to wire Scorecard to ExecutionContext Aug 2, 2026
@justaugustus justaugustus changed the title docs(scorecard): proposal to wire Scorecard to ExecutionContext spec: Proposal to wire Scorecard to ExecutionContext Aug 2, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant