Skip to content

Add artifact evidence index#156

Merged
ProfRandom92 merged 1 commit into
mainfrom
codex/add-deterministic-artifact-evidence-index
May 20, 2026
Merged

Add artifact evidence index#156
ProfRandom92 merged 1 commit into
mainfrom
codex/add-deterministic-artifact-evidence-index

Conversation

@ProfRandom92

Copy link
Copy Markdown
Owner

Motivation

  • Provide a small deterministic, machine-readable index over committed evidence and visualization artifacts so reviewers can see which generator owns each artifact, what evidence category it represents, which fixture families it covers, and whether it is deterministic / LLM-free / external-api-free.

Description

  • Add scripts/generate_evidence_index.py, a pure-Python deterministic generator that uses a static artifact mapping, inspects only checked-in artifacts, extracts top-level JSON keys and explicit families/family values, computes manifest-alignment, and writes a stable pretty JSON index to artifacts/evidence_index.json.
  • Commit the generated artifacts/evidence_index.json containing sorted artifact entries, JSON/SVG classification, evidence-bearing vs visualization-only flags, manifest alignment indicators, and deterministic/no-LLM/no-external-API signals.
  • Add tests/test_evidence_index.py that verifies artifact existence, generator reproducibility, top-level schema stability, determinism and sanitization (no timestamps/generated_at/env/user/host/hash/digest/absolute paths), path existence, sorted entries, JSON top-level key extraction, SVG visualization-only marking, and summary-count correctness.
  • Changed files: scripts/generate_evidence_index.py, artifacts/evidence_index.json, and tests/test_evidence_index.py; scope note: deterministic evidence index only with no fixture, taxonomy, README, validator, workflow, or hash/digest changes.

Testing

  • Ran the generator with python scripts/generate_evidence_index.py and confirmed the output path artifacts/evidence_index.json was produced as committed.
  • Ran focused tests pytest tests/test_evidence_index.py -q and related artifact tests (tests/test_failure_taxonomy.py, tests/test_fixture_manifest.py, tests/test_capability_boundary_replay_artifact.py, tests/test_tool_ordering_replay_artifact.py, tests/test_graph_diff_artifact.py) and observed they passed; full project checks including npm run check and the full pytest suite also passed.
  • Risks: this is metadata-only indexing and depends on the stability of committed artifact schemas and conventions, so future schema changes will require updating the static mapping and regenerating the index.

Codex Task

@ProfRandom92 ProfRandom92 merged commit fc17b46 into main May 20, 2026
4 checks passed

@gemini-code-assist gemini-code-assist Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Code Review

This pull request introduces a deterministic evidence index system, including a generation script, the resulting JSON index, and a comprehensive test suite. The scripts/generate_evidence_index.py script aggregates metadata from various artifacts, while tests/test_evidence_index.py ensures schema stability, determinism, and data integrity. Feedback was provided to improve the readability of the artifact filtering logic in the generation script by utilizing the walrus operator.

Comment on lines +137 to +142
artifacts = [
entry
for spec in sorted(ARTIFACT_SPECS, key=lambda item: item["path"])
for entry in [_build_artifact_entry(spec, manifest_families)]
if entry is not None
]

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

medium

The nested list comprehension used here to filter None values is a bit convoluted. Since the project targets Python 3.9+ (as evidenced by the use of dict[str, Any]), you can use the assignment expression (walrus operator) to make this more readable and idiomatic.

    artifacts = [
        entry
        for spec in sorted(ARTIFACT_SPECS, key=lambda item: item["path"])
        if (entry := _build_artifact_entry(spec, manifest_families)) is not None
    ]

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant