Skip to content

Repair v0.9 semantic decoy coverage gates#396

Merged
AbdelStark merged 1 commit into
mainfrom
issue-389-v0-9-semantic-decoy-coverage
Jun 6, 2026
Merged

Repair v0.9 semantic decoy coverage gates#396
AbdelStark merged 1 commit into
mainfrom
issue-389-v0-9-semantic-decoy-coverage

Conversation

@AbdelStark
Copy link
Copy Markdown
Owner

Parent #385
Closes #389

Summary

  • add execution-decoy coverage summaries with generated, pack-level, candidate, scorable, and missing-record counts by category
  • block semantic score gates with typed semantic_decoy_pair_count_blocker reasons when scorable pair minimums are not met
  • tighten semantic decoy rows to preserve same-source benchmark/problem alignment and record schema metadata
  • document pre-score coverage reporting in the API, observability spec, and benchmark template

Validation

  • uv run pytest tests/eval/test_execution_surprise_decoys.py tests/eval/test_execution_surprise_claim_gates.py tests/eval/test_semantic_decoy_pack.py tests/eval/test_execution_eval_cli.py tests/docs/test_execution_substrate_docs.py tests/docs/test_release_templates.py
  • uv run python -m compileall -q -x 'tests/fixtures/codestate/invalid_(before|after)\\.py$' codelewm tests scripts/build-passfail-pack\n- uv run pytest tests/\n- git diff --check\n

@AbdelStark AbdelStark merged commit a3e11ac into main Jun 6, 2026
9 checks passed
@AbdelStark AbdelStark deleted the issue-389-v0-9-semantic-decoy-coverage branch June 6, 2026 10:00
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

v0.9 eval: repair semantic-decoy alignment and coverage gates

1 participant