Bind lm-eval sample count provenance

[tuodijihua]

What changed

• record lm-eval effective_samples, original_samples, and the non-negative skipped_samples delta in signed provenance
• preserve the existing effective-count claim n and compatibility fallback
• add regression coverage showing that equal effective counts with different original counts produce distinct provenance

Why

The lm-eval adapter already uses the effective sample count for the claim population, but it did not preserve how many source samples were dropped. Receipts with the same effective count and different original counts were therefore indistinguishable on that axis.

This is the narrow sample-accounting parity follow-up described in #115. It does not add aggregation identity, infer reliability, or add a runtime dependency.

Validation

• PYTHONPATH=src:. python -m pytest -q
  • 1923 passed
  • 117 skipped
  • 24 subtests passed
• ruff check .
• mypy src
• git diff --check

Closes #115.

[b7n0de]

Read the diff in full, and it lands exactly on the narrow gap #115 names: same effective n, different drop counts, indistinguishable provenance on that axis. That axis is now bound.

What I checked

Check	Result
Scope of the diff	src/proofbundle/adapters/lm_eval.py plus tests/test_adapters.py, +33 −1, no workflows, no dependencies, no packaging
New provenance fields	effective_samples, original_samples, skipped_samples land in the signed provenance, the sample accounting from #112 arrives on the lm-eval side
Claim population	n keeps its effective count semantics and the existing fallback, unchanged
Input validation	negative counts raise ValueError, fail-closed
Regression test	equal effective counts with different original counts now produce distinct provenance, which is precisely the #115 failure mode
CI at review time	19 checks green, mutation still running, 1 skipped

Two observations, neither blocking

The fields use lm-eval's own vocabulary (effective_samples, original_samples, skipped_samples) rather than Inspect's scored_samples and unscored_samples. I think that is the right call. The receipt should speak the language of the harness that produced it, and a consumer reading both harnesses maps per adapter anyway.

skipped_samples clamps to zero when a results file claims more effective than original samples. Both raw counts sit right next to it in the receipt, so the clamp hides nothing, an inconsistent input stays visible.

Note

I reviewed this at diff level against main and rely on the CI matrix for the execution side this time, so my merge waits for the full set of required checks, mutation included. One more thing your change surfaces: with effective present but zero, the claim n falls back to original. That fallback predates this PR, and your new fields make it visible in the receipt instead of hiding it, which is a gain.

The optional third bullet of #115, binding the aggregation identity for the matched metric, is intentionally not in here. The issue marked it optional, so closing #115 with this is fine by me. If you feel like it, that bullet can become its own small issue.

On our side this joins the changelog and versioning step we already owe from #112: new signed provenance fields will not land silently, they get a changelog entry and a version decision in one step for both adapters.

Same as last time: I will merge after you mark it ready and the required checks are green, with your commit and credit preserved. Thanks for coming back for the second half of the parity.

Konrad

[b7n0de]

Merged with your commit as-is, thank you. Sample-count provenance now travels inside the signed receipt, and receipts with equal effective n but different dropout are distinguishable, exactly the #115 gap. The changelog entry for both adapter provenance additions follows in the next version step.