Evidence and Test Matrix

Evidence and Test Matrix

Interpretation rule

The table below reports values published for v2.2.6-public-eval. A green count demonstrates that the published harness observed the expected results in its measured source tree. It is not, by itself, production certification, formal conformance, proof of non-bypassability, or independent validation.

Evidence family	Command	Published v2.2.6 result	What it helps establish
Static source manifest	python verify_manifest.py	PASS	Tracked public source agrees with the published manifest.
Public packet validation	python tools/validate_public_eval_packet.py	PASS	Required reviewer-facing files are present and internally consistent.
Release lineage	python tools/check_release_lineage.py	PASS	Active version and release tuple point to the intended release.
IETF release pointers	python tools/check_ietf126_release_pointers.py	PASS / 0 findings	Reviewer links target the current public-evaluation tag.
Selected IETF packet	python ietf126/run_review_packet.py	20 / 20 PASS	Selected field, vector, and behavior checks match expected outcomes.
Independent recomputation	python ietf126/independent_recompute.py	17 / 17 PASS	Canonical bytes and digests are recomputed without importing the main verifier package.
Independent crypto	python ietf126/independent_crypto_verify.py	19 / 19 PASS	Receipt, revocation, and carrier signatures are checked through an independent path.
Public evaluation vectors	python run_vectors.py	76 / 76 PASS	Positive and fail-closed negative vectors match expected decisions.
Strict pytest	PYTEST_DISABLE_PLUGIN_AUTOLOAD=1 PYTHONWARNINGS=error python -m pytest -q -p no:cacheprovider	323 / 323 PASS	Automated unit and integration expectations pass with plugins disabled and warnings fatal.
Full package coverage	make coverage	99.33% statements / 98.44% branches	Broad execution coverage across orprg_eval; not a security proof.
Security-critical coverage	make coverage	100.00% lines / 99.705% branches	High measured coverage across the explicitly selected security-critical module set.
Verifier branches	make coverage	100.00%	Every measured verifier branch executed in the published suite.
Strict-schema branches	make coverage	100.00%	Every measured strict-schema branch executed in the published suite.
Full release preflight	make qa-full	PASS	Aggregated evaluation, metadata, release, and integrity gates pass.
Deterministic build	python tools/build_release_asset.py ... twice plus byte comparison	exact-byte comparison required	The four release assets are reproducible in the locked environment.

Evidence hierarchy

Use evidence in this order:

1. Immutable bytes — ZIP and sidecars downloaded from the release.
2. Cryptographic bindings — checksum, manifest, provenance, and source tag.
3. Re-execution — commands run in a recorded clean environment.
4. Independent paths — recomputation and crypto verification that do not simply call the main verifier.
5. Test reports and coverage — useful breadth indicators, not substitutes for threat analysis.
6. Narrative claims — README, release notes, and wiki summaries.

What coverage does not prove

High line or branch coverage does not prove that:

• every effect-relevant field was included in canonicalization;
• test assertions are correct;
• cryptographic trust configuration is appropriate;
• all production paths traverse the gate;
• replay state is durable under every deployment failure mode;
• policy is correct or non-ambiguous; or
• an attacker cannot find an unmodeled state or integration path.

Coverage should be paired with negative vectors, independent recomputation, mutation testing, structured fuzzing, external implementations, and deployment-level bypass analysis.

Independent reviewer record

A credible external reproduction should publish:

• release tag and commit;
• downloaded artifact hashes;
• operating system, architecture, Python, pip, and dependency mode;
• exact commands;
• full pass/fail counts;
• deviations from documented setup;
• any generated file differences; and
• a statement that no production or sensitive data was used.

Use Contributing and Feedback for public-safe findings.