fix(tribunal): independently verify evidence and reconcile holdout state#9
Open
MatinDeevv wants to merge 1 commit into
Open
fix(tribunal): independently verify evidence and reconcile holdout state#9MatinDeevv wants to merge 1 commit into
MatinDeevv wants to merge 1 commit into
Conversation
There was a problem hiding this comment.
Pull request overview
This PR strengthens the Edge Tribunal’s “fail-closed” posture by separating descriptive vs. invariant dataset identity, adding physical artifact hashing/verification hooks, and introducing idempotent holdout registry operations intended to support independent evidence verification and safer holdout state reconciliation.
Changes:
- Split dataset identity into content fingerprint vs. layout/manifest hashes; add optional physical byte verification during
bind_data. - Add independent evaluation-row parsing + metric recomputation utilities and wire them into
record_evidencewhen physical artifact bindings are provided. - Make holdout claiming/consumption more idempotent and introduce (currently-unwired) registry reconciliation helpers; update tests/examples/docs to reflect the stricter “RESEARCH_ONLY unless independently verified” stance.
Reviewed changes
Copilot reviewed 24 out of 24 changed files in this pull request and generated 6 comments.
Show a summary per file
| File | Description |
|---|---|
| tests/test_edge_tribunal_verdict.py | Updates expectations to cap self-attested passing experiments at RESEARCH_ONLY. |
| tests/test_edge_tribunal_robustness_coverage.py | Adds coverage for mandatory robustness “missing” cells counting as failures. |
| tests/test_edge_tribunal_registry_reconciliation.py | Adds tests for idempotent claims/consumption and failed bind publication compensation behavior. |
| tests/test_edge_tribunal_independent_recompute.py | Adds tests for strict evaluation rows + metric recomputation and deterministic bootstrap behavior. |
| tests/test_edge_tribunal_dataset_identity.py | Adds tests for content fingerprint invariance vs. manifest metadata/layout. |
| tests/test_edge_tribunal_cli.py | Updates CLI expected verdict list to include RESEARCH_ONLY in the passing scenario. |
| tests/test_edge_tribunal_artifact_binding.py | Adds tests for artifact hashing constraints and physical test receipt/log verification. |
| examples/edge-tribunal/synthetic-scenarios.json | Updates expected verdict for the “forward-test-eligible” scenario to RESEARCH_ONLY. |
| examples/edge-tribunal/run_synthetic_examples.py | Records binding SHA into evidence payload prior to evidence mutation/finalization. |
| engine/experiments/verdict.py | Adds an independent-verification gate that demotes promotion rank when verification is incomplete. |
| engine/experiments/test_receipts.py | Introduces strict verification of immutable test receipts and associated logs. |
| engine/experiments/robustness.py | Enforces planned-cell registry semantics (when declared) and treats missing mandatory cells as failures. |
| engine/experiments/registry_reconciliation.py | Adds fail-closed reconciliation helpers between binding snapshots and holdout registry (currently unused). |
| engine/experiments/holdout_registry.py | Adds registry revisioning, idempotent claim/consume primitives, and compensation via release-on-owned-failure. |
| engine/experiments/evidence.py | Strengthens robustness evidence validation to ensure required slices are represented for v2-style plans. |
| engine/experiments/evidence_recompute.py | Adds independent metric + concentration recomputation and deterministic block bootstrap utilities. |
| engine/experiments/evaluation_rows.py | Adds strict JSONL contract parsing/validation for evaluation rows. |
| engine/experiments/edge_tribunal.py | Adds optional physical dataset binding verification and optional independent recompute path in record_evidence. |
| engine/experiments/dataset_binding.py | Adds dataset content fingerprint + layout hash fields; tightens contract mismatch checks; extends binding artifact fields. |
| engine/experiments/artifact_binding.py | Introduces allowed-root constrained physical hashing for artifact bindings. |
| engine/config/schemas/edge-tribunal-evidence.schema.json | Allows independent_verification in evidence schema. |
| engine/config/schemas/edge-tribunal-dataset-binding.schema.json | Adds required binding fields for content fingerprint/layout hash/byte verification and holdout claim token+revision. |
| docs/preregistered-experiments.md | Documents why dataset names/provenance prose do not establish untouched identity. |
| docs/edge-tribunal.md | Documents the independent verification boundary, dataset identity split, and intended holdout registry behaviors. |
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
Comment on lines
+189
to
+193
| verification = binding.get("independent_verification", {}) | ||
| required_verification = ( | ||
| "dataset_bytes_verified", "evaluation_rows_verified", "metrics_recomputed", | ||
| "uncertainty_recomputed", "test_receipts_verified", "registry_reconciled", | ||
| "robustness_plan_complete", |
Comment on lines
+445
to
+468
| if physical_file_bindings is not None: | ||
| if dataset_root is None: | ||
| raise dataset_binding.DatasetBindingError( | ||
| "dataset_root is required with physical_file_bindings") | ||
| from engine.experiments.artifact_binding import bind_physical_files | ||
| verified = bind_physical_files( | ||
| physical_file_bindings, allowed_root=dataset_root, | ||
| error_type=dataset_binding.DatasetBindingError) | ||
| by_path = {item["logical_path"]: item for item in verified} | ||
| dataset_manifest = dict(dataset_manifest) | ||
| files: list[dict[str, Any]] = [] | ||
| for item in dataset_manifest.get("files", []): | ||
| logical = normalize_logical_path(item["logical_path"]) | ||
| if logical not in by_path: | ||
| raise dataset_binding.DatasetBindingError( | ||
| f"no physical binding supplied for {logical}") | ||
| computed = by_path[logical] | ||
| if item.get("sha256") != computed["sha256"]: | ||
| raise dataset_binding.DatasetBindingError(f"hash mismatch for {logical}") | ||
| files.append(dict(item, size_bytes=computed["size_bytes"])) | ||
| if set(by_path) != {normalize_logical_path(item["logical_path"]) for item in files}: | ||
| raise dataset_binding.DatasetBindingError("unexpected physical dataset binding") | ||
| dataset_manifest["files"] = files | ||
| dataset_manifest["dataset_bytes_verified"] = True |
Comment on lines
+572
to
+581
| comparisons = { | ||
| "accepted_rows": payload["population"]["accepted_rows"], | ||
| "class_counts": payload["population"]["class_counts"], | ||
| "model_brier": payload["primary_model"]["multiclass_brier"], | ||
| "comparator_brier": payload["primary_comparator"]["multiclass_brier"], | ||
| "brier_improvement": payload["primary_model"]["brier_improvement"], | ||
| "model_log_loss": payload["primary_model"]["multiclass_log_loss"], | ||
| "comparator_log_loss": payload["primary_comparator"]["multiclass_log_loss"], | ||
| "log_loss_improvement": payload["primary_model"]["log_loss_improvement"], | ||
| } |
| invariant = { | ||
| "files": files, | ||
| "pair_scope": sorted(manifest["pair_scope"]), | ||
| "pair_order": list(manifest.get("pair_order", manifest["pair_scope"])), |
Comment on lines
141
to
+144
| if not isinstance(manifest["untouched_claim"], bool): | ||
| errors.append("untouched_claim must be a boolean") | ||
| elif manifest["untouched_claim"] is not True: | ||
| errors.append("untouched_claim must be exactly true for a clean dataset binding") |
Comment on lines
+12
to
+16
| def reconcile_binding(binding: dict[str, Any], registry_root: Path) -> dict[str, Any]: | ||
| document = holdout_registry.load_registry(registry_root) | ||
| matches = [entry for entry in document["entries"] | ||
| if entry["holdout_id"] == binding.get("holdout_id")] | ||
| if len(matches) != 1: |
MatinDeevv
force-pushed
the
agent1/tribunal-independent-verification
branch
3 times, most recently
from
July 19, 2026 22:10
91f560a to
6e72a9d
Compare
MatinDeevv
force-pushed
the
agent1/tribunal-independent-verification
branch
from
July 19, 2026 22:21
6e72a9d to
a9445e3
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Verification
4fc8880f1dd2194f9a58111d0e4fa5ceda958e65a9445e308e5242fe67d9c134b58fcfcc25817926252 passed, 158 deselected(27.29s,26.07s)410 passed in 301.37sgit diff --check: passedLimitations
The registry operation is a recoverable saga rather than cross-filesystem atomicity. Snapshot-chain tamper evidence assumes a trusted retained current/root commitment; a malicious owner replacing the complete tree and external root can rewrite history. The feature does not authorize trading and historical BID-only evidence cannot produce a live verdict.