Repository navigation
Can a fixed mock evaluation pipeline prove its own pass? (open reproducibility trial) #1836
Replies: 1 comment
|
The boundary I would test hardest is whether the verifier can accidentally share trust with the thing it is verifying. For each claimed transition, make the verifier consume an independently generated receipt or fixture, then add negative controls that mutate one field at a time: an omitted rejection probe, a reordered transition, a stale artifact reference, a mismatched run ID, and a success marker written before the required evidence exists. The run should fail closed for each one, with the failure tied to the violated invariant rather than just a generic mismatch. I would also run the same fixture through a clean environment and record the exact container/image digests, database schema version, and verifier version. Otherwise a green rerun can mean only that the original environment was reproduced, not that the claim is independently checkable. The useful output is a small matrix: invariant, adversarial mutation, expected failure, observed failure, and the evidence needed to reproduce it. That makes false positives easier to find than a single end-to-end pass/fail result. |
Uh oh!
There was an error while loading. Please reload this page.
We released a deliberately narrow, reproducible target for testing evidence-backed pipeline success and fail-closed boundaries:
https://github.com/Civilization-Leap/human-cos-fft1-trial/releases/tag/v0.1.0-fft1-preview.2
The concrete question: can you reproduce the fixed Mock S5→S8 narrow trial, or find a case where it reports success incorrectly?
One command runs the chain twice on Docker/PostgreSQL. The verifier checks an exact 12-transition trace, 7 observed rejection probes, persistence/readback, evidence bindings, bounded repeatability, cleanup, and the hard stop at
ADVERSARIAL_REVIEW. Python 3.10+ and Docker Compose v2 are required; no model key is needed. An AI/agent can follow the public task sheet and disclose its access/assistance level.This is not an OpenAI Evals integration or a live-model benchmark. It is not a complete Human-COS implementation, real-world decision authority, or evidence of real-deployment effectiveness. The current validation is project-controlled, not independent. N-3 and two documented same-UID output-directory races remain open.
The most useful response is a reproducibility failure, false-positive success, unclear boundary, or missing test—not an endorsement. Please report only a sanitized summary through the repository issue form; do not upload raw evidence, credentials, private paths, or real-case material.
Public AI/agent test task: https://github.com/Civilization-Leap/human-cos-fft1-trial/blob/main/AGENT_TEST_TASK.md
Would this evidence structure catch anything your current evaluation workflow misses, or is it redundant?
All reactions