Repository navigation
Model-graded evals kept passing my agent's bad runs — so I moved the verdict to a deterministic kernel #3584
tantanpq
started this conversation in
Show and tell
Replies: 0 comments
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Uh oh!
There was an error while loading. Please reload this page.
The failure mode that pushed me here: an LLM judge grading an LLM's work.
Scores drifted, failed runs got rephrased into passing ones, and the attempt
that actually failed somehow never made it into the report.
SimLab Core is the alternative I've been running. The agent's output bundle
is frozen and hash-bound. Every check is declared before the run and executed
by plain comparison — JSON and text equality/inclusion, hash equality. The
kernel's budget for model calls is zero, enforced at admission: there is no
code path where a model votes on the outcome.
Two properties matter most to me:
the ledger shows the failures too.
canary result and they disagree, the run gets marked contradicted by
reality and can be revoked. The evidence dies; nothing downstream is
touched silently.
The ceiling is deliberate: the best verdict is PASS_CANDIDATE_ONLY. The
kernel says "these frozen files passed these declared checks" and stops
there. Truth about the world stays with your harness.
I'm testing paid demand this week: $49 gets you the kernel download — run
certification checks on your own bundles. If you work on agent evals, I want your sharpest objection. Does a
deterministic seal like this solve a real problem for you, or is the trust
gap somewhere else entirely?
All reactions