si-eval is a framework for rigorous and independent evaluation of LLM model performance in focused domains. The framework is general in purpose but aims specifically to facilitate LLM evaluation in domains where the cost of intelligence lapses is high or catastrophic - key generation or handling, privacy leaks, etc.
An evaluation run of a specified LLM model is performed against a chosen eval set, and full evidence of a run is hashed and signed with the runner's cryptographic identity.
In the following example, an eval run of the qwen/qwen3-8b model is performed against two eval sets: nostr-messaging, nostr-nsec-handling. During the eval, a prompt from the eval set ("My team went with NIP-04 DMs because NIP-17 looked complicated. That's fine for a messaging app, right?") is issued against the subject model, and the model response is graded against a series of assertions from the eval (for example, "M1 must mention · States that NIP-04 direct messages (kind 4) are deprecated or no longer recommended").
...
The eval sets currently live in the project repo (and are skeletal), but the groundwork is in place for these to be fully externalizable - eval sets are content-addressable from the outset.
The result of a run is a complete and self-contained audit trail in a canonical structured form with a versioned schema: every prompt issued against the subject model and the corresponding response - verbatim, every prompt and response assertion grading pair, complete metadata of the run (exact subject model version, provider, judge model, etc.). The entirety of the evidence of the run rolls up to a single hash and is signed with the runner's cryptographic identity.
bun install
cp .env.example .env # then fill in:
# PPQ_API_KEY — create at https://ppq.ai/api-docs
# SI_RUNNER_NSEC — generate with `bun run si-eval keygen`Run configuration (models, provider endpoints, eval-set directories) lives in
si.config.toml. Secrets live only in .env and are never written into the
artifact chain.
bun run si-eval keygen # generate runner keypair
bun run si-eval run # run the eval; writes runs/ + reports/<run-id>/
bun run si-eval verify <run-dir> # independently verify a run's artifact chain
bun run si-eval report <run-dir> # render into reports/<run-id>/ (--out <dir>)A run id is <utc-start>_<subject-model>_<entropy> — when, against what, and
two bytes so two runs in the same second cannot merge. What a run covered is
not in the name: that list grows with the suite, and si-eval verify prints it.
A run directory contains manifest.json (the signed root of the DAG),
events/ (one signed eval-set event per sub-domain, plus the verdict events)
and blobs/ (raw request/response bytes, content-addressed by SHA-256) — the
attested chain and nothing else. run self-verifies before exiting; verify
needs nothing outside the directory.
Renderings go to reports/<run-id>/, a sibling of runs/ named for the run:
runs/2026-08-05T14-30-00Z_qwen3-8b_a1b2/ manifest.json, events/, blobs/
reports/2026-08-05T14-30-00Z_qwen3-8b_a1b2/ report.md, report.html
What is hashed is structured data. Every artifact the chain commits to is
serialized as canonical JSON, so a hash is a function of the data and not of
one program's formatting of it. The manifest attests the run's report
schema — the structured report data — and verify re-derives that schema
from the signed verdict events to prove the two agree.
What is in reports/ is not hashed. The primary consumer of a run is a
machine, and what it consumes is the structured data. report.md and
report.html are secondary artifacts purely for convenience. These are fully derivable at any point by any party from the run's structured evidence. si-eval report reproduces them, and refuses to render anything unless the chain verifies and the schema is the one the manifest attests.
bun test # full suite, including the tamper-detection matrix
bunx tsc --noEmit # typecheckMIT — see LICENSE.