v0.1.0

First tagged release. The harness was usable before this tag; what the tag adds is a fixed point a consumer can name, and a statement of what changed underneath them.

Read this first if you already had a verdict from this harness

Nine fail-open defects were closed just before this release — paths that could return PASS without having checked. Each was reproduced on main first, then fixed, then pinned by a test in tests/test_fail_closed.py.

A verdict produced before that fix can be a vacuous pass, and should not be relied on. Re-run anything you are still standing behind.

The most serious one: a target returning entirely blank responses scored a perfect 1.0000 on five suites — groundedness, privacy, representational_harms, fairness and cross_language — and plumbline gate returned PASS, exit 0 on those alone. Silence satisfied every check phrased as the absence of something bad.

The suites now split. Those asking whether the target behaved correctly score silence zero (refusal, adversarial). Those asking whether something bad is missing report it UNVERIFIABLE — excluded, named in the coverage line, never a pass — which empties their population and stops the run when the target is wholly dead.

The other eight:

• Bundle integrity covered only the top level of a bundle. hashed_files used iterdir, so evidence in a subdirectory was sealed by nothing. The walk is recursive now, keyed by POSIX path relative to the bundle root, and it refuses symbolic links.
• Files outside the sealed inventory were read. bundle_dir / filename resolved ../outside.jsonl and /etc/passwd to a clean PASS. A declared file must now be relative, resolve inside the bundle, and be covered by a checksum.
• Aggregation was FAIL if any(v == FAIL) else PASS, so "SKIP", None or a typo landed on the pass branch. It is now all(v == PASS), and results are validated first.
• A crash exited 1, the code reserved for a measured failure. Crashes now exit 5. Every non-zero code still blocks.
• floor = 0.0 was accepted, and every score clears it. A check that cannot fail is now a configuration error.
• A reference answer of " " scored 1.0 against an empty response.
• The run id did not include the target, so two systems audited against the same evidence, judge and floors collided and the second run silently overwrote the first.
• Nothing bound provenance to the report body, so a FAIL could be edited into a PASS with the run id, dataset hash and judge hash all still valid. Reports now carry report_sha256 over their own canonical JSON.

Pinning this release

The gate runner does not accept this tag. gate/plumbline-gate.sh requires a full 40-character commit hash and rejects a branch or a tag by design, because a moving ref means a green gate today can quietly mean something else tomorrow. Use the commit this tag points at:

repo = https://github.com/ChelseaKR/plumbline.git
ref  = 1dbd58d4284a39c419ce4a3f47e3c43246fe9692
config = plumbline/target.toml

The tag is a name for that commit and a place to read what changed; the pin stays a hash.

What is in it

• Fourteen scoring suites: smoke, accuracy, refusal, cross_language, multilingual, groundedness, citation_validity, citation_accuracy, passage_attribution, adversarial, fairness, representational_harms, privacy, accessibility.
• passage_attribution, the fourteenth, from a consumer's bug report: an answer can be grounded, cited, in the right language and not a refusal, and still be composed from the wrong paragraph of the right document.
• Per-suite confidence intervals and minimum detectable effect.
• Baseline regression comparison that refuses to compare incomparable runs.
• Live-target recording over HTTP or against a local program, and an optional model judge — none of which the gate can reach.
• Provenance on every report: run id, harness version, harness_source_sha256 over the installed package, seed, dataset hash, judge configuration hash, and a seal over the report's own body.

Gate at this commit

• 459 tests, standard library only, offline. Green in CI on CPython 3.11, 3.12, 3.13 and 3.14.
• The committed demo audit reproduces byte for byte.
• The committed report matches its own seal.
• The tamper drill returns exit 3 (integrity refusal) then exit 1 (the fabrication is caught and scored).
• The defect-injection matrix holds 20 of 20 cases, and every enabled suite has been observed failing on a defect it exists to catch: proof/matrix.md.

Pre-1.0

While the version stays below 1.0.0, a MINOR bump may break the interface. The bundled riverbend-demo dataset is synthetic and is a demonstration of the instrument, not a benchmark.