v0.2.0

What's Changed

• Standards: mechanical lane by @ChelseaKR in #2
• A suite that stopped running is not "nothing changed" by @ChelseaKR in #3
• build(deps): bump actions/checkout from 4.4.0 to 7.0.1 by @dependabot[bot] in #4
• build(deps): bump astral-sh/setup-uv from 9.0.0 to 10.0.0 by @dependabot[bot] in #6
• build(deps): bump actions/setup-python from 5.6.0 to 7.0.0 by @dependabot[bot] in #7
• build(deps-dev): update setuptools requirement from >=68 to >=84.0.0 in the python-dependencies group by @dependabot[bot] in #5
• sign: detached HMAC-SHA256 report signatures by @ChelseaKR in #8
• sarif: project findings for consuming repos' PR annotations by @ChelseaKR in #9
• history: append-only run history, and a trend view over it by @ChelseaKR in #10
• retire: recording retention and redaction lifecycle by @ChelseaKR in #11
• supply-chain: SBOM, OpenSSF Scorecard, keyless-signed release by @ChelseaKR in #12
• conversational_integrity: score every turn, not only the final one by @ChelseaKR in #13
• release: 0.2.0, and the provenance the version bump moved by @ChelseaKR in #14
• action: run the gate from GitHub Actions directly, pinned by uses: by @ChelseaKR in #15
• packaging: prepare PyPI publish as plumbline-eval by @ChelseaKR in #16
• docs: model card for the optional model judge by @ChelseaKR in #17
• docs: a dated responsible-tech statement by @ChelseaKR in #18
• docs: a longer-form "what it caught" writeup for publication by @ChelseaKR in #19
• mypy: wire it into make lint and CI, and fix what it found by @ChelseaKR in #20
• site: hold the published page to the accessibility standard it names by @ChelseaKR in #21
• docs: an operations runbook by @ChelseaKR in #22
• docs: a Definition of Done and a metrics ledger by @ChelseaKR in #23
• docs: move Quick start to the top, fold Unreleased into 0.2.0 by @ChelseaKR in #24
• fix: actions/upload-artifact pin was a 39-character SHA, not a real commit by @ChelseaKR in #25

New Contributors

• @ChelseaKR made their first contribution in #2
• @dependabot[bot] made their first contribution in #4

Full Changelog: v0.1.0...v0.2.0

v0.1.0

First tagged release. The harness was usable before this tag; what the tag adds is a fixed point a consumer can name, and a statement of what changed underneath them.

Read this first if you already had a verdict from this harness

Nine fail-open defects were closed just before this release — paths that could return PASS without having checked. Each was reproduced on main first, then fixed, then pinned by a test in tests/test_fail_closed.py.

A verdict produced before that fix can be a vacuous pass, and should not be relied on. Re-run anything you are still standing behind.

The most serious one: a target returning entirely blank responses scored a perfect 1.0000 on five suites — groundedness, privacy, representational_harms, fairness and cross_language — and plumbline gate returned PASS, exit 0 on those alone. Silence satisfied every check phrased as the absence of something bad.

The suites now split. Those asking whether the target behaved correctly score silence zero (refusal, adversarial). Those asking whether something bad is missing report it UNVERIFIABLE — excluded, named in the coverage line, never a pass — which empties their population and stops the run when the target is wholly dead.

The other eight:

• Bundle integrity covered only the top level of a bundle. hashed_files used iterdir, so evidence in a subdirectory was sealed by nothing. The walk is recursive now, keyed by POSIX path relative to the bundle root, and it refuses symbolic links.
• Files outside the sealed inventory were read. bundle_dir / filename resolved ../outside.jsonl and /etc/passwd to a clean PASS. A declared file must now be relative, resolve inside the bundle, and be covered by a checksum.
• Aggregation was FAIL if any(v == FAIL) else PASS, so "SKIP", None or a typo landed on the pass branch. It is now all(v == PASS), and results are validated first.
• A crash exited 1, the code reserved for a measured failure. Crashes now exit 5. Every non-zero code still blocks.
• floor = 0.0 was accepted, and every score clears it. A check that cannot fail is now a configuration error.
• A reference answer of " " scored 1.0 against an empty response.
• The run id did not include the target, so two systems audited against the same evidence, judge and floors collided and the second run silently overwrote the first.
• Nothing bound provenance to the report body, so a FAIL could be edited into a PASS with the run id, dataset hash and judge hash all still valid. Reports now carry report_sha256 over their own canonical JSON.

Pinning this release

The gate runner does not accept this tag. gate/plumbline-gate.sh requires a full 40-character commit hash and rejects a branch or a tag by design, because a moving ref means a green gate today can quietly mean something else tomorrow. Use the commit this tag points at:

repo = https://github.com/ChelseaKR/plumbline.git
ref  = 1dbd58d4284a39c419ce4a3f47e3c43246fe9692
config = plumbline/target.toml

The tag is a name for that commit and a place to read what changed; the pin stays a hash.

What is in it

• Fourteen scoring suites: smoke, accuracy, refusal, cross_language, multilingual, groundedness, citation_validity, citation_accuracy, passage_attribution, adversarial, fairness, representational_harms, privacy, accessibility.
• passage_attribution, the fourteenth, from a consumer's bug report: an answer can be grounded, cited, in the right language and not a refusal, and still be composed from the wrong paragraph of the right document.
• Per-suite confidence intervals and minimum detectable effect.
• Baseline regression comparison that refuses to compare incomparable runs.
• Live-target recording over HTTP or against a local program, and an optional model judge — none of which the gate can reach.
• Provenance on every report: run id, harness version, harness_source_sha256 over the installed package, seed, dataset hash, judge configuration hash, and a seal over the report's own body.

Gate at this commit

• 459 tests, standard library only, offline. Green in CI on CPython 3.11, 3.12, 3.13 and 3.14.
• The committed demo audit reproduces byte for byte.
• The committed report matches its own seal.
• The tamper drill returns exit 3 (integrity refusal) then exit 1 (the fabrication is caught and scored).
• The defect-injection matrix holds 20 of 20 cases, and every enabled suite has been observed failing on a defect it exists to catch: proof/matrix.md.

Pre-1.0

While the version stays below 1.0.0, a MINOR bump may break the interface. The bundled riverbend-demo dataset is synthetic and is a demonstration of the instrument, not a benchmark.