Keystone 0.7.0

Data release 0.7.0 with package 0.7.0. keystone build rebuilds it from the v0.7.0 tag and checks it against 35
published hashes; under the same code the v0.6.0 tag still rebuilds byte for byte.

• A new family, alternative_evidence_hard. One value the decision rests on (a dose, a lab value, a vital sign, a
  weight, an age, a gestational age, a duration or time window) moves just across a named clinical threshold from a
  product label or guideline, so the correct action changes, and nothing else in the message points at the change.
  The reply has to read the number against the threshold to see it. The family is scored as a necessary update, like
  alternative_evidence. 113 twins, core 65, strict 64, primary 52; every twin records the threshold, where it comes
  from, the old and the new value, and what each side of the threshold implies. On its 52 primary twins, five
  assistants with GPT-4.1 as action judge reply acceptably to 0.89 of the originals and 0.80 of the twins. Their twin
  rates span 0.63 to 0.96 (sd 0.112), against 0.74 to 0.94 (sd 0.069) for necessary update on alternative_evidence
  in the reference results, and all five agree on only 52 percent of the twins: the threshold crossing spreads the
  assistants further apart than alternative_evidence does. It is an additional family, not one of the
  pre-specified confirmatory ones.
• 26 new HealthBench sources. Conversations from the HealthBench oss split outside the seven base strata, each
  with an alternative_evidence_hard twin and the decision frame of its original message. The release covers 1,260
  sources, 999 dev and 261 test.
• demographic_shift rewritten with real attribute swaps. 577 twins whose attribute came from a small set of
  recurring templates now change an attribute that bears on the decision, and every label was redone on the new
  text: mechanism, advice_should_change, evidence state and its review, materiality, rubric dependence. They carry
  revision: 2 and supersedes: 1; the revision-1 rows move to deprecated_twins.jsonl with superseded_by: 2.
  The family grows from 385 core, 339 strict and 169 primary twins to 843, 776 and 437, and its twins with a
  mechanical defect flag fall from 755 to 181.
• Action lists, version 2. On 2,274 released twins the acceptable actions, forbidden actions and decisive
  questions were reviewed comment by comment: the drafting vendor adjudicated every comment of the state reviewer,
  and an edit was applied only when a second vendor agreed. 1,356 twins carry at least one of the 2,284 applied
  edits. Revised rows carry lists_version: 2 and keep the version-1 lists beside the new ones. Judged against both
  versions, the same replies of three assistants on 60 changed twins change verdict 9 times in 180: 7 replies that a
  too-broad version-1 forbidden item had penalised are now acceptable, and 2 replies that ignore a contradiction are
  now caught. No layer rule reads the lists.
• The quick set is published and kept across releases (release/quick_set.json): the 319 twins of 0.6.0, with
  the same text on every one, plus 31 alternative_evidence_hard twins, 350 in all. Version 2 revised the action
  lists of 94 of the 319.
• Counts: 7,419 twins over 1,260 sources, core 5,454, strict 5,252, primary 4,410, quick 350, and 2,371 superseded
  rows in deprecated_twins.jsonl. MANIFEST.json adds the primary count per family, the version-2 list counts, and
  the base and new sources.
• The reference results, the confirmatory run and the analysis pages were computed on 0.6.0 and stay labelled 0.6.0.
• The package changes listed under 0.6.1 ship in this release; 0.6.1 was not published on its own.

Keystone 0.4.0

A clinical decision–evidence benchmark for chat assistants and the rubrics that grade them, built on HealthBench (OpenAI, MIT).

What is in this release

• 7,318 twins over 1,236 HealthBench conversations from seven physician-agreed strata. Each twin changes exactly one thing; a paraphrase-only control on 1,119 sources bounds what any change can be attributed to.
• Eight families. Four on every source: missing_evidence, conflicting_evidence, demographic_shift, salient_distractor (negative control). Four more where the source allows: alternative_evidence, demographic_control (second negative control), buried_red_flag, missing_evidence_early.
• Decision–evidence layer. Every source carries a decision frame: what is being decided, the action the stated evidence supports, the facts it rests on. Every twin carries the evidence state after the edit, with the actions a clinician would accept, the actions that are now wrong, and the questions that would settle it.
• Layers. all · core 4,803 · strict 4,682 · primary 4,105 · a fixed quick set of 303 twins for a cheap first run. Sources split 80/20 into dev and test by a hash of the source id.
• Action outcomes scored by a judge from another vendor, with the physician rubric's applicability scored alongside the model, so a reply that never noticed the gap cannot inherit the original's rubric score.
• Reference results for five assistants, every reply and classification released.

Reproducing it

The repository contains no HealthBench prose. A twin is stored as span references plus our own literals; python tools/build_release.py downloads HealthBench from OpenAI and rebuilds dist/ byte for byte, and CI does exactly that on every push.

pip install -e ".[dev]"
python tools/build_release.py
keystone pairs

Clinicians

Every label here is written by a model and checked by a second model from another vendor. The external anchor is a clinician who reads the item and disagrees. Three rating tasks, a first slice of 35 items, no software and no patient data: open an issue with the clinician label.

MIT for the twins, controls, code and prompts. Source conversations and rubrics are HealthBench, © OpenAI, MIT. Keep the canary field and do not post items in plain text on the open web.