haenv 1.3.0

HAEnv 1.3.0

HAEnv generates synthetic longitudinal patient records, poses one clinical decision at an index time T, and scores health agents against a gold answer that the world fixed before the record was written. 1.3.0 is the first release since 1.2.2.

Highlights

• Four task packs on one patient world: ① budgeted differential diagnosis, ② triage, ③ medication adjustment, ④ lab-change attribution. Each pack is a pipeline: tools/generation/make_pack.py builds a pack of any size from a seed and audits it, with no model call.
• 1.3.0 board: eight models, 100 questions per pack, three answers per question. The main score averages S = G·D·(1+Q)/2 over the three answers and weights the four packs equally; 95% intervals come from a paired question-cluster bootstrap. gpt-6-sol leads at 0.831 (0.808–0.854). The evaluated question sets ship as frozen pack files, and the board is recomputed from stored answers without calling a model.
• Code-graded decisions: in every pack the core decision is checked by code against the world's truth; one LLM judge scores only management-plan quality Q. A cross-family judge check keeps the model order.
• Twin records (haenv twin): one-fact variants of a built batch for controlled comparisons.
• Paid-run tooling: a shared spend ledger with per-provider budgets, concurrency probing (haenv probe concurrency), an offline rehearsal against a fake provider (haenv loadtest), and resumable runs that resend only missing cells.
• Interactive demo: build a patient, turn it into a question and compare every model's answers with the truth at https://thetahealth.github.io/mirobody-env/

Breaking changes from 1.2.2

• Source layout: the packages moved under src/ (src/haenv, src/haenv_kernel, src/verifier_core); job files moved to benchmarks/inputs/, frozen sets to benchmarks/frozen/, and tools into subfolders such as tools/generation/ and tools/evaluation/.
• Outputs: runs write under artifacts/ (set HAENV_OUTPUT_ROOT to choose another root) instead of results/ next to the job.
• Packs: pack ② is rebuilt (--pack p2, wearable-trend triage); the earlier pack2 triage job remains only for the historical boards.
• Scores are not comparable with 1.2.x boards: the judging and world fingerprints changed.

Upgrading

pip install -U haenv            # or: git pull && uv sync

Update paths in your own scripts to the new layout (benchmarks/inputs/..., tools/generation/make_pack.py) and read outputs from artifacts/. Re-score stored batches with the current code before comparing them with the 1.3.0 board; every batch records the fingerprints it was scored with.

See the [changelog](https://github.com/thetahealth/mirobody-env/blob/v1.3.0/CHANGELOG.md), the [board](https://github.com/thetahealth/mirobody-env/blob/v1.3.0/README.md#leaderboard) and the [data card](https://github.com/thetahealth/mirobody-env/blob/v1.3.0/docs/DATA_CARD.md).

All patients are synthetic. Nothing here is medical advice.

haenv 1.2.2

haenv 1.2.2

Documentation-only patch release. Runtime code, dependencies, generation and scoring rules,
frozen packs, and leaderboard data are unchanged from 1.2.1.

Documentation

• Keep the English and Chinese README concise while explaining how new synthesis tasks
  combine patient timelines, noise and event-density controls with applicable shared checks
  and task-specific verification.
• Add a bilingual rare-disease case study with reused components, examples of verification
  failures, and the explicit batch-gate exception. Link it from the README and documentation index.
• Clarify that the rare-disease implementation remains on rare-code-bench and is not bundled
  in this package. The workflow uses the existing haenv-synth and haenv-extend skills.

Install

pip install haenv==1.2.2

Changelog ·
Case study ·
中文案例

haenv 1.2.1

haenv 1.2.1

This release changes how paid runs settle and resume, makes the generation-time audit and the
build faster, and updates the demo page. Scores do not change: the 1.2.0 board, re-scored from
its stored responses with 1.2.1, gives the same scores, intervals and pair results for every model
and pack. The judging fingerprint moves from a5f058bdd9de77b9 to 950578a687aacc5e because the
files it covers changed in how a run resumes, how a paid request settles and how the shortcut
check is computed; the world fingerprint (633cb0576e0472f0) is unchanged.

Paid runs

• A paid OpenRouter request records its generation id from the response headers as soon as they
  arrive. A request cut off before its answer is saved settles at the provider's recorded cost,
  on resume and through haenv spend reconcile; only a request with no id stays at its reserved
  bound. haenv spend reconcile also reads in-flight markers and failed judge receipts.
• A resumed run resends only the aborts the harness side caused: harness errors, and empty or
  unparseable replies whose last failed attempt was a network, timeout, 5xx, 429 or key-pool
  failure. An empty body, broken JSON, a reply cut at max_tokens and a spent tool budget are
  the model's result and stay as they are.
• The cost line names tariff-priced charges (Google, DashScope) as estimates.

Building and auditing packs

• The generation-time audit draws label shuffles only when a score reaches its line. The
  permutation p readings are a profile, computed with --profile on tools/pack_audit.py and
  tools/make_pack.py and reported as "not computed" otherwise.
• Deterministic builds write cases.jsonl and payloads.jsonl on the generation processes and
  compute the shortcut check's permutation p-values in parallel. Files, digests and p-values are
  byte-identical to a serial run.

Demo page

• The patient builder applies the production emission gate: a patient production would refuse
  is not drawn, the page lists the reasons, shows the refused course greyed, and suggests the
  nearest accepted setting. The dose record is drawn.
• Act 2 shows the record streams under each chart.
• Act 3 puts the overall board on its own row, adds a pair matrix of all ten models, and shows
  pack ① in the model panel.

Install

pip install haenv==1.2.1
haenv --help

Full history: CHANGELOG.md.

haenv 1.2.0

haenv 1.2.0

haenv is a health evaluation environment: one pipeline builds the world, writes the questions,
runs the models and writes the report. This release publishes the whole pipeline — the code that
evaluates, generates questions and builds worlds, the job files of four task packs, the question
generators and the generation-time audit. It can be used to evaluate models and to generate
training data.

Four task packs

Pack	What it asks	Decision	Scoring
① Differential diagnosis	buy the tests that matter; find a second condition behind findings another one explains	diagnosis lines and required tests	code first, one LLM judge for what code cannot match
② Acute triage	where the patient should go now	disposition (4 classes)	code
③ Chronic medication adjustment	the next step for a treated patient	medication decision (5 classes)	code
④ Follow-up interpretation	whether a change since the last result is real, noise, pre-analytical or a method difference, and its size	source of change (4 classes) and size	code

• One request per item; everything needed to answer is in the question. Pack ① keeps buying tests on demand.
• Gold answers are functions of the world's truth and are recomputed from the world's records for every item at generation time.
• Classes are dealt by hash inside each stratum (with or without a hidden condition), independent of every other case feature.

Packs as a pipeline

• tools/make_pack.py --pack {m2,pack2,p3,p4} --n N [--seed S] writes the job, builds it with the
  deterministic generator and runs the audit, with no model call. A pack of any size follows each
  pack's proportion table (largest remainder); a cell that cannot be filled stops the build and
  names the shortfall.
• The same pack, N and seed give a byte-identical pack. Generation runs on a process pool whose
  output equals a serial run.
• The shipped job files are public sample packs at N = 50 with the public seed. Official boards use
  a private seed: batches record its sha256, N and the job, never the seed; the seed of a retired
  board is published so it can be reproduced.

Scoring

• Chance-corrected decision scores, cc = (m · BA − 1) / (m − 1): a constant or uniformly random answer scores 0.
• Every measure is scored, profile (computed and shown, not in the composite) or retired.
• Confidence intervals use a two-level bootstrap over items and repeated answers.
• The over-triage and missed-clinician-review gates are reported outside the hard-gate multiplier;
  clinician review is scored once, as review_utility_cc.

The 1.2.0 board

Ten models answer the four packs, 50 items each, every item three times; the packs are drawn with a
private seed (sha256 8f90c70f…7f9b9). The overall score is the equal-weight mean of the four
pack scores, with 95% intervals from a case × round bootstrap (4,000 draws).

Rank	Model	Overall	95% CI
1	gemini-3.1-pro	0.732	0.674–0.788
2	gemini-3.7-flash	0.677	0.602–0.742
3	gpt-6-sol	0.671	0.617–0.729
4	deepseek-v4-pro	0.560	0.484–0.634
5	gpt-6-luna	0.547	0.470–0.616
6	kimi-k3	0.547	0.470–0.622
7	minimax-m3	0.521	0.439–0.600
8	glm-5.3-flash	0.483	0.408–0.564
9	deepseek-v4-flash	0.426	0.347–0.500
10	qwen3.7-flash	0.389	0.317–0.465

Pack leaders: ① gemini-3.1-pro 0.834 · ② gemini-3.7-flash 0.532 · ③ minimax-m3 0.806 ·
④ gpt-6-sol 0.886. Among the top six, 9 of 15 pairs are separable overall (difference beyond
1.96 standard errors); by pack ① 8, ② 0, ③ 3, ④ 10. The per-pack table and the board data are in
the README and web/demo/board.json.

Generation-time audit

tools/pack_audit.py is shared by the four packs: gold recomputation, gold or answer text in the
question, realism, sex consistency, unregistered phrases and surface shortcuts. A question-blind
stub blocks a batch when its chance-corrected score is at least 0.20 and at least the 99th
percentile of the same maximum under 1,000 within-stratum label shuffles. haenv run scores a pack
only with a passing audit marker. The audit needs scikit-learn and joblib.

World

• Longitudinal values (labs, vital signs, wearables, weight) are rendered by code from the world's truth.
• Lab observations carry pre-analytical factors, method differences and biological variation.
• Case composition carries background comorbidity; in packs ③ and ④ a known comorbidity can change the gold answer.

Framework

• The judging kernel is the package haenv_kernel.
• Packs plug in through entry points that register their judges and gold blocks; an installed wheel builds and scores the four packs.
• Default models are reached directly: Gemini through Google, Qwen through DashScope, the others
  through OpenRouter pinned to one upstream with fallbacks off.

Relation to 1.x

• Judging and world fingerprints both change: batches and boards of 1.2.0 are not comparable with 1.x.
• Pack ① keeps 16 bridge items byte-identical to v1.0.1.
• The two frozen v1.0.1 question packs ship unchanged.

Install

pip install haenv==1.2.0
haenv --help

The four quick-start commands (build, verify, run --offline, report) make no model call.

Full history: CHANGELOG.md.

v1.1.1

The judging fingerprint (67dd5866790e9cb9), the world fingerprint (25f31527b09e6abe), the
frozen question packs and the board are those of 1.1.0. The solving code changed: a billed batch
started under 1.1.0 does not resume under 1.1.1; finish it under 1.1.0, or run it again with
--fresh.

Fixed

• On Python 3.10 the fingerprints of unchanged code differed from those computed on 3.11 and
  later: under 1.1.0 the judging fingerprint read 275d059bec243917 instead of 67dd5866790e9cb9
  and the world fingerprint 6966e2d16b58a0ca instead of 25f31527b09e6abe. Rows scored on 3.10
  could not share a board with rows scored on 3.11 or later, and a billed batch could not resume
  across the two. The fingerprints hash source normalised by ast.unparse; Python 3.11 changed
  its output, and 3.10's Unicode 13.0 database escapes the characters Unicode 14.0 added. On 3.10
  the normalisation now follows 3.11's rules, and Python 3.10 to 3.14 give this release the same
  judging, world and solving fingerprints.
• The agent skills' frontmatter is valid YAML holding only the fields the Agent Skills standard
  defines. Two descriptions contained : , which strict YAML parsers reject, and argument-hint
  is not a field of the standard.
• README: the "Evaluate your own agent" example runs with --limit 1, which skips semantic
  judging, needs no OpenRouter key and exits 6 to mark the scores incomplete. The README says so;
  the OpenRouter key and the judge's cost belong to a run without --limit.
• On a release, the package description's tree/ links name the release tag, as its blob/ links
  and figure addresses already did.

Changed

• CI installs and runs the wheel built from the sdist, the same way the release builds it.

Install: pip install haenv==1.1.1. Full history: CHANGELOG.md.

v1.1.0

The judging fingerprint (67dd5866790e9cb9), the world fingerprint (25f31527b09e6abe), the
frozen question packs and the board are those of 1.0.1. The solving code changed: a billed batch
started under 1.0.1 does not resume under 1.1.0; finish it under 1.0.1, or run it again with
--fresh.

Added

• A model on a backend of your own can take part in a billed run. Its configuration entry
  declares price (input_per_million_usd, output_per_million_usd; 0 for a free endpoint);
  each request is reserved at that price and counted from the token counts the endpoint returns
  in usage, against the same --judge-budget-usd as the semantic judge. The ledger books these
  costs as tariff bounds (a declared price is not an invoice). A batch keeps the price it was
  created with: a changed price does not resume it. On openrouter, relay, google and
  dashscope the price still comes from the provider, and a declared price there is refused.
• Three agent skills ship under skills/, in the Agent Skills layout (skills/<name>/SKILL.md):
  haenv-synth (generate patients and questions), haenv-bench (run a pack and read the board),
  haenv-extend (add a judge, a stream, an event or a task type from outside the repository). The
  README says what they are for.
• The README states what a task is here: a gold-standard shape plus an answer contract, with the
  format (single / gated / slices / multi) a condition inside it, and one pipeline generating,
  gating and scoring all of them. A case is assigned to a task by the gold it carries, not by the
  job file's label.

Changed

• haenv run with a real model checks its keys before it opens a batch. A model or the semantic
  judge without a key is named with the variable it needs and the state of the key file, and the
  run exits 2 with nothing sent. A run with --limit is not judged and needs no judge key.
• A refusal from the billed-run accounting (an unverified route, an unreachable price endpoint,
  a changed run identity) is one printed line and exit code 2 instead of a traceback. The Google
  capacity check says whether the endpoint was not reached or answered with an HTTP error, and
  gives up after 45 s as a whole: its 30 s timeout applied to each connection attempt, and with the
  route to the endpoint dropped it waited for more than six minutes.
• On a release, the package description names the release tag instead of main in its figure
  and link addresses, so each version's PyPI page keeps that version's files.
• The README (English and Chinese) returns to the 1.0.0 layout, and its one animation is the
  terminal recording of the quick start. The demo animations, the first-screen buttons and the
  comparison matrix are no longer in it. The corrections made for 1.0.1 remain: nine hard gates
  enter the multiplier and acted_on_unverified_signal is reported only; the semantic dimensions
  are judged by one LLM judge, whose OpenRouter key the env file needs; the token scale is per
  answered cell; figures and documents are linked by absolute URL.

Fixed

• README: the "Evaluate your own agent" example stopped with "Cannot budget unverified solver
  route(s)": no price source existed for a backend of one's own. The example now declares price.
• README: the "Install from PyPI" commands run again in the same directory (build opens a new
  batch with --fresh).
• README: after pip install, your own backends and models go in a file named by
  HAENV_CONFIG_OVERLAY, since the packaged config.yaml is inside the installed package.
• README: regenerating a diagnosis pack offline (--gen deterministic) emits 144 of its 145
  specifications; JD-32v2 fails its anchor check.
• CONTRIBUTING: commit types are feat, fix, refactor, docs, test and chore; CI,
  packaging and release changes are chore. It listed ci and build as types of their own.
  A release reaches main as a short series of commits, one per change, ending in the
  commit that sets the version.

Install: pip install haenv==1.1.0. Full history: CHANGELOG.md.

v1.0.1

The judging fingerprint moves from 25ced8ada8427f45 to 67dd5866790e9cb9; the world fingerprint
(25f31527b09e6abe) and the frozen question packs are those of 1.0.0. No score changes: the six
release batches were re-scored under the new code from their stored answers, all 3,417 rows carry
the scored values they carried in 1.0.0, and the board is identical.

The judging segment moved with how a judging run executes and how its identity is compared, not
with what it scores: the number of judge lanes is no longer part of a run's judging identity, a run
resumes across comment-only and infra-only changes, and judge lane scheduling and result writing
belong to the infra segment.

Changed

• Configuration has a typed schema with one declared default per key and a role on every field, so
  a misspelled key inside a section is an error. --set KEY=VALUE overrides one value, and each run
  appends the resolved configuration and the source of each layer to <batch>/config_runs.jsonl.
• pydantic>=2.6 is a runtime dependency, required by the typed schema.
• A judge run resumes across comment-only and infra-only changes.
• A judge run given no lane count uses 10 lanes; the policy files no longer carry one (they
  carried 64 or 4).
• The budget ledger (budget.json) is append-only with periodic compaction: after its first line
  it holds one change per line, so read it with haenv.semantic_budget.read_state, not
  json.load.
• eval.model_wall_budget_s (off by default) bounds a model's total wall clock: once it is spent,
  no new cell of that model is started, and the cells left unstarted are recorded in the batch's
  scheduler.json (wall_budget_stops).
• New connections are paced, and a connection that fails before any request bytes are sent is
  retried instead of being counted as a model failure.
• Demo: a patient timeline, an answer matrix, a step-by-step replay and result charts; the board
  is ranked in tiers with 95% intervals from a case-level bootstrap, alongside the noise floor and
  the separable model pairs; the English page is fully in English.
• README (English and Chinese) is organised around four demo animations, presents the leaderboard
  in tiers, and links figures and documents by absolute URL so that it renders on the package
  index.
• The source distribution no longer carries the README figures.
• LICENSE holds the MIT text alone, so that it is detected as MIT; the data licence and the
  kernel's own licence are stated in LICENSE-DATA and core/LICENSE.
• CONTRIBUTING.md states the commit and release-branch rules.

Fixed

• docs/DATA_CARD.md describes the hard gates as graded: nine of the ten gates are
  non-compensatory and one is reported outside the multiplier; in slices the action-level gates
  also zero the slice where they fire. The scoring registry's entries are listed by where their
  readings are reported.
• The demo's count of generated patient records covers generated batches only. A batch whose case
  bodies were copied from another one (a re-score or an arm split) no longer adds its source's
  records again, so the count is 25,400 rather than 30,619; the measured dirty-data rates are
  unchanged.

v1.0.0

First public release.

• Two question packs, 145 synthetic cases each, 64 conditions: ddx-timeline (questions at several time points) and ddx-workup (test ordering against a check budget).
• Ten models evaluated on both; one case is scored by code-derived gold, and the board is preliminary (see the data card).
• The leading models tie on the composite and separate on clinical behaviour: no pair inside the first tier is separable on the composite, while clinician review, grounded tool targets and test selection split that tier.
• pip install haenv runs the whole pipeline offline: generate patients, verify every item, answer and score, with no API keys and no cost.
• Published wheel and sdist build from the public tree; the release workflow uploads through PyPI trusted publishing.

The data card lists the scoring rules and every known gap; the README covers the quick start.