You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
This commit was created on GitHub.com and signed with GitHub’s verified signature.
Changed
Verb-first CLI: --study replaces the pre-verb launch flags. The study
flags now live on the subcommands that spawn a study
(list/run/score/doctor), so an invocation reads verb first: mira run --study study.rs instead of mira --script study.rs run. --study PATH resolves the runner by extension (.rs → the cargo-script
shim, .py → uv run); the specific runners are --study-script, --study-uv, --study-python (runs python3; replaces both --python and --python3), --study-bin, --study-example, and --study-cmd, mutually
exclusive. report and publish no longer carry launch flags (they only
read saved runs). The old spellings (--script/--bin/--example/--cmd/ --uv/--python/--python3) still parse — before or after the verb — as
hidden deprecated aliases that warn and point at the replacement; they
disappear from help, docs, and examples, and will be removed at 1.0. --launcher/--package/--manifest-path keep their names but are now
subcommand-scoped too (write mira run --launcher NAME, not mira --launcher NAME run). mira.toml launcher keys are unchanged.
Fixed
README overview diagram now renders on off-repo surfaces such as crates.io by
using a registry-safe absolute SVG URL.
Added
mira export <run_id> --format atif. A new host subcommand that emits
one standalone ATIF-v1.7 trajectory document per case from a saved run, for
SFT / RL / trajectory-visualization tooling. Read-only over the run store (no
study process, no re-execution, like report). Each document's steps come
from the case's real trajectory when an execution artifact carries one
(--artifacts DIR); otherwise a trajectory is synthesized from the saved flat
summary fields via the new Trajectory::from_transcript (the lossy inverse of project_into). Mira's verdict is stamped into the document's ATIF root extra — reward = { pass, score, scorers: [{ name, value, pass, reason }] }
and mira = { eval, sample, target, run_id } provenance — on export only,
never on the study↔host wire. Output is one <case_key>.atif.json per case
under results/<run_id>/export/ (or --out DIR); --out - streams all
documents as NDJSON to stdout. Skipped (unexecuted) cases are omitted. No wire
or schema change — export is CLI/host-only.
Trajectory scorers. Four deterministic scorers grade the structure of
the ATIF trajectory — the data only Transcript.trajectory carries: tool_called_with(tool, pointer, expected) (some invocation of tool has
the JSON value expected at the RFC 6901 JSON Pointer in its arguments), tool_arg_matches(tool, pointer, regex) (the regex variant, string values
only — a non-string at the pointer fails with a reason), observation_contains(tool, needle) (the observation content correlated via source_call_id contains a substring; multimodal content is graded on its
text projection), and steps_within(max) (the ATIF step-count budget,
distinct from turns_within). A transcript without a trajectory fails
them with reason "subject reported no trajectory" (the ttft_within
precedent: an unverifiable check fails; N/A stays reserved for infra). All
four are hand-mirrored in the Python and TypeScript SDKs and pinned by new schema/v1/conformance/scorers.json vectors run by all three parity
runners. Docs, specs, and the agent skill now consistently position the ATIF trajectory as the primary structured contract, with events as the
advanced debug channel not to be used where the trajectory covers the need.
ATIF trajectory producers. Both built-in agent-shaped subjects now emit
the structured trajectory. CliSubject gains TranscriptSource::AtifFile("trajectory.json") — the recommended source
for tool-using external agents: the agent writes one ATIF JSON document into
the workdir (absolute path hinted via a new MIRA_TRAJECTORY_PATH env var,
alongside MIRA_TARGET/MIRA_PROVIDER), Mira parses it into Transcript.trajectory and derives every flat field from it; a missing or
invalid file becomes a subject-kind Transcript.error. The JSONL events
variants remain as the advanced path for producer-shaped streams. And mira-everruns gains atif_from_events, a typed fold of the everruns Event stream into ATIF — one agent step per reasoning iteration, with
structured tool calls (tool.started/tool.call_requested arguments
included), observations correlated via source_call_id (failures counted,
same policy as tool-name extraction), reasoning content, and per-step token
metrics — which RuntimeSubject now attaches to every transcript (the raw events channel is unchanged).
ATIF trajectories: the primary structured trajectory contract (protocol
1.1).Transcript gains an optional trajectory field carrying an ATIF
document (new mira::trajectory module: steps with structured tool calls,
arguments, correlated observations, per-step reasoning and metrics — emitted
as ATIF-v1.7; any ATIF-v1.x parses, other prefixes are rejected
gracefully). The flat fields (final_response, tool_calls, iterations, usage) are now projections of the trajectory: the framework derives them
wherever a transcript is produced or received (fill-if-default, never
overwriting explicit values), so a subject or polyglot study can return {"trajectory": …} alone and every existing scorer keeps working — Transcript::from_trajectory is the one-call constructor, and Transcript::tool_invocations() exposes names + arguments + observation
content (trajectory-first, falling back to the legacy name list). events is
repositioned as an advanced, producer-shaped debug channel — independent of
and never required alongside the trajectory. The protocol bumps 1.0 → 1.1
(additive) with a new trajectory capability and capability_params.trajectory = {format: "ATIF", version: "1.7"}; both SDKs
gain the generated Trajectory wire types, a hand-mirrored projection
(mira.trajectory / trajectory.ts), and serve-loop normalization, all
pinned by a new three-runner conformance fixture
(schema/v1/conformance/trajectory.json).
Single-file studies (--script study.rs). A study no longer needs a
crate: write one .rs file with cargo-script frontmatter (RFC 3502) for its
deps and run it with mira --script study.rs. cargo -Zscript is nightly-only,
so the host shims it onto stable — it parses the frontmatter, materializes a
content-hashed throwaway crate (re-anchoring relative path deps, adding a [[bin]] and an isolating [workspace]), and cargo runs it with a shared
target dir. The file format matches native cargo-script, so the same study runs
under cargo -Zscript unchanged; MIRA_SCRIPT_NATIVE=1 opts into that today.
Most bundled examples (examples/<name>.rs) are now single-file; multi-file /
heavy-dep ones (cli_subject, metrics, matrix, llmsim) stay crates.
mira doctor. One-command diagnosis of a Mira setup, in three layers: mira.toml (parse errors, unknown/misspelled keys with a "did you mean"
suggestion, launcher mistakes, presets and timeouts that can't work), the
study's advertised listing (duplicate sample ids / target labels / axis
values that collide case keys, empty datasets and matrices, unavailable
targets, presets and [targets.LABEL] sections that match nothing), and the
saved-run store (interrupted runs, torn temp files, invalid case results,
missing reports). mira doctor --fix applies the safe repairs: removing
leftover *.tmp files and re-rendering a finished run's missing report.json/report.html from its stored results. Warnings never fail;
errors exit non-zero, so doctor can gate CI.
Publish runs to everruns. A new mira-publish-everruns crate plus mira publish <run_id> and mira run --publish everruns send a saved run's
results to an everruns instance, which hosts and
visualizes eval results it did not execute. Credentials reuse the everruns
CLI: --everruns-* flags, then EVERRUNS_API_KEY/EVERRUNS_API_URL/ EVERRUNS_ORG_ID, then ~/.config/everruns/credentials.json — so a prior everruns login is enough. One Mira run becomes one everruns run group (one
EvalRun per eval, idempotent on the run id); everruns trusts Mira's verdict
and does not re-grade.
Fixed
mira-everruns reported zero tool calls.RuntimeSubject filled Transcript.tool_calls via Mira's generic summarize_events, which only
recognizes { name, input } tool objects — a shape the everruns event stream
never emits (tool calls arrive as tool.completed events keyed by data.tool_name). Every tool-selection scorer silently saw zero calls while
the tools actually ran. The adapter now extracts tool names from the tool.completed events it owns, with a regression test pinning that event
shape against schema drift (EVE-676).
The repository's own mira.toml placed default_launcher below the [environment] section header, which nests it inside that table — so the
setting was silently ignored (found by mira doctor). It now sits at the
top level.