Skip to content

Release v0.4.0

Choose a tag to compare

@github-actions github-actions released this 11 Jul 15:54
· 10 commits to main since this release
9c62727

Changed

  • Verb-first CLI: --study replaces the pre-verb launch flags. The study
    flags now live on the subcommands that spawn a study
    (list/run/score/doctor), so an invocation reads verb first:
    mira run --study study.rs instead of mira --script study.rs run.
    --study PATH resolves the runner by extension (.rs → the cargo-script
    shim, .pyuv run); the specific runners are --study-script,
    --study-uv, --study-python (runs python3; replaces both --python and
    --python3), --study-bin, --study-example, and --study-cmd, mutually
    exclusive. report and publish no longer carry launch flags (they only
    read saved runs). The old spellings (--script/--bin/--example/--cmd/
    --uv/--python/--python3) still parse — before or after the verb — as
    hidden deprecated aliases that warn and point at the replacement; they
    disappear from help, docs, and examples, and will be removed at 1.0.
    --launcher/--package/--manifest-path keep their names but are now
    subcommand-scoped too (write mira run --launcher NAME, not
    mira --launcher NAME run). mira.toml launcher keys are unchanged.

Fixed

  • README overview diagram now renders on off-repo surfaces such as crates.io by
    using a registry-safe absolute SVG URL.

Added

  • mira export <run_id> --format atif. A new host subcommand that emits
    one standalone ATIF-v1.7 trajectory document per case from a saved run, for
    SFT / RL / trajectory-visualization tooling. Read-only over the run store (no
    study process, no re-execution, like report). Each document's steps come
    from the case's real trajectory when an execution artifact carries one
    (--artifacts DIR); otherwise a trajectory is synthesized from the saved flat
    summary fields via the new Trajectory::from_transcript (the lossy inverse of
    project_into). Mira's verdict is stamped into the document's ATIF root
    extrareward = { pass, score, scorers: [{ name, value, pass, reason }] }
    and mira = { eval, sample, target, run_id } provenance — on export only,
    never on the study↔host wire. Output is one <case_key>.atif.json per case
    under results/<run_id>/export/ (or --out DIR); --out - streams all
    documents as NDJSON to stdout. Skipped (unexecuted) cases are omitted. No wire
    or schema change — export is CLI/host-only.
  • Trajectory scorers. Four deterministic scorers grade the structure of
    the ATIF trajectory — the data only Transcript.trajectory carries:
    tool_called_with(tool, pointer, expected) (some invocation of tool has
    the JSON value expected at the RFC 6901 JSON Pointer in its arguments),
    tool_arg_matches(tool, pointer, regex) (the regex variant, string values
    only — a non-string at the pointer fails with a reason),
    observation_contains(tool, needle) (the observation content correlated via
    source_call_id contains a substring; multimodal content is graded on its
    text projection), and steps_within(max) (the ATIF step-count budget,
    distinct from turns_within). A transcript without a trajectory fails
    them with reason "subject reported no trajectory" (the ttft_within
    precedent: an unverifiable check fails; N/A stays reserved for infra). All
    four are hand-mirrored in the Python and TypeScript SDKs and pinned by new
    schema/v1/conformance/scorers.json vectors run by all three parity
    runners. Docs, specs, and the agent skill now consistently position the ATIF
    trajectory as the primary structured contract, with events as the
    advanced debug channel not to be used where the trajectory covers the need.
  • ATIF trajectory producers. Both built-in agent-shaped subjects now emit
    the structured trajectory. CliSubject gains
    TranscriptSource::AtifFile("trajectory.json") — the recommended source
    for tool-using external agents: the agent writes one ATIF JSON document into
    the workdir (absolute path hinted via a new MIRA_TRAJECTORY_PATH env var,
    alongside MIRA_TARGET/MIRA_PROVIDER), Mira parses it into
    Transcript.trajectory and derives every flat field from it; a missing or
    invalid file becomes a subject-kind Transcript.error. The JSONL events
    variants remain as the advanced path for producer-shaped streams. And
    mira-everruns gains atif_from_events, a typed fold of the everruns
    Event stream into ATIF — one agent step per reasoning iteration, with
    structured tool calls (tool.started/tool.call_requested arguments
    included), observations correlated via source_call_id (failures counted,
    same policy as tool-name extraction), reasoning content, and per-step token
    metrics — which RuntimeSubject now attaches to every transcript (the raw
    events channel is unchanged).
  • ATIF trajectories: the primary structured trajectory contract (protocol
    1.1).
    Transcript gains an optional trajectory field carrying an
    ATIF
    document (new mira::trajectory module: steps with structured tool calls,
    arguments, correlated observations, per-step reasoning and metrics — emitted
    as ATIF-v1.7; any ATIF-v1.x parses, other prefixes are rejected
    gracefully). The flat fields (final_response, tool_calls, iterations,
    usage) are now projections of the trajectory: the framework derives them
    wherever a transcript is produced or received (fill-if-default, never
    overwriting explicit values), so a subject or polyglot study can return
    {"trajectory": …} alone and every existing scorer keeps working —
    Transcript::from_trajectory is the one-call constructor, and
    Transcript::tool_invocations() exposes names + arguments + observation
    content (trajectory-first, falling back to the legacy name list). events is
    repositioned as an advanced, producer-shaped debug channel — independent of
    and never required alongside the trajectory. The protocol bumps 1.0 → 1.1
    (additive) with a new trajectory capability and
    capability_params.trajectory = {format: "ATIF", version: "1.7"}; both SDKs
    gain the generated Trajectory wire types, a hand-mirrored projection
    (mira.trajectory / trajectory.ts), and serve-loop normalization, all
    pinned by a new three-runner conformance fixture
    (schema/v1/conformance/trajectory.json).
  • Single-file studies (--script study.rs). A study no longer needs a
    crate: write one .rs file with cargo-script frontmatter (RFC 3502) for its
    deps and run it with mira --script study.rs. cargo -Zscript is nightly-only,
    so the host shims it onto stable — it parses the frontmatter, materializes a
    content-hashed throwaway crate (re-anchoring relative path deps, adding a
    [[bin]] and an isolating [workspace]), and cargo runs it with a shared
    target dir. The file format matches native cargo-script, so the same study runs
    under cargo -Zscript unchanged; MIRA_SCRIPT_NATIVE=1 opts into that today.
    Most bundled examples (examples/<name>.rs) are now single-file; multi-file /
    heavy-dep ones (cli_subject, metrics, matrix, llmsim) stay crates.
  • mira doctor. One-command diagnosis of a Mira setup, in three layers:
    mira.toml (parse errors, unknown/misspelled keys with a "did you mean"
    suggestion, launcher mistakes, presets and timeouts that can't work), the
    study's advertised listing (duplicate sample ids / target labels / axis
    values that collide case keys, empty datasets and matrices, unavailable
    targets, presets and [targets.LABEL] sections that match nothing), and the
    saved-run store (interrupted runs, torn temp files, invalid case results,
    missing reports). mira doctor --fix applies the safe repairs: removing
    leftover *.tmp files and re-rendering a finished run's missing
    report.json/report.html from its stored results. Warnings never fail;
    errors exit non-zero, so doctor can gate CI.
  • Publish runs to everruns. A new mira-publish-everruns crate plus
    mira publish <run_id> and mira run --publish everruns send a saved run's
    results to an everruns instance, which hosts and
    visualizes eval results it did not execute. Credentials reuse the everruns
    CLI: --everruns-* flags, then EVERRUNS_API_KEY/EVERRUNS_API_URL/
    EVERRUNS_ORG_ID, then ~/.config/everruns/credentials.json — so a prior
    everruns login is enough. One Mira run becomes one everruns run group (one
    EvalRun per eval, idempotent on the run id); everruns trusts Mira's verdict
    and does not re-grade.

Fixed

  • mira-everruns reported zero tool calls. RuntimeSubject filled
    Transcript.tool_calls via Mira's generic summarize_events, which only
    recognizes { name, input } tool objects — a shape the everruns event stream
    never emits (tool calls arrive as tool.completed events keyed by
    data.tool_name). Every tool-selection scorer silently saw zero calls while
    the tools actually ran. The adapter now extracts tool names from the
    tool.completed events it owns, with a regression test pinning that event
    shape against schema drift (EVE-676).
  • The repository's own mira.toml placed default_launcher below the
    [environment] section header, which nests it inside that table — so the
    setting was silently ignored (found by mira doctor). It now sits at the
    top level.