v0.5.0

Added

• Support for Python 3.15.
• evaltrack report --run-file, which reports the run in a run file. CI can then report a run with
  no repository, and link the report on a pull request without a remote. If a remote is
  configured, the report still compares the run with baseline. See
  CLI › report.

Documentation

• CI/CD has a setup without a remote, so a team
  can start with a report on each pull request and add a remote later. It also has complete GitHub
  Actions workflows for both setups, and for promote on merge.

v0.4.0

Added

• evaltrack report, which writes a recorded run as one HTML file with the run embedded. It shows
  the run as the dashboard does and opens anywhere with no server and no network, for CI artifacts
  and sharing results. The page also holds the run's history over the mainline and the baseline
  run, so Compare to mainline works in it. A value over 16 KB is left out of the page unless
  --full is passed, so a report of a large run stays small. See
  CLI › report.
• Report in the dashboard, on a run, which saves the same page for the run on screen, to share
  with someone without the dashboard.

Changed

• The dashboard's history leaves out a promoted run whose body does not parse, and logs it, instead
  of failing the whole history. A damaged baseline reflog still fails it.
• evaltrack.ui no longer re-exports create_app and MountedRepository. Import them from
  evaltrack.ui.app.
• Breaking: the mainline is the configured remote and nothing else. A dashboard with only a
  local repository mounted no longer measures reliability history over the local baseline.
  promote and push --ref baseline into the local repository now exit 2. To tell the local
  repository apart, both read pyproject.toml, so an invalid [tool.evaltrack] table now stops
  them with --repository too.

v0.3.0

Added

• The documentation site at evaltrack.jesravnbol.dk, versioned by
  minor release, with an API reference generated from the docstrings.
• RunRepository, RunSummary and ReflogEntry are exported from the package root. They are what
  open_repository(), list_runs() and get_ref() hand back, so a reader can now name every type
  on the repository read path.
• At the end of a pytest session, evaltrack reports each attempt whose stored output is larger than
  256 KB. The report names the test and the case, and says what to do. See
  Stored outputs › Large outputs.
• evaltrack.converters.register registers a converter, which changes how evaltrack stores the
  values of one type, for example to keep the tool calls of an agent run. See
  Stored outputs › Converters.
• The dashboard stays responsive on a large run. It sends each value over 16 KB to the browser as a
  preview with its size and a hash, and sends the whole value when a pane needs it. This applies to
  inputs, expected outputs, metadata and attempt outputs. Two runs still diff. A large value is
  compared by a hash of the same part that is compared for a small value. The server still reads the
  whole stored run. It keeps the last few runs that it read, so the panes of an open run do not read
  the run again.
• While a run loads, the dashboard shows how much of it has arrived. The run list shows the stored
  size of each run, which the summary sidecar now records.

Changed

• Breaking: evaltrack stores a dataclass without its private fields, the fields whose name
  starts with _. A library often keeps its internal state in such fields, for example the message
  history of an agent framework. When a task returned such an object, a run grew to hundreds of
  megabytes. See
  Stored outputs › Private fields.
• evaltrack writes a run as compact JSON, not indented. The saving is largest for deeply nested
  values. The run that prompted this change went from 221 MB to 79 MB. A run of mostly long text
  saves much less. evaltrack still reads indented runs, so the run schema version does not change.
  evaltrack export prints the compact form.

Removed

• Breaking: the keep_raw_results setting. Remove it from [tool.evaltrack], or evaltrack
  stops with an error that says so. A run no longer stores the eval runner's own result objects, and
  a recorded test has no raw_results field. These objects grew with every output, and a loaded run
  cannot give them back to the runner. If your eval runner records traces, use them for the full
  data. Runs recorded with raw results still load, and the download and export include them. The
  dashboard does not show them.

v0.2.0

Added

• Amazon S3 as a remote repository backend, addressed as s3://bucket[/prefix] and installed with
  the [s3] extra. Authentication uses boto3's default credential chain. See
  Repositories › Amazon S3.

v0.1.0

The first public release.

evaltrack is a pytest plugin that records, gates and tracks LLM evals. Evals are tests, so they run
in the suite and the CI pipeline you already have, with the eval runner you already use. Mark a test
with @pytest.mark.evaltrack and you get:

• A gate on assertions and scores. Your runner's assertions become pass/fail for the test. Score
  bars turn a recorded number into a gate too, so a case under the bar fails.
• A way to handle flaky evals. Rerun only the evals that failed instead of the whole pipeline,
  and track each case's pass rate over time.
• A record of every run, in a repository you own. Local files, or Azure Blob as the supported
  remote.
• A dashboard you run locally to inspect runs and compare them across pull requests and
  releases. Over the runs you have promoted it tracks each case's pass rate and each score, so
  changes in quality that never trip the gate are still visible.

pydantic-evals and DeepEval are supported
out of the box. Another runner needs a small translator.

pip install "evaltrack[ui,pydantic-evals]"

Python 3.11 to 3.14. See the README to get started and
docs/ for the rest.