Skip to content

v0.3.0

Choose a tag to compare

@jesrav jesrav released this 26 Sep 13:24
· 8 commits to main since this release
cf2416f

Added

  • The documentation site at evaltrack.jesravnbol.dk, versioned by
    minor release, with an API reference generated from the docstrings.
  • RunRepository, RunSummary and ReflogEntry are exported from the package root. They are what
    open_repository(), list_runs() and get_ref() hand back, so a reader can now name every type
    on the repository read path.
  • At the end of a pytest session, evaltrack reports each attempt whose stored output is larger than
    256 KB. The report names the test and the case, and says what to do. See
    Stored outputs › Large outputs.
  • evaltrack.converters.register registers a converter, which changes how evaltrack stores the
    values of one type, for example to keep the tool calls of an agent run. See
    Stored outputs › Converters.
  • The dashboard stays responsive on a large run. It sends each value over 16 KB to the browser as a
    preview with its size and a hash, and sends the whole value when a pane needs it. This applies to
    inputs, expected outputs, metadata and attempt outputs. Two runs still diff. A large value is
    compared by a hash of the same part that is compared for a small value. The server still reads the
    whole stored run. It keeps the last few runs that it read, so the panes of an open run do not read
    the run again.
  • While a run loads, the dashboard shows how much of it has arrived. The run list shows the stored
    size of each run, which the summary sidecar now records.

Changed

  • Breaking: evaltrack stores a dataclass without its private fields, the fields whose name
    starts with _. A library often keeps its internal state in such fields, for example the message
    history of an agent framework. When a task returned such an object, a run grew to hundreds of
    megabytes. See
    Stored outputs › Private fields.
  • evaltrack writes a run as compact JSON, not indented. The saving is largest for deeply nested
    values. The run that prompted this change went from 221 MB to 79 MB. A run of mostly long text
    saves much less. evaltrack still reads indented runs, so the run schema version does not change.
    evaltrack export prints the compact form.

Removed

  • Breaking: the keep_raw_results setting. Remove it from [tool.evaltrack], or evaltrack
    stops with an error that says so. A run no longer stores the eval runner's own result objects, and
    a recorded test has no raw_results field. These objects grew with every output, and a loaded run
    cannot give them back to the runner. If your eval runner records traces, use them for the full
    data. Runs recorded with raw results still load, and the download and export include them. The
    dashboard does not show them.