Skip to content

vitals 0.3.0

Choose a tag to compare

@simonpcouch simonpcouch released this 15 May 20:18

New features

  • generate_structured() extracts structured data from model responses
    via ellmer::parallel_chat_structured(), analogous to how generate()
    wraps parallel_chat() (#153).

  • model_graded_qa() now encourages brevity in its default instructions
    (#197). This reduces the tendency of model-graded scorers to "talk themselves
    out of" a reasonable score.

Log viewer

  • Updated the vendored Inspect Log Viewer to version 0.3.161 (#194).

  • Task IDs now follow Inspect's task_identifier format
    (task_name/model/hash), including the model name and a hash of
    solver/scorer arguments. This ensures evals with different models or
    arguments appear as separate log viewer entries rather than being
    collapsed as "retries."

  • The home page now includes all of the metadata associated with the eval.

  • Model events in the log no longer hardcode max_tokens = 4096. The
    logged value now reflects the provider's actual setting, and the field
    is omitted when unset (#213).

Bug fixes

  • Accuracy calculation for ordered factor scores with more than two
    levels (e.g. I < P < C) no longer inflates partial-credit scores
    when the highest grade is absent from results.