Repository navigation
vitals 0.3.0
New features
-
generate_structured()extracts structured data from model responses
viaellmer::parallel_chat_structured(), analogous to howgenerate()
wrapsparallel_chat()(#153). -
model_graded_qa()now encourages brevity in its defaultinstructions
(#197). This reduces the tendency of model-graded scorers to "talk themselves
out of" a reasonable score.
Log viewer
-
Updated the vendored Inspect Log Viewer to version 0.3.161 (#194).
-
Task IDs now follow Inspect's
task_identifierformat
(task_name/model/hash), including the model name and a hash of
solver/scorer arguments. This ensures evals with different models or
arguments appear as separate log viewer entries rather than being
collapsed as "retries." -
The home page now includes all of the metadata associated with the eval.
-
Model events in the log no longer hardcode
max_tokens = 4096. The
logged value now reflects the provider's actual setting, and the field
is omitted when unset (#213).
Bug fixes
- Accuracy calculation for ordered factor scores with more than two
levels (e.g.I < P < C) no longer inflates partial-credit scores
when the highest grade is absent from results.