Skip to content

0.4.7

Choose a tag to compare

@github-actions github-actions released this 24 Sep 13:58
· 13 commits to main since this release
  • Add the openrouter LLM provider, reaching the OpenRouter <https://openrouter.ai/>_ gateway over its OpenAI-compatible endpoint. Only PROVIDER_OPENROUTER_KEY is required; the URL defaults to https://openrouter.ai/api/v1. Adds the openrouter_provider_order and openrouter_reasoning_effort hyperparameters in the experiment init templates.
  • Add a spending cap to experiment run: OpenRouter call costs are read from usage.cost, logged to a new cost_logs table and counted against PROVIDER_OPENROUTER_MAX_SPEND (default 5.00 USD). Reaching it pauses the run gracefully, leaving the remaining rows pending. The tally is per run and resets when the command exits.
  • Edit the litellm example profile of the experiment init template: deepseek-ai/DeepSeek-V4-Flash-0731 replaces minimax-m2.7:229b.
  • Add a total_costs metric to experiment stats: the sum of cost_logs.cost_usd, shown as USD.
    0.4.6 (2026-09-21)

  • Breaking: add data_id and prompt_id columns to every experiment_<provider> / try_experiment_<provider> table, written from mapping.csv by generate, update and try.
  • An older database without the two columns is refused on open: re-run experiment generate, or add and fill the columns in place manually using SQL (check migrations.md for details on the manual migration)
  • Breaking: analysis add-agreement now scores Cohen's kappa per prompt and pairs profiles rather than provider/model/profile triples.
  • Add intermediate_values_dir to pairwise_cohen_kappa, writing the rows behind each kappa to one CSV per prompt
  • Edit experiment try that no longer needs a generated database: it warns that the try will not be stored, then runs and saves the response file as usual. A --file naming a missing database still aborts.
  • Add --suggest to experiment try: prints one randomly chosen combination of data.csv, prompts/ and llms-for-experiment.csv as a ready command, and runs nothing.