You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
Add the openrouter LLM provider, reaching the OpenRouter <https://openrouter.ai/>_ gateway over its OpenAI-compatible endpoint. Only PROVIDER_OPENROUTER_KEY is required; the URL defaults to https://openrouter.ai/api/v1. Adds the openrouter_provider_order and openrouter_reasoning_effort hyperparameters in the experiment init templates.
Add a spending cap to experiment run: OpenRouter call costs are read from usage.cost, logged to a new cost_logs table and counted against PROVIDER_OPENROUTER_MAX_SPEND (default 5.00 USD). Reaching it pauses the run gracefully, leaving the remaining rows pending. The tally is per run and resets when the command exits.
Edit the litellm example profile of the experiment init template: deepseek-ai/DeepSeek-V4-Flash-0731 replaces minimax-m2.7:229b.
Add a total_costs metric to experiment stats: the sum of cost_logs.cost_usd, shown as USD.
0.4.6 (2026-09-21)
Breaking: add data_id and prompt_id columns to every experiment_<provider> / try_experiment_<provider> table, written from mapping.csv by generate, update and try.
An older database without the two columns is refused on open: re-run experiment generate, or add and fill the columns in place manually using SQL (check migrations.md for details on the manual migration)
Breaking:analysis add-agreement now scores Cohen's kappa per prompt and pairs profiles rather than provider/model/profile triples.
Add intermediate_values_dir to pairwise_cohen_kappa, writing the rows behind each kappa to one CSV per prompt
Edit experiment try that no longer needs a generated database: it warns that the try will not be stored, then runs and saves the response file as usual. A --file naming a missing database still aborts.
Add --suggest to experiment try: prints one randomly chosen combination of data.csv, prompts/ and llms-for-experiment.csv as a ready command, and runs nothing.