Repository navigation
LLMDyno 0.5.0
There is no 0.4.0. Full library parity, the integrations and remote mode ship together in this release.
Added
- The library can do everything the dashboard does. Every web API route has a library call (116 of them), and a
test keeps it that way. New objects with their methods:Model,Provider,Harness,Benchmark,Suite,Run,
Transcript,SettingsandJob. Among the new calls:- models:
search_models(),repo_info(),estimate()(fit before downloading),download_model(),
import_models(),recommended_models(),Model.contexts(),.profile(),.pause(),.resume(),.delete(); - profiles:
Profile.duplicate(),.export(),import_profile(),profile_defaults(), andupdate()for the name,
notes, tags and pricing; - providers:
provider(),add_provider(),Provider.test(),.models(), keys, budget and spend,vendors(),
lookup_price(); - harnesses:
harness(),Harness.build(),.dockerfile(),.preview(), runtimes, custom harnesses; - benchmarks:
benchmark(),create_benchmark(),Benchmark.update(),.duplicate(),.export(),.validate(),
import_benchmark(), suites, and the SWE-bench and polyglot setup calls; - runs:
run(),Run.transcript()withturns(),tool_calls(),thinking()andusage(),.files(),
.read_file(),.download(),.log(),.gpu(),.requests(),.retry(),.regrade(),.cancel(); - plans:
Plan.retry(),.rename(),.delete(),.get_runs(); - results: every Results-page option (
benchmarks=,pass_threshold=,quality_weights=,efficiency_ref=),
cells(),rubric(),agreement(),reviews(),reference(),export(),save_weights(), the sample data; - judging:
review()with a custom rubric, guidance and single runs,judge_defaults(),reviews(),
delete_reviews(); - Artificial Analysis, settings, Hugging Face token, specs, GPU, jobs, cleanup, panic and Docker repair.
- models:
- Command line for everything: every library call is a
llmdynocommand, generated from the library
(llmdyno list-models,llmdyno run transcript RUN_ID --view tool-calls,--jsonfor scripts). A test keeps the
two in step. llmdyno.watch(plan or run or job)yields live events (run and plan changes, GPU samples, job progress and, with
transcript=True, every agent event and model request).bench(),Plan.wait()andPlan.resume()take
on_event=. Live events no longer need the web server to be running.llmdyno.aio: every library function as a coroutine, plusaio.watch()andaio.call().llmdyno.job(id)and theJobobject (status, progress, log,wait(),cancel()).- Library objects expose their data:
obj["field"],obj.get(),obj.to_dict()andobj.refresh(). - The full error family is public:
InvalidError,ForbiddenError,RiskNotAcknowledgedError,RateLimitError,
UpstreamErrorandInsufficientStorageErrorjoinNotFoundError,ConflictErrorandPreflightError. - Integrations with mainstream libraries, each an optional extra (
pip install "llmdyno[all]"for all of them;
a missing one says what to install). See docs/INTEGRATIONS.md.- Data and notebooks:
to_pandas(),to_polars()andto_parquet()for every table and transcripts; rich tables
forResults,PlanandRunin Jupyter;Plan.live()widget; tqdm progress inbench(). - Charts like the dashboard's:
results.plot.leaderboard(),.heatmap(),.pareto(),.radar(),.tokens(),
in matplotlib or plotly. - Tracking and tracing:
log_to_mlflow(),log_to_wandb()(offline works),log_to_langfuse()andto_otel()
(GenAI semantic conventions, real timestamps);bench(auto_log=...). - Model gateways: a LiteLLM vendor and
litellm_model()(Azure OpenAI, Bedrock, Vertex AI, …, prices from
LiteLLM's table),local_servers(),ollama_model()andopenai_compatible(); the Hugging Face token from
hf auth loginis reused;Results.push_to_hub()andload_results_from_hub(). - Your own agent:
python_harness("my_agent:run", requirements=[...])benchmarks a Python agent in the same
sandbox as the built-in harnesses, with templates for LangGraph, the OpenAI Agents SDK, smolagents and PydanticAI. - Inspect AI:
inspect_task(),import_inspect_samples(),Results.to_inspect_log(). - CI:
llmdyno ci --baseline ... --max-drop 2andcompare_results(), a pytest plugin
(@pytest.mark.llmdyno(min_score=...)), and a GitHub Actions workflow in examples/ci.
- Data and notebooks:
- Remote mode:
llmdyno serve --host 0.0.0.0asks other machines for an access token (made on first use, shown
once,llmdyno token --rotatereplaces it) and logs the changes they make;llmdyno.connect(url, token)sends every
library call to that server, with live events over its WebSocket;--url/--api-tokenon every command; the web
UI asks for the token when opened from another machine. Requests from the server's own machine need nothing, as
before. See docs/REMOTE.md. - A documentation site (MkDocs) with guides and an API reference generated from the code, and example notebooks that
CI runs.
Changed
- Internals: the web API's logic moved into
llmdyno/services(one module per area) and the library into
llmdyno/sdk, both over the same services. The HTTP API is unchanged: every endpoint answers with the same status,
headers and body as in 0.3.1. cloud_model()saves the given or environment API key every time instead of comparing it with the stored one first
(same result, and it works without reading the secret store).- Library calls raise the same errors the web API answers with (a missing benchmark is
NotFoundError, a built-in one
you try to edit isForbiddenError, …) instead of Python'sKeyErrororPermissionError; the mapping lives in one
place,llmdyno.errors.from_exception, used by both. - The
customvendor is no longer the only one that may run without an API key: vendors carrykey_optional
(customand the newlitellm). - The library is type-checked: pyright strict on
llmdyno.sdkandllmdyno.errors(fully typed signatures),
standard on the services, web API, CLI and integrations; CI runs it with every extra installed.
Fixed
- Uploading a file that is not a zip as a benchmark answered with a server error; it now says what is wrong (400).
- A dependency action posted without an
actionanswered with a server error; it is now a 400 naming the problem.