Skip to content

LLMDyno 0.5.0

Choose a tag to compare

@github-actions github-actions released this 22 Nov 06:39
· 7 commits to main since this release

There is no 0.4.0. Full library parity, the integrations and remote mode ship together in this release.

Added

  • The library can do everything the dashboard does. Every web API route has a library call (116 of them), and a
    test keeps it that way. New objects with their methods: Model, Provider, Harness, Benchmark, Suite, Run,
    Transcript, Settings and Job. Among the new calls:
    • models: search_models(), repo_info(), estimate() (fit before downloading), download_model(),
      import_models(), recommended_models(), Model.contexts(), .profile(), .pause(), .resume(), .delete();
    • profiles: Profile.duplicate(), .export(), import_profile(), profile_defaults(), and update() for the name,
      notes, tags and pricing;
    • providers: provider(), add_provider(), Provider.test(), .models(), keys, budget and spend, vendors(),
      lookup_price();
    • harnesses: harness(), Harness.build(), .dockerfile(), .preview(), runtimes, custom harnesses;
    • benchmarks: benchmark(), create_benchmark(), Benchmark.update(), .duplicate(), .export(), .validate(),
      import_benchmark(), suites, and the SWE-bench and polyglot setup calls;
    • runs: run(), Run.transcript() with turns(), tool_calls(), thinking() and usage(), .files(),
      .read_file(), .download(), .log(), .gpu(), .requests(), .retry(), .regrade(), .cancel();
    • plans: Plan.retry(), .rename(), .delete(), .get_runs();
    • results: every Results-page option (benchmarks=, pass_threshold=, quality_weights=, efficiency_ref=),
      cells(), rubric(), agreement(), reviews(), reference(), export(), save_weights(), the sample data;
    • judging: review() with a custom rubric, guidance and single runs, judge_defaults(), reviews(),
      delete_reviews();
    • Artificial Analysis, settings, Hugging Face token, specs, GPU, jobs, cleanup, panic and Docker repair.
  • Command line for everything: every library call is a llmdyno command, generated from the library
    (llmdyno list-models, llmdyno run transcript RUN_ID --view tool-calls, --json for scripts). A test keeps the
    two in step.
  • llmdyno.watch(plan or run or job) yields live events (run and plan changes, GPU samples, job progress and, with
    transcript=True, every agent event and model request). bench(), Plan.wait() and Plan.resume() take
    on_event=. Live events no longer need the web server to be running.
  • llmdyno.aio: every library function as a coroutine, plus aio.watch() and aio.call().
  • llmdyno.job(id) and the Job object (status, progress, log, wait(), cancel()).
  • Library objects expose their data: obj["field"], obj.get(), obj.to_dict() and obj.refresh().
  • The full error family is public: InvalidError, ForbiddenError, RiskNotAcknowledgedError, RateLimitError,
    UpstreamError and InsufficientStorageError join NotFoundError, ConflictError and PreflightError.
  • Integrations with mainstream libraries, each an optional extra (pip install "llmdyno[all]" for all of them;
    a missing one says what to install). See docs/INTEGRATIONS.md.
    • Data and notebooks: to_pandas(), to_polars() and to_parquet() for every table and transcripts; rich tables
      for Results, Plan and Run in Jupyter; Plan.live() widget; tqdm progress in bench().
    • Charts like the dashboard's: results.plot.leaderboard(), .heatmap(), .pareto(), .radar(), .tokens(),
      in matplotlib or plotly.
    • Tracking and tracing: log_to_mlflow(), log_to_wandb() (offline works), log_to_langfuse() and to_otel()
      (GenAI semantic conventions, real timestamps); bench(auto_log=...).
    • Model gateways: a LiteLLM vendor and litellm_model() (Azure OpenAI, Bedrock, Vertex AI, …, prices from
      LiteLLM's table), local_servers(), ollama_model() and openai_compatible(); the Hugging Face token from
      hf auth login is reused; Results.push_to_hub() and load_results_from_hub().
    • Your own agent: python_harness("my_agent:run", requirements=[...]) benchmarks a Python agent in the same
      sandbox as the built-in harnesses, with templates for LangGraph, the OpenAI Agents SDK, smolagents and PydanticAI.
    • Inspect AI: inspect_task(), import_inspect_samples(), Results.to_inspect_log().
    • CI: llmdyno ci --baseline ... --max-drop 2 and compare_results(), a pytest plugin
      (@pytest.mark.llmdyno(min_score=...)), and a GitHub Actions workflow in examples/ci.
  • Remote mode: llmdyno serve --host 0.0.0.0 asks other machines for an access token (made on first use, shown
    once, llmdyno token --rotate replaces it) and logs the changes they make; llmdyno.connect(url, token) sends every
    library call to that server, with live events over its WebSocket; --url / --api-token on every command; the web
    UI asks for the token when opened from another machine. Requests from the server's own machine need nothing, as
    before. See docs/REMOTE.md.
  • A documentation site (MkDocs) with guides and an API reference generated from the code, and example notebooks that
    CI runs.

Changed

  • Internals: the web API's logic moved into llmdyno/services (one module per area) and the library into
    llmdyno/sdk, both over the same services. The HTTP API is unchanged: every endpoint answers with the same status,
    headers and body as in 0.3.1.
  • cloud_model() saves the given or environment API key every time instead of comparing it with the stored one first
    (same result, and it works without reading the secret store).
  • Library calls raise the same errors the web API answers with (a missing benchmark is NotFoundError, a built-in one
    you try to edit is ForbiddenError, …) instead of Python's KeyError or PermissionError; the mapping lives in one
    place, llmdyno.errors.from_exception, used by both.
  • The custom vendor is no longer the only one that may run without an API key: vendors carry key_optional
    (custom and the new litellm).
  • The library is type-checked: pyright strict on llmdyno.sdk and llmdyno.errors (fully typed signatures),
    standard on the services, web API, CLI and integrations; CI runs it with every extra installed.

Fixed

  • Uploading a file that is not a zip as a benchmark answered with a server error; it now says what is wrong (400).
  • A dependency action posted without an action answered with a server error; it is now a 400 naming the problem.