Skip to content

0.1a11

Pre-release
Pre-release

Choose a tag to compare

@simonw simonw released this 14 Nov 23:59

OpenAI-compatible endpoints can now be configured directly, with no extra LLM plugin. #12

plugins:
  datasette-llm:
    default_model: lmstudio-qwen
    endpoints:
      - model: lmstudio-qwen
        model_name: qwen/qwen3.8-27b
        api_base: http://localhost:1234/v1
        responses: true
      - model: hosted-assistant
        model_name: organization/assistant
        api_base: https://models.example.com/v1
        key: openrouter
        options:
          temperature: 0.2
  • A model reference can now be a dictionary with an api_base, pointing at any OpenAI-compatible Chat Completions or Responses endpoint. Endpoint models support tools and streaming by default.
  • New endpoints setting, global or per purpose, registers endpoint models without turning on the models allowlist. Endpoint dictionaries also work in default_model, a purpose's model, and models lists.
  • llm.model() and llm.group() accept a model-reference dictionary as well as a model ID.
  • key names the credential for a model. It is looked up in datasette-secrets first, then in LLM's stored keys under the same name, so key: openrouter uses the key saved with llm keys set openrouter. An endpoint without a key is treated as unauthenticated and never receives a provider's default key.
  • An endpoint's model is its local ID and must not match an installed model ID or alias. Use model_name for the ID the remote server expects.
  • Aliases now work in models and blocked_models.
  • Model lookup is faster: each call takes one snapshot of the installed models, so the work no longer grows with installed models multiplied by configuration entries.
  • A models, endpoints, blocked_models or purposes key left empty in YAML is now treated as absent. An empty purpose-level models key previously raised a TypeError.
  • Put credentials in key, not headers. Extra headers are visible in Datasette's /-/config.json.

Compatibility changes

  • Allowlists, blocklists and llm_filter_models hooks now apply to model() and group(), including explicit model IDs and configured defaults. Selecting an excluded model raises ModelNotFoundError. Previously they only affected models().
  • model() and group() now raise llm.NeedsKeyException if the selected model's key is missing and require_keys is on. Previously the error surfaced when the first prompt ran.
  • An explicit key on a model no longer falls back to the provider's default credential. If that name is missing from both datasette-secrets and LLM's stored keys, selecting the model raises NeedsKeyException. Remove key to restore normal provider key resolution.
  • Model-reference dictionaries now reject unknown fields, and key: null or options: null, with datasette_llm.ModelConfigurationError. Omit optional fields instead. An invalid reference under a purpose only affects lookups for that purpose.
  • A purpose's model allowlist now also includes the global default_model.
  • Unknown model IDs raise datasette_llm.ModelNotFoundError instead of llm.UnknownModelError.
  • Requires LLM 0.37.1 or newer.