Skip to content

v3.1.0 — LLM backend abstraction

Choose a tag to compare

@rajivpant rajivpant released this 25 Apr 14:25
· 147 commits to main since this release

Highlights

  • LLM backend abstraction. Every LLM call now routes through a swappable backend interface (src/ragbot/llm/). Two backends ship:

    • litellm (default) — wraps litellm.completion(). Best provider/model coverage, handles long-tail provider quirks. Pinned >=1.83.0 (post-March-2026 supply-chain incident range).
    • direct — opt-in. Calls anthropic, openai, and google-genai SDKs directly. Smaller dependency surface, no third-party gateway.

    Selection: RAGBOT_LLM_BACKEND={litellm|direct}. Adding a new backend (Bifrost, Portkey, OpenRouter, etc.) is one new file plus one selection arm.

  • Web UI controls for reasoning effort (auto, off, minimal, low, medium, high) and the cross-workspace skills auto-include toggle.

  • /api/chat accepts thinking_effort and additional_workspaces — the same surface as the CLI flags.

  • Provider-quirk handling lives in backends. Backends absorb GPT-5.x max_completion_tokens rename, Claude 4.7+ thinking.type.adaptive shape, and the Anthropic-thinking-requires-temperature=1 constraint. The chat code path stays clean.

Bug fixes (caught during manual testing)

  • Chat CLI workspace lookup ignored ~/.synthesis/console.yaml discovery; now honours overrides only and falls through to the full discovery chain.
  • get_relevant_context budget loop used break instead of continue, blocking smaller results when an oversized whole-file script chunk topped the rankings.
  • Anthropic + thinking now forces temperature=1.0 automatically.
  • Claude 4.7+ uses the new thinking.type.adaptive shape instead of LiteLLM's older enabled shape (which 4.7+ rejects).
  • transformers>=5.6.0 pinned to track tokenizers 0.22.x compatibility (older transformers crashed sentence_transformers import).

API additions

  • ChatRequest.thinking_effort: 'auto'|'off'|'minimal'|'low'|'medium'|'high'
  • ChatRequest.additional_workspaces: string[] (empty array opts out of skills auto-include; omitted lets the server auto-include)

Tests

  • 26 new tests for the LLM backend abstraction (contract, selection, kwargs builder, direct healthcheck).
  • 6 regression tests covering the bug fixes above.
  • Final pass: 284 passed, 5 skipped.

Strategic note

LiteLLM remains a defensible default for ragbot in April 2026, primarily because of its provider/model coverage breadth. The March-2026 supply-chain incident (1.82.7–1.82.8) is bounded by the >=1.83.0 pin, but the API-compatibility lag for Claude 4.7+ shows the recurring tax of multi-provider gateways. The abstraction layer in this release is the response: keep LiteLLM as the default, make swap-in alternatives a configuration change rather than a code rewrite.

Setup

See README.md, CONFIGURE.md, and README-DOCKER.md.