Skip to content

v2.1.0 — LM Studio + cloud chat + MCP polish

Choose a tag to compare

@Basaltlabs-app Basaltlabs-app released this 20 Apr 10:57
· 5 commits to main since this release

Features

  • LM Studio provider (closes #2) — first-class support for LM Studio's OpenAI-compatible local server. gauntlet run --model lmstudio/<name>, gauntlet discover lists loaded models, host configurable via LMSTUDIO_HOST env / gauntlet config --lmstudio-host / default http://localhost:1234. Metadata inferred from model ID.
  • Cloud ChatClient wiring — openai/*, anthropic/*, and google/* model specs now run directly (previously NotImplementedError). Full frontier sweep typically under $5; Gemini has a free tier.
  • MCP server:
    • Self-driving tool instructions with explicit "do NOT shell out" directives so MCP clients (Claude Code, Gemini CLI, Cursor) run the suite without custom prompts.
    • Auto-detects the client app via MCP clientInfo.
    • New gauntlet_status(session_id) tool for resumability.

Fixes

  • Temporal Reasoning probe — the old "Reply with ONLY the name" framing caused some models to loop for minutes when the correct answer was "Neither". Prompt now lists 'Alice' | 'Bob' | 'Neither' explicitly.
  • Leaderboard provider mis-attribution — collect_fingerprint(..., "ollama") hardcode fixed; non-Ollama runs now attribute correctly.

Safety

  • Agent-invocation guard on gauntlet run — refuses to benchmark local models from non-interactive shells (MCP-client subprocess spawns) unless GAUNTLET_ALLOW_LOCAL=1. Prevents accidentally loading large local models and tanking the user's machine. Cloud providers + interactive humans unaffected.

Polish

  • Host resolution honors config file (env > file > default) for Ollama and LM Studio; persistent gauntlet config --*-host flags now actually take effect.
  • Error messages and auto-detect include LM Studio alongside Ollama.
  • README: new LM Studio and Cloud Baselines sections; updated provider filter tables.

Tests

  • 12 new LM Studio tests (host precedence, parsing, factory, metadata inference).

Install

```bash
pip install --upgrade gauntlet-cli
```

Full changelog: v2.0.3...v2.1.0