Repository navigation
v2.1.0 — LM Studio + cloud chat + MCP polish
Features
- LM Studio provider (closes #2) — first-class support for LM Studio's OpenAI-compatible local server.
gauntlet run --model lmstudio/<name>,gauntlet discoverlists loaded models, host configurable viaLMSTUDIO_HOSTenv /gauntlet config --lmstudio-host/ defaulthttp://localhost:1234. Metadata inferred from model ID. - Cloud ChatClient wiring —
openai/*,anthropic/*, andgoogle/*model specs now run directly (previouslyNotImplementedError). Full frontier sweep typically under $5; Gemini has a free tier. - MCP server:
- Self-driving tool instructions with explicit "do NOT shell out" directives so MCP clients (Claude Code, Gemini CLI, Cursor) run the suite without custom prompts.
- Auto-detects the client app via MCP
clientInfo. - New
gauntlet_status(session_id)tool for resumability.
Fixes
- Temporal Reasoning probe — the old "Reply with ONLY the name" framing caused some models to loop for minutes when the correct answer was "Neither". Prompt now lists
'Alice' | 'Bob' | 'Neither'explicitly. - Leaderboard provider mis-attribution —
collect_fingerprint(..., "ollama")hardcode fixed; non-Ollama runs now attribute correctly.
Safety
- Agent-invocation guard on
gauntlet run— refuses to benchmark local models from non-interactive shells (MCP-client subprocess spawns) unlessGAUNTLET_ALLOW_LOCAL=1. Prevents accidentally loading large local models and tanking the user's machine. Cloud providers + interactive humans unaffected.
Polish
- Host resolution honors config file (env > file > default) for Ollama and LM Studio; persistent
gauntlet config --*-hostflags now actually take effect. - Error messages and auto-detect include LM Studio alongside Ollama.
- README: new LM Studio and Cloud Baselines sections; updated provider filter tables.
Tests
- 12 new LM Studio tests (host precedence, parsing, factory, metadata inference).
Install
```bash
pip install --upgrade gauntlet-cli
```
Full changelog: v2.0.3...v2.1.0