Skip to content

v2.0.2: Skip server-error probes from scoring

Choose a tag to compare

@Basaltlabs-app Basaltlabs-app released this 17 Apr 15:41
· 10 commits to main since this release

Fix: Ollama 500 errors no longer tank your scores

Server errors (HTTP 500, 502, 503, connection failures) from Ollama or llama.cpp were being scored as behavioral failures (0.0), unfairly penalizing models for infrastructure issues. On memory-constrained machines (M1 8GB), multi-turn probes frequently trigger Ollama OOM kills, dragging scores down even though the model itself is performing correctly.

Now: Server-error probes are marked as "Skipped (server error)" and excluded from the module score entirely. They still appear in the results display (so you can see what happened) but they don't affect the score. If 5/8 temporal probes crash and 3/8 run successfully, the score is based on the 3 that ran.

Upgrade

pip install --upgrade gauntlet-cli