Skip to content

v2.1.2 — community submission pipeline + MCP env loading + QOL

Latest
Latest

Choose a tag to compare

View all tags
@Basaltlabs-app Basaltlabs-app released this 04 May 08:00
v2.1.2
3c803c1
This commit was created on GitHub.com and signed with GitHub’s verified signature.
GPG key ID: B5690EEEBB952194
Verified
Learn about vigilant mode.

Why this release matters

Three independent silent-failure paths converged into "I ran benchmarks and nothing showed up." All three are fixed here, plus a substantial quality-of-life pass while we were in the area. First update in two weeks.

Reliability fixes

  • CLI submissions now include hardware_tier + attestation — previously every CLI row landed in Supabase with hardware_tier="", leaving tier-filtered dashboard views (Consumer-Mid, Consumer-High, Cloud) systematically empty.
  • MCP fingerprint is environment-aware. Real collect_fingerprint() on desktop MCP clients (Gemini CLI, Claude Desktop, Cursor); placeholder only on Vercel. Was throwing away real RAM/CPU/GPU data and submitting unknown/0.
  • submit_result rejection logs bumped DEBUG → WARNING with response body included. API rejections were silent before; now you actually see what the server didn't like.
  • CTRL-C-safe submission ordering. Payload is enqueued to ~/.gauntlet/pending/ before the network call; only HTTP 200 deletes the file. Mid-flight kill no longer loses the result.
  • Hardware fingerprint collected once per multi-model run (was rebuilt per model — pointless psutil / sysctl / nvidia-smi calls on every iteration).

New commands & flags

  • gauntlet --version / -V — long overdue.
  • gauntlet doctor — env-var detection, .env discovery, Supabase reachability, hardware preview, recent runs, retry queue. Drains the queue on its way through. Designed to answer "why didn't my last run show up?" in five seconds.
  • gauntlet history — Rich table of past local runs with --limit N and --model <substr> filtering.
  • --no-submit flag and GAUNTLET_PRIVATE=1 — opt out of community submission for iteration runs.
  • MCP startup banner prints Community push: ENABLED/DISABLED to stderr so misconfiguration surfaces upfront, not after a 20-minute run.

New plumbing

  • dotenv-style env loader at package import. Searches $GAUNTLET_ENV_FILE → .env.vercel.local/.env.local/.env walking up to repo root → ~/.gauntlet/.env. Allowlist of 10 keys; never overwrites parent-process env. Fixes the Gemini-CLI / Claude-Desktop subprocess env-blindness on macOS.
  • Persistent retry queue at ~/.gauntlet/pending/. Failed submissions queue to disk and replay on the next CLI invocation. Permanent 4xx rejections drop after one warning. Capped at 200 files. Self-healing without user intervention.
  • MCP write path now goes through the same validator as /api/submit. New gauntlet/mcp/submit_validator.py enforces score-range, model-name length, category sanity, score-consistency, probe-count, attestation shape, probe-details size caps, and dedup. Closes the gap where /mcp accepted unvalidated payloads.
  • /api/version endpoint returns {server, min_supported, recommended, latest, upgrade_command, upgrade_url, release_notes}.
  • GAUNTLET_API_URL env override for self-hosting / staging.
  • Auto-update notification on benchmark runs.

Performance

  • Read endpoint cache TTLs: max-age=30, s-maxage=60 → max-age=300, s-maxage=900, stale-while-revalidate=86400.
  • /api/predict + /api/recommend share a 60s in-memory matrix cache with stale-fallback. Was fetching 2000 rows per request; now once per minute. Serves stale on Supabase outage instead of 503.
  • Dashboard polling pauses when the tab is hidden, refetches on focus. Background tabs no longer churn Supabase + Vercel functions.

Code health

  • HMAC submit key deduplicated to gauntlet.core.config.get_submit_key(). gauntlet.basaltlabs.app URL also deduplicated; 5 hardcoded callsites consolidated.
  • submit_handler refactored to use the shared validator. Single source of truth across /api/submit and the MCP write path.
  • str(e) leakage plugged in three API handlers.
  • MIN_CLI_VERSION bumped 1.3.5 → 2.0.0. Anything below pre-dates the fingerprint + attestation fields and was being silently accepted with hardware_tier="".

Tests

  • 42 new tests covering env-loader behavior, save-status formatting, and every validator rule.
  • Total: 540 passing, 0 failures.

Docs

  • New README section: Running gauntlet as a local MCP server. Per-client config blocks for Gemini CLI / Claude Desktop / Cursor, the macOS Dock-launch gotcha, and the GAUNTLET_ENV_FILE override for CI.
  • Clarified MCP data-quality language: desktop MCP runs now produce real fingerprints; the "serverless" caveat applies only to the hosted endpoint.

Breaking change

MIN_CLI_VERSION is now 2.0.0. Submissions from CLI versions below 2.0.0 will be rejected with HTTP 400 and a clear "run pipx upgrade gauntlet-cli" message. If you've been pinned, this is the moment to upgrade.

Upgrade

pipx upgrade gauntlet-cli
gauntlet --version   # → gauntlet 2.1.2
gauntlet doctor      # confirm everything is wired

Deferred to a follow-up

  • Splitting gauntlet/dashboard/server.py (>1200 lines) into per-feature modules. TODO(refactor) block at the top of the file names the natural split points. Pure mechanical move — should land as its own focused PR.

22 files changed · +1,969 / −179 · 540 tests passing

Assets 2
Loading