Skip to content

tracta v0.2.0

Choose a tag to compare

View all tags
@NakliTechie NakliTechie released this 30 Oct 03:18
· 7 commits to main since this release
v0.2.0
2246999

First public release.

Added

  • tracta, a command-line check of whether a model provider still serves the model it promised: baseline, check and status, with a Markdown and JSON report per check.
  • Static binaries for macOS (arm64, amd64), Linux (amd64, arm64) and Windows (amd64), built by scripts/build-binaries.sh with SHA-256 sums.
  • Providers: any OpenAI-compatible /chat/completions endpoint, and the Claude Messages API through Anthropic's Go SDK.
  • Decision rule: an exact paired sign test of today's run against a second baseline, anchored on the first, with no assumption that every question is equally noisy.
  • False-alarm control over repeated checks: check k spends α·6/(π²k²) of a total budget α (0.05 by default); checks without a verdict spend nothing.
  • 436 knowledge-boundary questions per run; the multiplications and the order come from a private per-watch seed.
  • The Python reference version (python3 -m tracta), and golden tests that require the Go binary to reproduce its verdicts, counts and p-values.
  • Experiments behind the method (experiments/drift-calibration/): ten published model families from 0.35B to 4B, a second inference engine, pinned model files (models.json), and a test on the Claude API.

Known limits

  • Against two strong models that both know the answers, the check has little margin: a Claude Haiku 5.5 → Sonnet 5.5 swap was detected through only 10 of 436 answers.
  • Only the macOS arm64 binary has been run end to end; the other platform builds are untested.

Verify a download: shasum -a 256 -c SHA256SUMS --ignore-missing

Explainer and method: see the README and experiments/drift-calibration/.

Assets 8
Loading