Skip to content

quantfit v0.4.0

Choose a tag to compare

View all tags
@Sahil170595 Sahil170595 released this 17 Jul 01:42
v0.4.0
2d256d0
This commit was created on GitHub.com and signed with GitHub’s verified signature.
GPG key ID: B5690EEEBB952194
Verified
Learn about vigilant mode.

Reports become auditable artifacts (ROADMAP milestone 0.4a — the CI-gated half of 0.4; GGUF judging and over-VRAM validation are 0.4b and are not in this release).

Highlights

  • Drift report schema v1: verify-safety --report out.json emits an auditable JSON artifact — judge + probe-dataset revision pins, the pinned judge input contract, decode parameters, resolved per-arm dtypes (the literal "auto" is rejected by schema), an environment fingerprint (python/torch/transformers/CUDA/GPU), per-arm and judge runtimes, and the full drift vector with CIs and MDEs. DriftReport.from_json structurally validates: wrong schema, missing fields, and type-confused values are refused, never coerced.
  • Stats are the scipy numbers: Wilson intervals match scipy.stats.binomtest(...).proportion_ci(method="wilson") to 1e-9 and the MDE delivers its stated 80% power via scipy.stats.binom — cross-checked in CI on every push. Full-precision z (a 0/12 clean axis prints its bound as 24.2%).
  • Vocabulary: "baseline", not "fp16" — the live report proved arms load at their native dtype (bf16 for Qwen2.5). The CLI flag is now --baseline (--fp16 still works as a legacy alias); schema keys are baseline_refused/quant_refused.
  • Exit codes are a CI contract across the CLI: verdicts never collide with operational errors — verify-safety 0/3/4/2, check 0/3/2, verify 0/3/2, all documented in --help and unit-tested.
  • Public API matches what quantfit is: import quantfit lazily exposes verify_safety, SafetyDrift, DriftReport, quantize, and capacity_plan — and got lighter doing it (no more huggingface_hub at import). Stats helpers export from quantfit.safety.
  • Hermetic supply-chain tests for the GGUF binary pipeline: SHA256 pin/verify/delete-on-mismatch, refuse-before-download, atomic promote-after-verify, cached-archive re-verification.

Breaking (vs 0.3.0)

  • SafetyDrift fields: unsafe_fp16_refused/safe_fp16_refused → unsafe_baseline_refused/safe_baseline_refused; verify_safety's first param is baseline_model_id
  • check won't-fit and verify FAIL exit 3 (was 2)
  • quantfit.check_fit/FitReport removed (VRAM-only 0.1-era surface); fit.plan → capacity_plan

Validation

Live on an RTX 4080 Laptop: verify-safety --report reproduces the identical drift vector across every commit in this release (dangerous axis clean 0/12, CI upper 24.2%; over-refusal 2/10, 20.0%, CI 5.7–51.0% — a real, reproducible finding on a published third-party AWQ artifact), and the emitted report round-trips the hardened parser. 121 hermetic tests across Python 3.10–3.14; clean-venv wheel installs verified on Ubuntu and Windows.

Full changelog: https://github.com/Sahil170595/quantfit/blob/main/CHANGELOG.md

Assets 2
Loading