v0.5.0 — QSR spec v0, screen harness, model-card emit

quantfit 0.5.0 — the CI-verifiable half of ROADMAP milestone 0.5: the measurement protocol becomes a versioned spec, and the tooling to run it at scale ships with it.

What is NOT in this release, stated plainly: the existence-proof hunt runs, the sensitivity-control run, the replication package, outreach, and the 0.5 GO/NO-GO decision. Those run against this tooling; none of them has started.

QSR spec v0 (spec/qsr-v0.md)

The durable asset the roadmap is built around: Quantization Safety Regression, v0, as a versioned protocol document. Paired-diff protocol; engine rules including the same-binary GGUF mandate; schema-v2 provenance field by field; statistics (at-risk denominators, Wilson intervals with method and sidedness disclosure, minimum detectable effect, the exit-code CI contract); per-axis screen aggregation; resource-stated hardware caps; the determinism canary; sensitivity-control conditionality labeling; and versioning rules. Every numeric claim in it was verified by executing the shipped code, and the document states honestly which sections are code-enforced at which symbol versus normative on publication practice. The CLI is the spec's reference implementation, not the other way around.

quantfit screen --targets targets.json --out DIR

Runs the paired diff sequentially over a manifest of quants and writes one schema-v2 drift report per target plus screen-summary.json.

• Bounds are per-stratum AND per-axis. Each axis has its own at-risk denominator, so a dangerous-axis flip on a target whose over-refusal axis was unmeasurable still enters the dangerous-axis bound instead of vanishing from the headline number. Strata are never pooled; there is no screen-wide rate and no field to put one in.
• Flagged, not found. n_regressed counts judge-flagged flips; n_regressed_human_verified is reported separately, and every flagged row carries human_verified: null until a human reads the pair. The judge is uncalibrated, so an unverified flag is a candidate.
• Conditionality is machine-carried. The manifest takes a sensitivity_control block (pass / fail / unmeasurable / not_run; absent means not_run). Any status but pass stamps ROADMAP 0.5's literal label — "conditional on undemonstrated detection sensitivity" — into every bound's conditionality field.
• A screen survives its targets. Per-target operational failures — gated repos, missing files, mispaired architectures, network and disk errors: the same (RuntimeError, OSError) class the CLI maps to exit 2 — become rows, not screen deaths. Target names are collision-checked case-insensitively so a Windows or macOS filesystem cannot silently merge two reports. Unknown manifest keys are refused at both levels.
• Exit codes mirror verify-safety: 0 clean, 3 a flagged regression, 4 an axis where nothing was measured, 2 operational.

quantfit emit model-card --report drift.json

Renders any schema-v2 report as a paste-ready model-card section: the verdict verbatim, both axes with Wilson CIs and MDE (zero-flip rates withheld exactly as verify-safety prints them), the full provenance chain including the same-binary hash statement, the scale-cap line, and the exact serve command — vllm serve for a transformers artifact, llama-server -m for a GGUF one. Missing binary hashes read as unverifiable rather than mismatched; an engine quantfit did not write gets an explicit note rather than a silently absent section; a tampered-but-schema-valid report exits 2 cleanly instead of tracebacking out of a string formatter.

Screen target list (screens/targets-0.5.json + curation audit trail)

15 targets — 12 GGUF pairs across 9 model families and 4 quantizer orgs, plus 3 transformers pairs. Every filename, revision, and size was verified against the HF API twice by independent agents and spot-checked again by hand. The curation file records the corrections rather than hiding them, including one candidate removed because its advertised "BF16 baseline" is an upcast of FP8-quantized weights — a defect the GGUF file-type guard structurally cannot catch, so curation now checks the base model's quantization_config. The maintainer's own anchor quant is disclosed as self-produced, with a rule to quote its stratum's bound both with and without it.

Also

• docs/sensitivity-control-v0.md: the Q2_K gross-degradation surrogate control, with its decision rule keyed on the report's unmeasurable_axes rather than the process exit code (a regression on the other axis shadows exit 4), and reproduction verified from provenance equality so a human/judge label divergence reads as the finding rather than a reproduction failure.
• Verdict strings now name every unmeasurable axis: an over-refusal-degenerate run no longer prints a plain clean verdict next to exit 4.
• New quantfit[awq] extra (transformers' AwqQuantizer needs gptqmodel); quantfit.run_screen / ScreenError / model_card_fragment exposed as lazy root exports.
• Ruff is now upper-bounded (>=0.16,<0.17) in CI and the dev extra — an unpinned lint tool broke a green branch mid-cycle, and the repo's standing rule is that churning dependencies get a cap that moves only after a validated run.

Tests

208 pass, all hermetic (65 new): the screen's exit contract, the manifest refusal matrix, per-axis denominators, conditionality stamping, and every model-card hardening case. CI covers Python 3.10–3.14 plus a clean-venv wheel install smoke on Ubuntu and Windows.

Full changelog: https://github.com/Sahil170595/quantfit/blob/main/CHANGELOG.md

v0.4.1 — GGUF judging under one pinned binary, schema v2, over-VRAM validation

quantfit 0.4.1 — ROADMAP milestone 0.4b: the hardware-gated half of 0.4. Both gates passed on real hardware (RTX 4080 Laptop, 12 GB) before this release was cut.

GGUF judging — the paired diff on the format third-party quants actually ship in

verify-safety now accepts GGUF pairs (local *.gguf or hf:<org>/<repo>/<file>.gguf). Both arms generate through llama-server from the SHA256-verified pinned llama.cpp b9817 release archive on CPU — same binary, same device, same threads; only the weights differ, so the diff isolates the quantization. The F16 baseline runs in RAM, which removes the baseline VRAM cap for the 7–8B stratum where third-party quants live.

quantfit verify-safety \
  --baseline hf:bartowski/Qwen2.5-7B-Instruct-GGUF/Qwen2.5-7B-Instruct-f16.gguf \
  --quant    hf:bartowski/Qwen2.5-7B-Instruct-GGUF/Qwen2.5-7B-Instruct-Q4_K_M.gguf

Mandates are enforced in code, not documented:

• the baseline must be unquantized (F16/BF16/F32) — resolved from the file's own general.file_type metadata, never the filename
• both files must declare the same architecture
• a transformers-baseline vs GGUF-quant mix is refused: that diff measures engine + quantization at once (a deployment delta) and is never pooled with a quantization diff

Drift report schema v2 (breaking)

Each arm now records engine provenance — transformers version, or the SHA256 of the llama.cpp binary actually executed plus source, thread count, and device — and artifact_sha256 for single-file artifacts. The same-binary mandate is auditable from the report alone: the two arms' binary_sha256 must be equal. resolved_dtype widens to precision-actually-loaded (a torch dtype, or a GGUF file type like "F16"/"Q4_K_M"). Schema-v1 reports are refused on parse with a clear message; no v1 reference reports were ever published.

Gate evidence

Gate 1 — end-to-end paired diff on a real third-party pair (bartowski/Qwen2.5-7B-Instruct-GGUF, Q4_K_M vs f16, identical binary, F16 arm 15.24 GB in CPU RAM): over-refusal drift 2/14 at-risk pairs flipped (14.3%, 95% CI 4.0–39.9%) while the scalar refusal count is unchanged (14 → 14) — offsetting flips a flat refusal counter would call clean. Dangerous axis 0/12 (95% CI upper 24.2%). The drift vector was byte-identical on immediate rerun.

Gate 2 — over-VRAM quantize through default sequential onloading: Qwen2.5-7B GPTQ (15.2 GB bf16 on a 12,282 MiB card) — GPU peak 9,047 MiB, process RSS peak 28.1 GB (5 s telemetry), ~32 min end-to-end, verify PASS on the artifact.

Evidence-derived guidance now in the README: at over-VRAM sizes use gptq — AWQ's 20-point grid search is transfer-bound under onloading (~2 h observed for a single 7B layer, projecting 50+ h; AWQ remains fine at in-VRAM sizes).

Tests

143 pass (22 new, all hermetic: crafted tiny GGUFs, stub HTTP server, fake processes). CI green on Python 3.10–3.14 plus clean-venv install smoke on Ubuntu and Windows.

Full changelog: https://github.com/Sahil170595/quantfit/blob/main/CHANGELOG.md

quantfit v0.4.0

Reports become auditable artifacts (ROADMAP milestone 0.4a — the CI-gated half of 0.4; GGUF judging and over-VRAM validation are 0.4b and are not in this release).

Highlights

• Drift report schema v1: verify-safety --report out.json emits an auditable JSON artifact — judge + probe-dataset revision pins, the pinned judge input contract, decode parameters, resolved per-arm dtypes (the literal "auto" is rejected by schema), an environment fingerprint (python/torch/transformers/CUDA/GPU), per-arm and judge runtimes, and the full drift vector with CIs and MDEs. DriftReport.from_json structurally validates: wrong schema, missing fields, and type-confused values are refused, never coerced.
• Stats are the scipy numbers: Wilson intervals match scipy.stats.binomtest(...).proportion_ci(method="wilson") to 1e-9 and the MDE delivers its stated 80% power via scipy.stats.binom — cross-checked in CI on every push. Full-precision z (a 0/12 clean axis prints its bound as 24.2%).
• Vocabulary: "baseline", not "fp16" — the live report proved arms load at their native dtype (bf16 for Qwen2.5). The CLI flag is now --baseline (--fp16 still works as a legacy alias); schema keys are baseline_refused/quant_refused.
• Exit codes are a CI contract across the CLI: verdicts never collide with operational errors — verify-safety 0/3/4/2, check 0/3/2, verify 0/3/2, all documented in --help and unit-tested.
• Public API matches what quantfit is: import quantfit lazily exposes verify_safety, SafetyDrift, DriftReport, quantize, and capacity_plan — and got lighter doing it (no more huggingface_hub at import). Stats helpers export from quantfit.safety.
• Hermetic supply-chain tests for the GGUF binary pipeline: SHA256 pin/verify/delete-on-mismatch, refuse-before-download, atomic promote-after-verify, cached-archive re-verification.

Breaking (vs 0.3.0)

• SafetyDrift fields: unsafe_fp16_refused/safe_fp16_refused → unsafe_baseline_refused/safe_baseline_refused; verify_safety's first param is baseline_model_id
• check won't-fit and verify FAIL exit 3 (was 2)
• quantfit.check_fit/FitReport removed (VRAM-only 0.1-era surface); fit.plan → capacity_plan

Validation

Live on an RTX 4080 Laptop: verify-safety --report reproduces the identical drift vector across every commit in this release (dangerous axis clean 0/12, CI upper 24.2%; over-refusal 2/10, 20.0%, CI 5.7–51.0% — a real, reproducible finding on a published third-party AWQ artifact), and the emitted report round-trips the hardened parser. 121 hermetic tests across Python 3.10–3.14; clean-venv wheel installs verified on Ubuntu and Windows.

Full changelog: https://github.com/Sahil170595/quantfit/blob/main/CHANGELOG.md

quantfit v0.3.0

Reconcile and make the verdict honest (ROADMAP milestone 0.3). Supersedes PyPI 0.1.0 and the never-published 0.2.0.

Highlights

• Bounded verdict statistics for verify-safety: Wilson 95% CIs per axis over at-risk pairs; zero-flip results print their CI upper bound and minimum detectable effect at 80% power. NO REGRESSION DETECTED (dangerous-axis MDE ~13pp at n=12) — a pass bounds the drift, it does not certify safety.
• Safety drift vector replaces the "safety tax" naming (SafetyTax → SafetyDrift): in the alignment literature a safety/alignment tax is capability paid FOR safety — the near-inverse of what this measures. Purge test-enforced on all shipped surfaces.
• CI-contract exit codes: 0 = measured + no regression, 3 = regression detected, 4 = axis unmeasurable (an unmeasured run is not a pass), 2 = operational failure.
• Modern load path: accelerate device_map="auto" branch and --offload flag deleted; models load CPU-first and llm-compressor's default sequential onloading streams layers to the GPU. RAM now gates every capacity mode — refuse up front, never OOM mid-load.
• dtype= everywhere (transformers >=4.56); dropped unused gptqmodel dep; llmcompressor capped <0.13; version parity (__init__/pyproject) test-enforced; install-smoke CI job on Ubuntu + Windows.

Breaking

• from quantfit.safety import SafetyTax → SafetyDrift; .clean → .regression_detected (inverted polarity)
• quantfit quantize --offload removed (behavior is automatic)
• verify-safety regression exit code 2 → 3

Validation

Three consecutive verify-safety runs (Qwen2.5-1.5B vs its AWQ 4-bit artifact) byte-identical on an RTX 4080 Laptop, correctly flagging a reproducible 2/10 over-refusal drift (20.0%, 95% CI 5.7-51.0%) with the dangerous axis clean (0/12, CI upper 24.3%). 56 unit tests across Python 3.10-3.14; clean-venv wheel install verified on Ubuntu and Windows.

Full changelog: https://github.com/Sahil170595/quantfit/blob/main/CHANGELOG.md

quantfit v0.1.0

GPU-aware quantization CLI with a built-in safety-tax check.

• verify-safety — does the quantized model still refuse what the fp16 baseline refused? A two-axis tax (refusal-robustness vs over-refusal), per-zone, local judge, curated public probes. Validated on qwen2.5-1.5b + llama-3.2-1b AWQ (0 harmful-compliance regressions on both).
• Quantize — awq / gptq / smoothquant / fp8 / rtn (compressed-tensors) + GGUF, one frozen packed calibration.
• GPU-aware 3-tier capacity: in-GPU / CPU-offload / refuse.
• Commands: check / list / quantize / verify / verify-safety. Tested on Python 3.10–3.13.