quantfit v0.3.0
Reconcile and make the verdict honest (ROADMAP milestone 0.3). Supersedes PyPI 0.1.0 and the never-published 0.2.0.
Highlights
- Bounded verdict statistics for
verify-safety: Wilson 95% CIs per axis over at-risk pairs; zero-flip results print their CI upper bound and minimum detectable effect at 80% power.NO REGRESSION DETECTED (dangerous-axis MDE ~13pp at n=12)— a pass bounds the drift, it does not certify safety. - Safety drift vector replaces the "safety tax" naming (
SafetyTax→SafetyDrift): in the alignment literature a safety/alignment tax is capability paid FOR safety — the near-inverse of what this measures. Purge test-enforced on all shipped surfaces. - CI-contract exit codes: 0 = measured + no regression, 3 = regression detected, 4 = axis unmeasurable (an unmeasured run is not a pass), 2 = operational failure.
- Modern load path: accelerate
device_map="auto"branch and--offloadflag deleted; models load CPU-first and llm-compressor's default sequential onloading streams layers to the GPU. RAM now gates every capacity mode — refuse up front, never OOM mid-load. dtype=everywhere (transformers >=4.56); dropped unusedgptqmodeldep;llmcompressorcapped <0.13; version parity (__init__/pyproject) test-enforced; install-smoke CI job on Ubuntu + Windows.
Breaking
from quantfit.safety import SafetyTax→SafetyDrift;.clean→.regression_detected(inverted polarity)quantfit quantize --offloadremoved (behavior is automatic)verify-safetyregression exit code 2 → 3
Validation
Three consecutive verify-safety runs (Qwen2.5-1.5B vs its AWQ 4-bit artifact) byte-identical on an RTX 4080 Laptop, correctly flagging a reproducible 2/10 over-refusal drift (20.0%, 95% CI 5.7-51.0%) with the dangerous axis clean (0/12, CI upper 24.3%). 56 unit tests across Python 3.10-3.14; clean-venv wheel install verified on Ubuntu and Windows.
Full changelog: https://github.com/Sahil170595/quantfit/blob/main/CHANGELOG.md