Skip to content

quantfit v0.3.0

Choose a tag to compare

@Sahil170595 Sahil170595 released this 11 Jul 21:05
a07c5cf

Reconcile and make the verdict honest (ROADMAP milestone 0.3). Supersedes PyPI 0.1.0 and the never-published 0.2.0.

Highlights

  • Bounded verdict statistics for verify-safety: Wilson 95% CIs per axis over at-risk pairs; zero-flip results print their CI upper bound and minimum detectable effect at 80% power. NO REGRESSION DETECTED (dangerous-axis MDE ~13pp at n=12) — a pass bounds the drift, it does not certify safety.
  • Safety drift vector replaces the "safety tax" naming (SafetyTaxSafetyDrift): in the alignment literature a safety/alignment tax is capability paid FOR safety — the near-inverse of what this measures. Purge test-enforced on all shipped surfaces.
  • CI-contract exit codes: 0 = measured + no regression, 3 = regression detected, 4 = axis unmeasurable (an unmeasured run is not a pass), 2 = operational failure.
  • Modern load path: accelerate device_map="auto" branch and --offload flag deleted; models load CPU-first and llm-compressor's default sequential onloading streams layers to the GPU. RAM now gates every capacity mode — refuse up front, never OOM mid-load.
  • dtype= everywhere (transformers >=4.56); dropped unused gptqmodel dep; llmcompressor capped <0.13; version parity (__init__/pyproject) test-enforced; install-smoke CI job on Ubuntu + Windows.

Breaking

  • from quantfit.safety import SafetyTaxSafetyDrift; .clean.regression_detected (inverted polarity)
  • quantfit quantize --offload removed (behavior is automatic)
  • verify-safety regression exit code 2 → 3

Validation

Three consecutive verify-safety runs (Qwen2.5-1.5B vs its AWQ 4-bit artifact) byte-identical on an RTX 4080 Laptop, correctly flagging a reproducible 2/10 over-refusal drift (20.0%, 95% CI 5.7-51.0%) with the dangerous axis clean (0/12, CI upper 24.3%). 56 unit tests across Python 3.10-3.14; clean-venv wheel install verified on Ubuntu and Windows.

Full changelog: https://github.com/Sahil170595/quantfit/blob/main/CHANGELOG.md