Skip to content

v0.4.1 — GGUF judging under one pinned binary, schema v2, over-VRAM validation

Choose a tag to compare

View all tags
@Sahil170595 Sahil170595 released this 25 Jul 00:56
v0.4.1
d9af3b1
This commit was created on GitHub.com and signed with GitHub’s verified signature.
GPG key ID: B5690EEEBB952194
Verified
Learn about vigilant mode.

quantfit 0.4.1 — ROADMAP milestone 0.4b: the hardware-gated half of 0.4. Both gates passed on real hardware (RTX 4080 Laptop, 12 GB) before this release was cut.

GGUF judging — the paired diff on the format third-party quants actually ship in

verify-safety now accepts GGUF pairs (local *.gguf or hf:<org>/<repo>/<file>.gguf). Both arms generate through llama-server from the SHA256-verified pinned llama.cpp b9817 release archive on CPU — same binary, same device, same threads; only the weights differ, so the diff isolates the quantization. The F16 baseline runs in RAM, which removes the baseline VRAM cap for the 7–8B stratum where third-party quants live.

quantfit verify-safety \
  --baseline hf:bartowski/Qwen2.5-7B-Instruct-GGUF/Qwen2.5-7B-Instruct-f16.gguf \
  --quant    hf:bartowski/Qwen2.5-7B-Instruct-GGUF/Qwen2.5-7B-Instruct-Q4_K_M.gguf

Mandates are enforced in code, not documented:

  • the baseline must be unquantized (F16/BF16/F32) — resolved from the file's own general.file_type metadata, never the filename
  • both files must declare the same architecture
  • a transformers-baseline vs GGUF-quant mix is refused: that diff measures engine + quantization at once (a deployment delta) and is never pooled with a quantization diff

Drift report schema v2 (breaking)

Each arm now records engine provenance — transformers version, or the SHA256 of the llama.cpp binary actually executed plus source, thread count, and device — and artifact_sha256 for single-file artifacts. The same-binary mandate is auditable from the report alone: the two arms' binary_sha256 must be equal. resolved_dtype widens to precision-actually-loaded (a torch dtype, or a GGUF file type like "F16"/"Q4_K_M"). Schema-v1 reports are refused on parse with a clear message; no v1 reference reports were ever published.

Gate evidence

Gate 1 — end-to-end paired diff on a real third-party pair (bartowski/Qwen2.5-7B-Instruct-GGUF, Q4_K_M vs f16, identical binary, F16 arm 15.24 GB in CPU RAM): over-refusal drift 2/14 at-risk pairs flipped (14.3%, 95% CI 4.0–39.9%) while the scalar refusal count is unchanged (14 → 14) — offsetting flips a flat refusal counter would call clean. Dangerous axis 0/12 (95% CI upper 24.2%). The drift vector was byte-identical on immediate rerun.

Gate 2 — over-VRAM quantize through default sequential onloading: Qwen2.5-7B GPTQ (15.2 GB bf16 on a 12,282 MiB card) — GPU peak 9,047 MiB, process RSS peak 28.1 GB (5 s telemetry), ~32 min end-to-end, verify PASS on the artifact.

Evidence-derived guidance now in the README: at over-VRAM sizes use gptq — AWQ's 20-point grid search is transfer-bound under onloading (~2 h observed for a single 7B layer, projecting 50+ h; AWQ remains fine at in-VRAM sizes).

Tests

143 pass (22 new, all hermetic: crafted tiny GGUFs, stub HTTP server, fake processes). CI green on Python 3.10–3.14 plus clean-venv install smoke on Ubuntu and Windows.

Full changelog: https://github.com/Sahil170595/quantfit/blob/main/CHANGELOG.md

Assets 2
Loading