Skip to content

v3.0: audit dataset, rebuild and calibration evidence, and the quantcheck screen

Choose a tag to compare

View all tags
@aditi-p31 aditi-p31 released this 22 Oct 03:39
· 7 commits to main since this release
v3.0
652d30d

Data, scripts and analysis behind the revised paper Broken on Arrival: Non-Functional Quantized LLM Artifacts in Public Model Registries and How to Catch Them (arXiv:2609.05881, version 2).

What changed from v1

  • The v1 defect diagnosis is corrected. The five official Ollama tags that v1 reported as defective conversions are exact, uncalibrated llama.cpp quantizations, tensor-identical to rebuilds from the original checkpoints. They are still non-functional, and adding an importance matrix rescues most of them.
  • Every reported score is a serial re-score (one EvalPlus evaluation at a time); see rerun-2026-10/serial/.

New in this release

  • rerun-2026-10/: rebuilds, calibration, per-tensor comparisons, perplexity, generation-settings and miss-audit runs, the dated run plan, scripts and logs.
  • rerun-2026-10/serial/numbers_serial.py reproduces the paper's numbers.

No model files are included. Code: MIT. Data: CC BY 4.0.

Assets 2
Loading