v0.10.0

xin3he released this 12 Feb 06:25

1f02a08

Highlights

Code Refactor
- [STEP 1] refactor output formats (#1159) by @n1ck-guo
- [STEP 2] refactor format and export (#1192) by @n1ck-guo
- refactor eval and add UT (#1324) by @xin3he
- refactor auto_round api, using local symbol table to pass all args (#1358) by @n1ck-guo
- Extensible Module Weight Type Conversion Framework. (#1375) by @xin3he
- UT refactor for preview (#1160) by @xin3he
- Refactor module replacement (#1153) by @yiliu30
GGUF format add support for MoE models with non-linear expert layers. (#1244) by @n1ck-guo
Support general MOE replacement for MOE models (Transformers 5.0 compatible) (#1374) by @lvliang-intel
Delay materializing the replaced model weights until quantization (#1307) by @yiliu30
support llm dynamic wint8aint8 export (#1376) by @mengniwang95
Add Chinese README (#1335) by @ZaneMark
Add G2-specific FP8_STATIC support (#1148) by @yiliu30

Bugfixes and improvements

[LLMC] Wrap block for DDP (#1407) by @yiliu30
support GPTQ_FORMAT for "gptqmodel:exllamav2" backend (#1434) by @xin3he
Implement MXFP4Handler MXFP8Handler NVFP4Handler (#1385) by @xin3he
Support quant for meituan-longcat/LongCat-Flash-Lite (#1388) by @Kaihui-intel
fix CUDA CI (#1431) by @xin3he
Update BackendInfos for AutoGPTQ based on transformer version and save g_idx for gptqmodel backend (#1429) by @xin3he
Fix cuda model ut [glm4][Molmo] (#1428) by @Kaihui-intel
remove require_intel_extension_for_pytorch and fix TypeError: unhashable type: 'set' (#1425) by @xin3he
fix bug in PR-1244 (#1422) by @xin3he
fix: update cleanup logic for quantized model path in tests (#1419) by @xin3he
fix test_evaluation and AutoTuner new issue (#1417) by @xin3he
Update torch to 2.9.1 in CI (#1313) by @XuehaoSun
adapt vllm_ext to new extra config (#1409) by @mengniwang95
update auto-round-kernel package name to auto-round-lib (#1410) by @chensuyue
[fix]AttributeError: 'Autotuner' object has no attribute '_cache_lock' (#1414) by @xin3he
refine qwen3_vl_moe experts forward (#1413) by @WeiweiZhang1
fix Qwen3-VL model auto_awq export, add auto_awq vllm ut (#1384) by @WeiweiZhang1
better warning for not support mllm while quant_non_text_module (#1411) by @n1ck-guo
fix: update model imports from AutoModelForVision2Seq to AutoModelFor.. (#1412) by @xin3he
Enable xpu test for CI (#1352) by @chensuyue
fix cuda ut fail: gptqmodel and gguf format with fp8 input model (#1399) by @n1ck-guo
align scale for MLA kv (#1395) by @yiliu30
Fix version (#1393) by @XuehaoSun
fix layer config (#1373) by @wenhuach21
[Regression] Detach scale tensor to prevent holding computation graph references (#1389) by @xin3he
fix cuda ut fail (#1370) by @n1ck-guo
Disable replace FP8Expert (#1379) by @yiliu30
fix --device_map cuda and xpu issue (#1383) by @wenhuach21
rm duplicate args of the quantization extra config (#1334) by @WeiweiZhang1
fix exporting fp8 static for format auto_round (#1381) by @n1ck-guo
Standardize comment language in utils.py (#1378) by @shantoislamdev
Add Links; Add step_by_step_CN; Fix several grammatical issues (#1366) by @ZaneMark
Fix FP8 MLA (#1343) by @yiliu30
feat: handle per-tensor FP8 dequantization for Devstral models (#1356) by @SwekeR-463
fix copy python file issue (#1372) by @wenhuach21
add support for w4a16_mixed (#1326) by @n1ck-guo
fix bug of exporting fp8 static (#1361) by @n1ck-guo
Refactor FP8 dequantization and detection using registry pattern (#1348) by @scopophobic
Autoround in vLLM Office Hours (#1322) by @yiliu30
Fix cpu ut for transformers v5 (#1333) by @Kaihui-intel
Fix CPU CI (#1344) by @XuehaoSun
add copilot-instructions (#1341) by @xin3he
Fix issue where predefined ignore layers incorrectly apply to GGUF (#1340) by @wenhuach21
Fix attribute bug for diffusion model (#1338) by @mengniwang95
update release package and bump version (#1327) by @chensuyue
format add check for scheme args (#1332) by @n1ck-guo
(feat): add support for g2 fp8 on cpu with LUT (#1254) by @SwekeR-463
fix tied weights keys bug (#1330) by @wenhuach21
Update publication dates in README and add recipe configuration detai.. (#1329) by @xin3he
[GGUF] using quant_nontext_module to control whether quant vision model (#1317) by @n1ck-guo
simplify pull_request_template.md (#1328) by @xin3he
add blockwise way in pure rtn to ease module replacements (#1320) by @wenhuach21
set predefined ignore layers (#1318) by @wenhuach21
fix low_cpu and set it to True as default (#1288) by @wenhuach21
Add issue templates and PR template with required fields (#1309) by @Copilot
fix XPU CI hang issue (#1261) by @xin3he
fix bug of format (#1287) by @n1ck-guo
Enable q_scale calibration for deepseek v2 on Gaudi (#1299) by @mengniwang95
open source delta loss (#1300) by @wenhuach21
fix ignore layers regression (#1302) by @wenhuach21
Remove itrex format (#1301) by @Kaihui-intel
refine moe modellings to release orginal expert module's ram (#1265) by @WeiweiZhang1
Update unit test script (#1290) by @XuehaoSun
Update version (#1282) by @XuehaoSun
auto-round-kernel installation method (#1221) by @chensuyue
[vllm-ext] allows setting flashinfer workspace (#1259) by @yiliu30
add permission for workflow (#1190) by @chensuyue
Update HPU CI to gaudi v1.23.0 (#1271) by @XuehaoSun
gguf fix bug of tensors not on same tensor and gguf:q2_k_mixed (#1263) by @n1ck-guo
Fix critic bug causing GGUF to run on CPU (#1260) by @wenhuach21
update supported scheme readme (#1257) by @wenhuach21
Update nightly release workflow (#1256) by @chensuyue
Add nightly release workflow (#1255) by @chensuyue
fix cuda ut fail (#1237) by @n1ck-guo
Fix low cpu speed issue (#1251) by @wenhuach21
WNA16 does not apply optimized RTN for moe layers by default (#1245) by @wenhuach21
Organize test directory structure with logical categorization (#1243) by @Copilot
Update setting of disable opt rtn (#1249) by @WeiweiZhang1
set disable_opt_rtn to optional bool and change default value to None (#1231) by @WeiweiZhang1
[HPU]Export lazy mode env explicit (#1239) by @yiliu30
[API Change]rename fp_layers to ignore_layers (#1233) by @wenhuach21
Add FP8 dtype info to quant config (#1228) by @yiliu30
gguf add support for Mistral-Magistral-Devstral and Granite (#1226) by @n1ck-guo
Add CodeQL analysis workflow configuration (#1223) by @XuehaoSun
open source alg_ext (#1219) by @wenhuach21
align input scale for moe experts for fp8 (#1216) by @SwekeR-463
[pre-commit.ci] pre-commit autoupdate (#1222) by @pre-commit-ci[bot]
fix torch dtype (#1217) by @wenhuach21
Extend ark CI test trigger scope (#1218) by @chensuyue
Update version (#1215) by @XuehaoSun
fix calibration device issue in some scenarios (#1213) by @wenhuach21
update diffusion args (#1210) by @mengniwang95
fix scale dtype (#1207) by @wenhuach21
Fix llama4 kv quant (#1205) by @mengniwang95
fix backend issue (#1209) by @wenhuach21
remove ark support for cpu (#1206) by @wenhuach21
fix bug of new version gptqmodel and fix few cuda ut (#1204) by @n1ck-guo
Revert "set disable_opt_rtn as default for int8 RTN" (#1203) by @wenhuach21
set disable_opt_rtn as default for int8 RTN (#1201) by @WeiweiZhang1
Add FP8 KV Support for DS (#1180) by @yiliu30
add online quant & offline sgl api generation cuda UT (#1198) by @WeiweiZhang1
fix llama4 kv quant (#1199) by @mengniwang95
[Important Change] change mxfp4 to OCP definition (#1200) by @wenhuach21
[Important Change]change mxfp8 to OCP standard (#1183) by @wenhuach21
rewrite fill_default_value logic for robust (#1165) by @xin3he
fix llama4 ut (#1197) by @WeiweiZhang1
Update version (#1191) by @XuehaoSun
fix cuda ut fail (#1189) by @n1ck-guo
Enable qwen3 vl moe quant and load (#1182) by @WeiweiZhang1
fix scale_dtype is None issue (#1185) by @xin3he
fix device mismatch issue for unifying qkv scale (#1175) by @xin3he
Enable ark CI model test (#1177) by @chensuyue
Fix code for llmc llama4 quantization (#1161) by @mengniwang95
Fix inference issue in ARK for AWQ format (#1170) by @luoyu-intel
Deprecate awq infer ut (#1171) by @WeiweiZhang1
Fix accuracy bug in the customized dataset (#1162) by @wenhuach21
handle key not found issue (#1156) by @xin3he
update norm bias tuning doc (#1155) by @wenhuach21
fix asym quantization issue (#1158) by @wenhuach21
Fix cuda test and llmc test (#1152) by @XuehaoSun
add more blog in README (#1147) by @xin3he
update cd workflow (#1151) by @chensuyue
fix bug and support shared_layers in cli (#1138) by @xin3he
add new posts and align the format (#1125) by @xin3he

New Contributors

@Zhenzhong1 made their first contribution in #1075
@SwekeR-463 made their first contribution in #1216
@ZaneMark made their first contribution in #1335
@scopophobic made their first contribution in #1348
@shantoislamdev made their first contribution in #1378
@lvliang-intel made their first contribution in #1374

Full Changelog: `v0.9.7...v0.10.0`

Contributors

chensuyue, pre-commit-ci, and 15 other contributors

Assets 2