Repository navigation
AREX-2 (Qwen3.8-27B fine-tune) converted with 0.7.4: works well at 5/6/8-bit, plain 4-bit loops at temperature 0 #2456
Replies: 1 comment
|
The AWQ result has a coverage limitation worth separating from the four-bit quality result. In 0.7.4's implementation, AWQ scaling only visits decoder layers with The mixed recipes also do not specifically protect the linear-attention projections. They raise selected A useful next comparison would be keeping those linear-attention projections at six or eight bits while leaving the rest at four, then rerunning the same coding tasks with identical sampling settings and recording the actual model size. That tests a precision allocation your existing mixed runs did not isolate. The CLI exposes fixed recipes, so this needs a custom predicate or local patch. I have not tested that custom conversion. This does not establish that those projections cause the loops, or that the drop is expected for every model of this architecture. I have not reproduced the generation results. |
Uh oh!
There was an error while loading. Please reload this page.
Thanks for mlx-vlm. I used 0.7.4 to convert BAAI/AREX-2 and wanted to share what I saw, in case the 4-bit result is useful.
What worked
python -m mlx_vlm convert --hf-path BAAI/AREX-2 --mlx-path AREX-2-6bit -q --q-bits 6and the same at 5 and 8 bits. Vision was kept. Each build passed a tool-call loop, recall in a 32,000-token prompt and 11 image questions, and loads in LM Studio 0.4.8. They are on mlx-community as AREX-2-4bit, -5bit, -6bit and -8bit.The 4-bit
That is over 6,141 tokens of fixed text, text model only.
</think>on 4 of 18 coding tasks. With temperature 1.0, top-p 0.95, top-k 20 it did not.AWQ scaling applied: {'layers': 16, 'groups': 48}and peaked at 55 GB of memory, against 14 GB for the plain conversions.Is a drop this large from 5-bit to 4-bit expected for this architecture, or is there a recipe I should try? Scripts and raw results are in the
testsfolder of https://huggingface.co/mlx-community/AREX-2-5bitAll reactions