Unsloth quants?

[IngwiePhoenix]

I like how the Unsloth quants are focused on required memory in use - it really helps a good amount. Their UD quants are also quite well documented. But they are not uncensored as they stem directly from the upstream model.

Can I apply Heretic to those as well? That would give me the best of both worlds :)

My target model is Qwen 3.8 Flash Next.

[IlyaKhalafi]

If you mean the UD-* files in unsloth/Qwen3.8-Flash-Next-GGUF, Heretic cannot work directly on those GGUF weights. Its bnb_4bit option uses bitsandbytes NF4, which is a different format.

The pipeline would be to start from a supported Transformers checkpoint, run Heretic, merge/export the result, then quantize that modified checkpoint to GGUF. That produces a new quantization and does not automatically preserve the prebuilt UD file's calibration or tensor quantization choices.

With bnb_4bit, merged export reloads the base model on CPU, so account for system RAM as well as VRAM. You can also export the LoRA adapter and merge it later.

I checked the loading/export code, but haven't tested Flash Next's architecture in Heretic.