Q3 ROADMAP

[robertgshaw2-neuralmagic]

SUMMARY:

• Avoid full pass through the model for quantization modifier
• Data free oneshot
• Runtime of GPTQ with large models – how to do a 70B model?
• Runtime of GPTQ with act order
• Remove “naive-quantized”
• Native integration of compressed-tensors into AutoModelForCausalLM (deprecate SparseAutoModel)
• run-compressed in AutoModelForCausalLM
• KV cache quantization
• Best practices documentation
• Improve runtime of GPTQ with multi-gpu sequential_update=True
• Support AWQ + other low bit methods
• Support MoE models end-to-end through vllm
• Support XXForConditionalLM and embedding models
  • Llava model quantization seems not be supported #73
• Support Vision Language models
  • Llava model quantization seems not be supported #73
  • MODEL REQUESTS #69 (comment))
• Documented examples of non-llama language models (gemma, phi, qwen, mixtral, deepseek, ... request others?)
• Documented sparse transfer learning with TRL
• Simplify Smoothquant for non-llama models
• Improve loading time of pack-quantized models in SparseAutoModel
• Understand why the BoS token has impact?
• Integration with PEFT
• 2:4 Sparsity

[halexan]

Looking forward to
Documented examples of non-llama language models (gemma, phi, qwen, mixtral, deepseek, ... request others?)

[CharlesRiggins]

This paper can be helpful for "understanding why the BoS token has an impact".

IntactKV: Improving Large Language Model Quantization by Keeping Pivot Tokens Intact

It is found that such huge outliers usually occur at the [BOS] token and some other uninformative initial tokens (e.g., "." or ",") at particular channels, regardless of the rest of the input sequence. We thus name these tokens pivot tokens given their dominating values in the activation. The attention scores will be concentrated on these pivot tokens than the rest ones, a.k.a attention sinks (Xiao et al., 2024).

[fengyang95]

May I ask when AWQ will be supported

[robertgshaw2-neuralmagic]

May I ask when AWQ will be supported

We are actively working on this now. Ideally in a week or so