HF Transformers Support

[Qubitium]

Hi MiniAi team! Are there plans underway to get HF Transformer support patched in via attrust_remote_code=True?

Currently trying to load this model via transformers for GPT-QModel quantization but we are running into a brick wall as we do not see HF Transformers documented anywhere for inference. Thanks!

[MiniMax-AI-Dev]

Thanks for asking! We are indeed planning to add support for HF Transformers, and we will consider your suggestion to support setting trust_remote_code=True

[Qubitium]

@MiniMax-AI-Dev I have converted the model and added prelim HF Transformer compat code here:

https://huggingface.co/ModelCloud/MiniMax-M2-BF16

Feel free to use the code for the official HF implementation if useful. The code is only made to validate the bf16 weights work and inference output produce sane output. The current code is modeled after sglang/vllm implemention minus all the fuse and optimizations.

In the HF repo there is a script to run the model under hf:

# you need at least 7x A100 80GB class gpu run this BFLOAT16 model in HF Transformers
python test_minimax_m2_hf.py

The code is actually copied from this working branch: ModelCloud/GPTQModel#2128

[rogeryoungh]

We now support transformers with trust_remote_code=True. Thank you for your feedback.