Closed
Description
Prerequisites
- I am running the latest code. Mention the version if possible as well.
- I carefully followed the README.md.
- I searched using keywords relevant to my issue to make sure that I am creating a new issue that is not already open (or closed).
- I reviewed the Discussions, and have a new and useful enhancement to share.
Feature Description
I was trying to convert https://huggingface.co/tencent/Hunyuan-A13B-Instruct to GGUF
using gguf-my-repo
and it failed with the following error message:
Error converting to fp16: INFO:hf-to-gguf:Loading model: Hunyuan-A13B-Instruct
WARNING:hf-to-gguf:Failed to load model config from downloads/tmpum6omczf/Hunyuan-A13B-Instruct: The repository downloads/tmpum6omczf/Hunyuan-A13B-Instruct contains custom code which must be executed to correctly load the model. You can inspect the repository content at /home/user/app/downloads/tmpum6omczf/Hunyuan-A13B-Instruct .
You can inspect the repository content at https://hf.co/downloads/tmpum6omczf/Hunyuan-A13B-Instruct.
Please pass the argument `trust_remote_code=True` to allow custom code to be run.
WARNING:hf-to-gguf:Trying to load config.json instead
INFO:hf-to-gguf:Model architecture: HunYuanMoEV1ForCausalLM
ERROR:hf-to-gguf:Model HunYuanMoEV1ForCausalLM is not supported
So I'm wondering whether this new model could be supported in both llama.cpp
and gguf-my-repo
.
Motivation
The newly introduced Hunyuan-A13B model features a total of 80 billion parameters with 13 billion active parameters. I'd like to convert it to GGUF
and serve it in llama.cpp
.
Possible Implementation
No response