Skip to content

Feature Request: Add new model support Hunyuan-A13B #14433

Closed
@yeahdongcn

Description

@yeahdongcn

Prerequisites

  • I am running the latest code. Mention the version if possible as well.
  • I carefully followed the README.md.
  • I searched using keywords relevant to my issue to make sure that I am creating a new issue that is not already open (or closed).
  • I reviewed the Discussions, and have a new and useful enhancement to share.

Feature Description

I was trying to convert https://huggingface.co/tencent/Hunyuan-A13B-Instruct to GGUF using gguf-my-repo and it failed with the following error message:

Error converting to fp16: INFO:hf-to-gguf:Loading model: Hunyuan-A13B-Instruct
WARNING:hf-to-gguf:Failed to load model config from downloads/tmpum6omczf/Hunyuan-A13B-Instruct: The repository downloads/tmpum6omczf/Hunyuan-A13B-Instruct contains custom code which must be executed to correctly load the model. You can inspect the repository content at /home/user/app/downloads/tmpum6omczf/Hunyuan-A13B-Instruct .
 You can inspect the repository content at https://hf.co/downloads/tmpum6omczf/Hunyuan-A13B-Instruct.
Please pass the argument `trust_remote_code=True` to allow custom code to be run.
WARNING:hf-to-gguf:Trying to load config.json instead
INFO:hf-to-gguf:Model architecture: HunYuanMoEV1ForCausalLM
ERROR:hf-to-gguf:Model HunYuanMoEV1ForCausalLM is not supported

So I'm wondering whether this new model could be supported in both llama.cpp and gguf-my-repo.

Motivation

The newly introduced Hunyuan-A13B model features a total of 80 billion parameters with 13 billion active parameters. I'd like to convert it to GGUF and serve it in llama.cpp.

Possible Implementation

No response

Metadata

Metadata

Assignees

No one assigned

    Labels

    enhancementNew feature or request

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions