Skip to content

mtmd_init_from_file: error: Failed to load CLIP model to some multimodal model #17138

@lihaofd

Description

@lihaofd

Name and Version

ggml_cuda_init: found 1 CUDA devices:
Device 0: NVIDIA GeForce RTX 4060 Laptop GPU, compute capability 8.9, VMM: yes
version: 6907 (bea0452)
built with MSVC 19.44.35219.0 for x64

Operating systems

Windows

GGML backends

HIP, CUDA

Hardware

4060 or radeon 8060S

Models

No response

Problem description & steps to reproduce

llama-server -m obsidian-q6.gguf --mmproj mmproj-obsidian-f16.gguf
https://huggingface.co/NousResearch/Obsidian-3B-V0.5-GGUF

clip_init: failed to load model 'mmproj-obsidian-f16.gguf': load_hparams: unknown projector type:

mtmd_init_from_file: error: Failed to load CLIP model from mmproj-obsidian-f16.gguf

srv load_model: failed to load multimodal model, 'mmproj-obsidian-f16.gguf'
srv operator(): operator(): cleaning up before exit...
main: exiting due to model loading error

I have go through https://github.com/ggml-org/llama.cpp?tab=readme-ov-file#multimodal
Looks below 4 models failed with same issue
https://huggingface.co/advanced-stack/bakllava-mistral-v1-gguf/
https://huggingface.co/NousResearch/Obsidian-3B-V0.5-GGUF/tree/main
https://huggingface.co/nakodanei/ShareGPT4V-13B_GGUF/
https://huggingface.co/cmp-nct/Yi-VL-6B-GGUF

First Bad Commit

No response

Relevant log output

clip_init: failed to load model 'mmproj-obsidian-f16.gguf': load_hparams: unknown projector type:

mtmd_init_from_file: error: Failed to load CLIP model from mmproj-obsidian-f16.gguf

srv    load_model: failed to load multimodal model, 'mmproj-obsidian-f16.gguf'
srv    operator(): operator(): cleaning up before exit...
main: exiting due to model loading error

Metadata

Metadata

Assignees

No one assigned

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions