Add support for HuggingFace GGUF models in Ollama

[FeelTheFonk]

Add support for HuggingFace GGUF models in Ollama

Description

This PR adds support for running GGUF models directly from HuggingFace through Ollama using the hf.co/ prefix in model configurations. This enables users to leverage a wider range of models without requiring manual model installation.

Changes

• Updated OllamaInfo.is_model_available() to recognize HuggingFace GGUF model paths
• Added documentation and examples for using GGUF models
• Added test case for GGUF model path validation

Example Configuration

{
    "lmstudio-qwen2.5-7b-instruct-1m-gguf": {
        "comment": "This runs on your own computer via Ollama using GGUF models from HuggingFace.",
        "class": "Ollama",
        "arguments": {
            "model": "hf.co/lmstudio-community/Qwen2.5-7B-Instruct-1M-GGUF:Q6_K",
            "temperature": 0.5,
            "request_timeout": 120.0,
            "is_function_calling_model": false
        }
    }
}

[neoneye]

I'm curious to your setup. I imagine you have uploaded your SSH key to Hugging Face. And when you run the model, the inference happens on Hugging Face?

Can you talk on PlanExe Discord?

[FeelTheFonk]

Hey! To download models from Hugging Face, you simply need a valid HF_TOKEN set as an environment variable locally. There's no need to use huggingface_cli or any other pip library—model inference is handled locally via Ollama rather than on Hugging Face’s servers.

I'll try to join your discord server when I can, thank you for your attention!

[neoneye]

I can't make sense of the find_model.startswith("hf.co/"), why is that needed?

I have tried recreating your scenario, using hf.co and I can run models locally. I'm unable to runs models on HF?

I was unaware that ollama could fetch GGUFs directly from HF, thanks. I have updated the docs roughly describing how to fetch GGUF models. Let me know if the docs can be improved further.
https://github.com/neoneye/PlanExe/blob/main/extra/ollama.md

[FeelTheFonk]

The find_model.startswith("hf.co/") check is needed to distinguish between standard Ollama models and models that should be downloaded directly from HuggingFace. This prefix tells your system to fetch the GGUF model from HuggingFace instead of looking for it in the local Ollama repository.

I think there's a misunderstanding in your documentation. When using HuggingFace GGUF models with the hf.co/ prefix, you must specify a specific quantization version (like :Q4_K_M, :Q6_K, etc.) and cannot use :latest. The :latest syntax only works with standard Ollama models, not with HuggingFace GGUF models.

For example:

• Correct: hf.co/unsloth/Llama-3.1-Tulu-3-8B-GGUF:Q4_K_M (or any other versions/LLM)
• Incorrect: hf.co/mlabonne/Meta-Llama-3.1-8B-Instruct-abliterated-GGUF:latest

The confusion might be because your ollama list output shows a model with :latest, but this likely won't work properly when actually trying to run inference. For HuggingFace GGUF models, you need to specify the exact quantization version you want to use.

To clarify the core functionality: Ollama downloads the model file from HuggingFace and then runs inference locally on the user's machine, not on HuggingFace's servers.

[neoneye]

I would like to talk with you on Discord about your fix.

In particular why return True when it's a huggingface model?
Is it because you do inference on huggingfaces server?

I have updated the docs with your recommendations. Thank you.
https://github.com/neoneye/PlanExe/blob/main/extra/ollama.md