Supporting Local Models: Integration & Architecture

[amirhouieh]

We should support local LLMs in Unbody. It’s a common need, and several users have already brought it up.

I'm thinking of two possible paths for integration:

1. Integrate with local model runners like Ollama or LM Studio
   These let users run models like LLaMA or Mistral locally, exposing a unified interface without Unbody handling deployments.

2. Support models directly inside Unbody
   This would mean integrating specific model runtimes (e.g., GGUF, Transformers, vLLM). More work, but more control.

Besides integration, we also need to think about structure:

• Should models be implemented as plugins?
• Should vectorizers and generators be handled separately?
• How do we allow users to register their own model wrappers?

The end goal is simple: bring-your-own-model support, whether local or hosted.

Curious to hear your thoughts or experiences with this.

[amirhouieh]

#2