[Feature Request] Conversational Search (RAG) with a local LLM

[elliot-sawyer]

Description

Is it possible to use Conversational Search (RAG) with a local LLM? The documentation suggests it is only possible with OpenAI and Cloudflare. I was wondering if any of the HuggingFace models could be used with an available GPU instead to avoid making slow network calls.

Metadata

Typesense Version: 26

OS: Linux

[piccaso]

It takes only care of the R part of RAG but yes, custom models and using GPU are supported.
And you also have the option to generate the embeddings yourself and store them.

Check out all the subtopics of this part of the documentation:
https://typesense.org/docs/26.0/api/vector-search.html#index-embeddings

[jasonbosco]

@piccaso Typesense does support the "AG" part of RAG, by integrating with ChatGPT / Cloduflare APIs: https://typesense.org/docs/26.0/api/conversational-search-rag.html

@elliot-sawyer We don't yet have a way to integrate with local LLMs. But I'll leave this open as a feature request.

[jasonbosco]

May I know which local LLMs you're looking for?

[elliot-sawyer]

I don't have a particular one in mind yet - would any of the Typesense models on HuggingFace be appropriate? I'll have an NVIDIA A100 available in a couple of months to do some Typesense work with, but only on the stipulation that I use a locally downloaded LLM (no network or API keys).

[jasonbosco]

I misspoke earlier. Turns out that we actually added support for vLLM through which you can run several local LLMs. Just haven't documented it yet.

Will post a link here once we update the docs.

[Ku3mi41]

@jasonbosco It's nice to know that this is being done. I already started testing this myself, without documentation (heh) and ran into an authorization problem. Now api_key is not used by vLLM at all, could you add this? In case when LLM inference on different server it's is important to have authentication.

[jasonbosco]

CC: @ozanarmagan

[piccaso]

I misspoke earlier. Turns out that we actually added support for vLLM through which you can run several local LLMs. Just haven't documented it yet.

Will post a link here once we update the docs.

Thats huge. If you manage to make this easily accessible it could be quite the hype.
Looking forward to try it!