Support for local/opensource Models

[Spider4U]

Dose it support private deployment with offline environment?

[yuhongsun96]

Could you clarify what you mean?

We've heard from a few people that support for self hosted Generative AI models could be useful, is that what you're thinking as well?

If you mean can you host Danswer on a private cloud, personal computer, or something to that effect, it's already capable of that.

In terms of outbound connections, Danswer reaches out to the connectors (like GitHub, Slack etc) and also to OpenAI currently for generating your answers.

[adarshxs]

Yes support for open source self hosted LLMs could be useful. Falcon 40B Or MPT 30B.
But the models have to be fine tuned specifically for this I guess.

[yuhongsun96]

We will be taking this up very soon, this is probably one of the most popular asks from the community. Thanks for creating an issue to track it

[definitiontv]

It's very easy to modify environment for most of the simple API calls used for openai to use localai instead. If you look at other projects implementation of that by looking at localai/examples. In addition I believe they now support embeddings as well. I haven't studied yr codebase yey..will have a go see if I can do this myself manually and report unless you already have a better direction

[Pipboyguy]

Its easy in principle yes, but my attempt #189 indicates that the frontend needs a minor tweak. Would appreciate a pointer

[jabdoa2]

I implemented a minimal local language model in #202. You can choose from any model in hugging face which support question answering. Most of the models I tried tend to give very short (but mostly correct) answers. Maybe I will extend this later to add some of the surrounding sentences in the answer.

For example on question such as "How to do X?" It will answer "Use Y" or even just "Y". Which is correctly derived from our documentation. However, in most cases that is explained in the documentation in the surrounding paragraph. For that reason it probably would make sense to post-process the answer a little bit if its very short. Maybe even just highlight the answer in the paragraph?

[yuhongsun96]

Many local/opensource models are now implemented:
gpt4all (quantized/lower-float-precision models running in Danswer containers)
local transformer models (smaller < 1B parameters models, running in Danswer containers)
opensource models hosted via HuggingFace
self-hosted models with a REST API (may require minor extension of code depending on the API contracts, default conforms to huggingface hosted endpoints), valid for any cloud provider or model hosting service

see: https://docs.danswer.dev/gen_ai_configs/overview