feat: rag api integration support (web search)

[tjbck]

Extension of #464

Additional information about web content pipeline, I think we can integrate search api for RAG. Like serpapi(free plan is 100 times search/month), bing search API (free plan is 1000 transactions /month),Wikipedia api or other.

[zinwelzl]

Hi.

Can you give example how to use it.

I try with
count words from #http://example.com/
but doesn't work for me

[tjbck]

@zinwelzl

You should start the prompt using the '#' command followed by the website url like such:

Make sure to click on the button to include the website as a document.

[ChingWeiChan]

@tjbck
I think we can make the url block clickable (hypertext) or catch the website title then can check the url whether we expected or not.

[tjbck]

@ChingWeiChan Great idea, I'll make it clickable!

Just merged #616, should work as expected!

[adan89lion]

Hi,

I tried some news articles with RAG, but it seems like models I tried are unable to process the "correct" content.

For instance, given the example article from CNN, the model is unable to extract answers from the article.

And if I want the model to introduce the document, it responded with content that are irrelevant to the article (website headers or SEO data I presume).

I think an integration with Mozilla's Readability library or similar projects can vastly improve the efficiency of website RAG support for open-webui.

[tbendien]

DuckDuckGo search API is free: https://pypi.org/project/duckduckgo-search/

[dillfrescott]

Can't wait for this! Super cool stuff!

[d416]

Huggingface chat is one of the best implementations of web search I’ve seen for UX so it’d be great if open-webui used this flow: https://huggingface.co/chat/

HuggingChat also open sourced and is OpenAPI compatible so it can be used with Ollama. The web search can be configured to use different search engines including searxng (private open source search engine)
https://github.com/huggingface/chat-ui?tab=readme-ov-file#web-search-config

Edit: chat-ui has Ollama support built in (it’s in the readme)

[strikeoncmputrz]

LLM_Web_search is a similar capability implemented in Text Generation WebUI that uses Langchain for RAG. It's worked very well for me and with a good system or character prompt I don't even need to ask it to search the web. https://github.com/mamei16/LLM_Web_search. I made a minor bug fix but otherwise am not affiliated.

[dillfrescott]

I think if open webui were to use a search engine, I nominate searxng (running locally) 100%.

I'm using it with another similar project and it works flawlessly and its totally free too, unlike a lot of these API's.

[sammcj]

Seconding the recommendation of searxng, it's really very good, self-hostable and works with many backend search engine providers.

[9cento]

+1
Can't wait for a web search feature, game changer!

[MohamedAliRashad]

Any updates ?

[knd775]

If you don't have anything to add, please don't comment like this. It just makes it harder for other people to follow.

@9cento
Reactions exist for a reason, use them instead

@MohamedAliRashad
Do you see any updates?

[ProjectMoon]

Idea for how to implement this: make a checkbox/toggle in the message prompt to enable "use web search for enhanced response accuracy". If this toggle is enabled, first send a hidden message to the LLM asking it to analyze the user's message for terms to search for. Then make a search request for each of these terms using some supported search engine, and send the top results to the normal web RAG pipeline.

This would require the admin to configure some search API endpoint. Starting with SearxNG would probably be good because it has an easily configurable JSON API.

[9cento]

Reactions exist for a reason, use them instead

That was a bump, frustrated mass-replier. And this is another bump.

[tjbck]

Implemented in dev.