feat: Add Language and country (and optionally other) parameters to DuckDuckGo websearch

[cociweb]

Check Existing Issues

• I have searched the existing issues and discussions.

Problem Description

Hello,
Currently, I try to add DDG as a Web search engine, but in my country, it is useless. When I try to create a search query, it parses the first X results. Unfortunately, these results are irrelevant in my country and language, too. (Additionally, I'm afraid these are commercial advertisements..)

Desired Solution you'd like

Since customizing the search query is intended for individual use, I think we can add a simple input text field and use its value after the ?q=< query >.
This option should be user level setting to fulfill the restrictions of the service.
The user should be responsible for this set of parameters. By default, it should be empty.
https://duckduckgo.com/duckduckgo-help-pages/settings/params.

Maybe, we should check the length of it to avoid URL cutting in case of a long query string?

Alternatives Considered

No response

Additional Context

No response

[rgaricano]

DDGS, now, is a metasearch engine, you have some info about DDGS in this PR: #16694

You can find info of DDGS params in: https://github.com/deedy5/ddgs

[micchickenburger]

DDGS, now, is a metasearch engine, you have some info about DDGS in this PR: #16694

You can find info of DDGS params in: https://github.com/deedy5/ddgs

Hi @rgaricano, how can these parameters be specified? It seems "Concurrent Requests" is the only parameter available in settings for DDGS.

[rgaricano]

With actual open-webui implementation of the ddgs lib isn't possible, the only implemented params are:

query (str): The query to search for
count (int): The number of results to return
backend (str): The search backend to use (auto, duckduckgo, google, brave, etc.)

(and only text searches)

but there are some ddgs websearch tools that also implement this lib, with more options, you can search for its in the openwebui.com site.

[rgaricano]

yes, you have reason, there isn't query search config for DDGS, sorry I was thinking in searxng.

It need some modifications and config params.

For reference:
Changes necessaries:

• Add Configuration Schema Fields to webClass (routers/retrieval.py):

# DDGS specific configurations
DDGS_REGION: Optional[str] = None
DDGS_SAFESEARCH: Optional[str] = None
DDGS_TIMELIMIT: Optional[str] = None
DDGS_LANGUAGE: Optional[str] = None
DDGS_PAGE: Optional[int] = None
DDGS_BACKEND: Optional[str] = None

• Add its to config.py:

# DDGS Configuration
DDGS_REGION = PersistentConfig(
    "DDGS_REGION",
    "rag.web.search.ddgs_region",
    os.getenv("DDGS_REGION", "us-en"),
)

DDGS_SAFESEARCH = PersistentConfig(
    "DDGS_SAFESEARCH",
    "rag.web.search.ddgs_safesearch",
    os.getenv("DDGS_SAFESEARCH", "moderate"),
)

DDGS_TIMELIMIT = PersistentConfig(
    "DDGS_TIMELIMIT",
    "rag.web.search.ddgs_timelimit",
    os.getenv("DDGS_TIMELIMIT", ""),
)

DDGS_LANGUAGE = PersistentConfig(
    "DDGS_LANGUAGE",
    "rag.web.search.ddgs_language",
    os.getenv("DDGS_LANGUAGE", ""),
)

DDGS_PAGE = PersistentConfig(
    "DDGS_PAGE",
    "rag.web.search.ddgs_page",
    int(os.getenv("DDGS_PAGE", "1")),
)

DDGS_BACKEND = PersistentConfig(
    "DDGS_BACKEND",
    "rag.web.search.ddgs_backend",
    os.getenv("DDGS_BACKEND", "auto"),
)

• Update states (in main.py)

# Add after existing web search configurations
app.state.config.DDGS_REGION = DDGS_REGION
app.state.config.DDGS_SAFESEARCH = DDGS_SAFESEARCH
app.state.config.DDGS_TIMELIMIT = DDGS_TIMELIMIT
app.state.config.DDGS_LANGUAGE = DDGS_LANGUAGE
app.state.config.DDGS_PAGE = DDGS_PAGE
app.state.config.DDGS_BACKEND = DDGS_BACKEND

• Update endpoints (in routers/retrieval.py):
  (in web configuration secton)

"DDGS_REGION": request.app.state.config.DDGS_REGION,
"DDGS_SAFESEARCH": request.app.state.config.DDGS_SAFESEARCH,
"DDGS_TIMELIMIT": request.app.state.config.DDGS_TIMELIMIT,
"DDGS_LANGUAGE": request.app.state.config.DDGS_LANGUAGE,
"DDGS_PAGE": request.app.state.config.DDGS_PAGE,
"DDGS_BACKEND": request.app.state.config.DDGS_BACKEND,

(& in response section)

request.app.state.config.DDGS_REGION = form_data.web.DDGS_REGION
request.app.state.config.DDGS_SAFESEARCH = form_data.web.DDGS_SAFESEARCH
request.app.state.config.DDGS_TIMELIMIT = form_data.web.DDGS_TIMELIMIT
request.app.state.config.DDGS_LANGUAGE = form_data.web.DDGS_LANGUAGE
request.app.state.config.DDGS_PAGE = form_data.web.DDGS_PAGE
request.app.state.config.DDGS_BACKEND = form_data.web.DDGS_BACKEND

• Modify DDGS function (retrieval/web/duckduckgo.py

def search_duckduckgo(
    query: str,
    count: int,
    filter_list: Optional[list[str]] = None,
    concurrent_requests: Optional[int] = None,
    region: str = "us-en",
    safesearch: str = "moderate",
    timelimit: Optional[str] = None,
    language: Optional[str] = None,
    page: int = 1,
    backend: str = "auto",
) -> list[SearchResult]:
    """
    Search using DuckDuckGo's Search API and return the results as a list of SearchResult objects.
    """
    search_results = []
    with DDGS() as ddgs:
        if concurrent_requests:
            ddgs.threads = concurrent_requests

        # Build parameters dict
        search_params = {
            "keywords": query,
            "region": region,
            "safesearch": safesearch,
            "max_results": count,
            "backend": backend,
        }
    
        if timelimit:
            search_params["timelimit"] = timelimit
        if language:
            search_params["language"] = language
        if page > 1:
            search_params["page"] = page

        try:
            search_results = ddgs.text(**search_params)
        except RatelimitException as e:
            log.error(f"RatelimitException: {e}")

    # Rest of function remains the same...

• Update websearch function (routers/retrieval.py)

elif engine == "duckduckgo":
    return search_duckduckgo(
        query,
        request.app.state.config.WEB_SEARCH_RESULT_COUNT,
        request.app.state.config.WEB_SEARCH_DOMAIN_FILTER_LIST,
        concurrent_requests=request.app.state.config.WEB_SEARCH_CONCURRENT_REQUESTS,
        region=request.app.state.config.DDGS_REGION or "us-en",
        safesearch=request.app.state.config.DDGS_SAFESEARCH or "moderate",
        timelimit=request.app.state.config.DDGS_TIMELIMIT,
        language=request.app.state.config.DDGS_LANGUAGE,
        page=request.app.state.config.DDGS_PAGE or 1,
        backend=request.app.state.config.DDGS_BACKEND or "auto",
    )

• Update Frontend websearch Settings (in src/lib/components/admin/Settings/WebSearch.svelte)

{:else if webConfig.WEB_SEARCH_ENGINE === 'ddgs' || webConfig.WEB_SEARCH_ENGINE === 'duckduckgo'}
    <div class="w-full mb-2.5">
        <div class=" self-center text-xs font-medium mb-1">
            {$i18n.t('Concurrent Requests')}
        </div>
        <input
            class="w-full rounded-lg py-2 px-4 text-sm bg-gray-50 dark:text-gray-300 dark:bg-gray-850 outline-hidden"
            placeholder={$i18n.t('Concurrent Requests')}
            bind:value={webConfig.WEB_SEARCH_CONCURRENT_REQUESTS}
            required
        />
    </div>

    <div class="w-full mb-2.5">
        <div class=" self-center text-xs font-medium mb-1">
            {$i18n.t('Region')}
        </div>
        <select
            class="w-full rounded-lg py-2 px-4 text-sm bg-gray-50 dark:text-gray-300 dark:bg-gray-850 outline-hidden"
            bind:value={webConfig.DDGS_REGION}
        >
            <option value="us-en">US English</option>
            <option value="uk-en">UK English</option>
            <option value="ca-en">Canada English</option>
            <option value="au-en">Australia English</option>
            <option value="de-de">Germany</option>
            <option value="fr-fr">France</option>
            <option value="es-es">Spain</option>
            <option value="it-it">Italy</option>
            <option value="jp-jp">Japan</option>
            <option value="cn-zh">China</option>
        </select>
    </div>

    <div class="w-full mb-2.5">
        <div class=" self-center text-xs font-medium mb-1">
            {$i18n.t('Safe Search')}
        </div>
        <select
            class="w-full rounded-lg py-2 px-4 text-sm bg-gray-50 dark:text-gray-300 dark:bg-gray-850 outline-hidden"
            bind:value={webConfig.DDGS_SAFESEARCH}
        >
            <option value="strict">Strict</option>
            <option value="moderate">Moderate</option>
            <option value="off">Off</option>
        </select>
    </div>

    <div class="w-full mb-2.5">
        <div class=" self-center text-xs font-medium mb-1">
            {$i18n.t('Language')}
        </div>
        <input
            class="w-full rounded-lg py-2 px-4 text-sm bg-gray-50 dark:text-gray-300 dark:bg-gray-850 outline-hidden"
            placeholder={$i18n.t('Language code (e.g., zh, en, es)')}
            bind:value={webConfig.DDGS_LANGUAGE}
        />
    </div>

    <div class="w-full mb-2.5">
        <div class=" self-center text-xs font-medium mb-1">
            {$i18n.t('Time Limit')}
        </div>
        <select
            class="w-full rounded-lg py-2 px-4 text-sm bg-gray-50 dark:text-gray-300 dark:bg-gray-850 outline-hidden"
            bind:value={webConfig.DDGS_TIMELIMIT}
        >
            <option value="">Any time</option>
            <option value="d">Past day</option>
            <option value="w">Past week</option>
            <option value="m">Past month</option>
            <option value="y">Past year</option>
        </select>
    </div>

    <div class="w-full mb-2.5">
        <div class=" self-center text-xs font-medium mb-1">
            {$i18n.t('Backend')}
        </div>
        <select
            class="w-full rounded-lg py-2 px-4 text-sm bg-gray-50 dark:text-gray-300 dark:bg-gray-850 outline-hidden"
            bind:value={webConfig.DDGS_BACKEND}
        >
            <option value="auto">Auto</option>
            <option value="lite">Lite</option>
            <option value="html">HTML</option>
        </select>
    </div>
{/if}

[Addono]

This would indeed be great to have!

I'm occasionally getting really poor search results. For example, here the queries have nothing in common with the search results:

Looking in the logs, those requests tend to get routed to Bing. Which matches this comment deedy5/ddgs#375 (comment) where the author of the DDGS package blames it on Bing too.

If it were possible to control the engines DDGS can pick from, or at least allow us to explicitly remove Bing, that should resolve some of those poor results.

[cociweb]

SEARXNG_LANGUAGE added In #19909. It should be similarily implemented I think.