Skip to content

Commit

Permalink
[fix] duckduckgo lite engine: set HTTP header 'Referer'
Browse files Browse the repository at this point in the history
We have had problems with this before, the bot protection from ddg-lite seems to
have included this referer in the rating [1][2].

From reverse engineering:

- The Referer ``https://google.com/`` was set in commt 257dc7d --> DDG lite
  does not like this referer anymore!

- The 'Referer' header is only set on second and follow up pages but not on the
  first page

- The vqd value is not needed on the first page, the ddg-lite client sets this
  value only on follow up pages / this can help to reduce the vqd requests from
  SearXNG.

Related to 'Referer' header & ddg requests:

[1] #2161
[2] #2081

Closes: #2796
Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
  • Loading branch information
return42 committed Oct 10, 2023
1 parent fa5b2a7 commit 9197efa
Showing 1 changed file with 4 additions and 1 deletion.
5 changes: 4 additions & 1 deletion searx/engines/duckduckgo.py
Original file line number Diff line number Diff line change
Expand Up @@ -222,7 +222,6 @@ def request(query, params):
# link again and again ..

params['headers']['Content-Type'] = 'application/x-www-form-urlencoded'
params['headers']['Referer'] = 'https://google.com/'

# initial page does not have an offset
if params['pageno'] == 2:
Expand All @@ -248,6 +247,10 @@ def request(query, params):
params['data']['nextParams'] = form_data.get('nextParams', '')
params['data']['v'] = form_data.get('v', 'l')

# request needs a vqd argument
params['data']['vqd'] = get_vqd(query)
params['headers']['Referer'] = 'https://lite.duckduckgo.com/'

params['data']['kl'] = eng_region
params['cookies']['kl'] = eng_region

Expand Down

0 comments on commit 9197efa

Please sign in to comment.