[fix-2146] set different HTTP Referer header to DuckDuckGo requests #2161
Conversation
For what ever reasons, ddg-lite don't like the Referer https://lite.duckduckgo.com/ In an interactive session in the WEB browser the the Reverer has exactly this value, but ddg-lite don't like this value when the request is build up by SearXNG. The new value is: https://google.com/ What fakes a user comes from a google link. Related: searxng#2081 Closes: searxng#2146 Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
Like it is often in hacking .. it was just a typo that brought it to my attention. But I could not explain myself why this is so 😃
Other engines are not related to this patch .. if they work for you in your test this doesn't say anything about a daily usage on a public instance .. e.g. the startpage CAPTCHA will come back at some point. |
Somebody's probably good at it at DDG then 👀
You're right and I admit I should've known this but it was my OCD+trauma talking tbh 😂 Thanks a lot for this patch! |
We have had problems with this before, the bot protection from ddg-lite seems to have included this referer in the rating [1][2]. From reverse engineering: - The Referer ``https://google.com/`` was set in commt 257dc7d --> DDG lite does not like this referer anymore! - The 'Referer' header is only set on second and follow up pages but not on the first page - The vqd value is not needed on the first page, the ddg-lite client sets this value only on follow up pages / this can help to reduce the vqd requests from SearXNG. Related to 'Referer' header & ddg requests: [1] searxng#2161 [2] searxng#2081 Closes: searxng#2796 Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
We have had problems with this before, the bot protection from ddg-lite seems to have included this referer in the rating [1][2]. From reverse engineering: - The Referer ``https://google.com/`` was set in commt 257dc7d --> DDG lite does not like this referer anymore! - The 'Referer' header is only set on second and follow up pages but not on the first page - The vqd value is not needed on the first page, the ddg-lite client sets this value only on follow up pages / this can help to reduce the vqd requests from SearXNG. Related to 'Referer' header & ddg requests: [1] #2161 [2] #2081 Closes: #2796 Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
For what ever reasons, ddg-lite don't like the Referer
In an interactive session in the WEB browser the Referer has exactly this value, but ddg-lite don't like this value when the request is build up by SearXNG. The new value is:
What fakes a user comes from a google link.
Related: #2081
Closes: #2146