[BUG/FEATURE] Bypass EU cookie consent #243

Suika · 2021-03-30T02:18:03Z

Describe the bug
I use proxy setup that rotates the proxing in the background and from time to time, when I connect to EU in this particular instance DE, whoogle will display a "Cookie consent" form from google.
After reloading the pages 2-3 times it vanishes, nonetheless it shows up every ~1-3 request via the addressbar search.

To Reproduce
Steps to reproduce the behavior:

Add you instance to your browser, in my case ungoogled-chromium
Either proxy or VPN the instance to the EU(DE)
Perform a serach via the url bar
See error

Deployment Method

Version of Whoogle Search

Latest build from [source] (i.e. GitHub, Docker Hub, pip, etc)
Version [version number]
Not sure

Desktop (please complete the following information):

OS: [Win/Linux]
Browser [ungoogled-chromium]
Version [88.0.4324.182]

Additional context

The text was updated successfully, but these errors were encountered:

readall · 2021-03-30T16:11:37Z

This seems some kind of change from goog. I am also facing the same on my docker instance.

benbusby · 2021-03-30T16:27:03Z

That's annoying. I'll try to replicate on my end and see what I can do.

@readall I assume your instance is also running in the EU?

Hunam6 · 2021-03-30T16:30:00Z

I assume your instance is also running in the EU?

Mine is too

benbusby · 2021-03-30T17:10:35Z

From a cursory bit of testing, it looks like accepting this dialog sets a CONSENT cookie with a value that roughly matches the following format YES+[country code].[language]+[V#]+[B?]+[###].

Examples:

Netherlands (UK English) -- CONSENT: YES+NL.en-GB+V9+BX+723

Sweden (Swedish) -- CONSENT: YES+SE.sv+V10+B+256

I tried manually swapping the Sweden and Netherlands cookies, and it didn't seem to have a problem with that, so I'm curious if there's just a default value that can be set in requests.send() to bypass this altogether. Unfortunately it seems that whatever cryptic method they're using to generate the exact cookie value isn't easily accessible, but that might not be an issue (for now).

I'll try to look into this some more soon. Thanks everyone for reporting it!

dr460nf1r3 · 2021-03-30T19:28:31Z

I can confirm that this happens since yesterday every few searches.

Suika · 2021-03-30T23:32:24Z

@benbusby ok, here is a strange one. Set cookie CONSENT=PENDING+600 and you should be fine for as many queries as you can do. Set it to ~220 you might get a consent page, set it to <=210 and you will always get the consent page. Max is 999 and it seems to be doing fine, anything higher resets the number to 060.

Don't explicitly know how it is tied together, but that one cookie seems to keep the consent away for a duration of time.

readall · 2021-03-31T06:18:20Z

our instance is also running in th

@benbusby Yes, my VPS is hosted within EU.

readall · 2021-03-31T06:21:50Z

@benbusby ok, here is a strange one. Set cookie CONSENT=PENDING+600 and you should be fine for as many queries as you can do. Set it to ~220 you might get a consent page, set it to <=210 and you will always get the consent page. Max is 999 and it seems to be doing fine, anything higher resets the number to 060.

@Suika if you don't mind, where is code can this setting be done. I would like to try that.

Suika · 2021-03-31T10:59:14Z

@benbusby ok, here is a strange one. Set cookie CONSENT=PENDING+600 and you should be fine for as many queries as you can do. Set it to ~220 you might get a consent page, set it to <=210 and you will always get the consent page. Max is 999 and it seems to be doing fine, anything higher resets the number to 060.

@Suika if you don't mind, where is code can this setting be done. I would like to try that.

Quick and dirty would be https://github.com/benbusby/whoogle-search/blob/develop/app/request.py#L210-L212 just slap another entry in like 'cookie': 'CONSENT=PENDING+999'

readall · 2021-03-31T12:21:14Z

Quick and dirty would be https://github.com/benbusby/whoogle-search/blob/develop/app/request.py#L210-L212 just slap another entry in like 'cookie': 'CONSENT=PENDING+999'
@Suika
Thanks for this hack. In effect the cookie-consent appears for a very brief moment (like few mili seconds) and then the search results load. So the irritation is temporarily addressed.

Suika · 2021-03-31T13:16:37Z

Quick and dirty would be https://github.com/benbusby/whoogle-search/blob/develop/app/request.py#L210-L212 just slap another entry in like 'cookie': 'CONSENT=PENDING+999'
@Suika
Thanks for this hack. In effect the cookie-consent appears for a very brief moment (like few mili seconds) and then the search results load. So the irritation is temporarily addressed.

You saw that probably just after you changed it. That is normal. Any consequent searches should not show anything. At least it doesn't for me.

See discussion on #243

benbusby · 2021-03-31T13:55:39Z

Just pushed a hotfix that includes @Suika's fix while I look into this a bit more. It's available on the develop and heroku-app-beta branches, as well as the beta and buildx-experimental images (once the CI build finishes).

federicotorrielli · 2021-04-01T08:13:45Z

From today on it's not possibile anymore to search in EU, this pops up anytime on 0.3.1, can we get a new release? Thanks!

Hunam6 · 2021-04-01T08:23:17Z

@federicotorrielli
I'm using the heroku-app-beta branch as specified in the previous comment by @benbusby and it works as intended.

jeroenev · 2021-04-01T08:25:22Z

From today on it's not possibile anymore to search in EU, this pops up anytime on 0.3.1, can we get a new release? Thanks!

same for me, it used to show every few searches, but now it shows every single time I click search.

federicotorrielli · 2021-04-01T08:57:12Z

whoogle beta branch isn't really working for me right now btw, I tried creating more instances but this keeps showing:

Hunam6 · 2021-04-01T09:15:58Z

@federicotorrielli I can relate, I have this bug since I updated my fork.
This bug was introduced here: whoogle-mirror/whoogle-search@449999d...282208f

gpopesc · 2021-04-01T09:30:19Z

use beta tag in docker compose or docker cli in order to pull the corresponding image.
the latest version 0.3.1 it doesn't work for EU users
it works for me
image: benbusby/whoogle-search:beta

LeonMusCoden · 2021-04-01T10:01:41Z

buildx-experimental works again, thanks.

docker pull benbusby/whoogle-search:buildx-experimental
docker run --publish 5000:5000 --detach --name whoogle-search benbusby/whoogle-search:buildx-experimental

benbusby · 2021-04-01T14:28:41Z

The heroku-app-beta branch should be fixed now (minor regression, unrelated to this issue).

inlophe · 2021-04-02T03:46:31Z

Is it just me or the beta and buildx-experimental docker tag stuck at starting for anyone else? I tried with both compose and run directly on 2 different servers and all stuck at starting then change to unhealthy after a while. But when I revert to the latest tag, it works (but with EU cookie consent ofc). Running it with -it produces the same log for latest and beta/buildx-experimental

gpopesc · 2021-04-02T08:15:25Z

@inlophe no problem for me running beta version since yesterday
version: "3"
services:
whoogle-search:
image: benbusby/whoogle-search:beta
container_name: whoogle-search
ports:
- 6051:5000
restart: unless-stopped

See discussion on #243

Suika · 2021-04-07T09:14:25Z

Sooo, did this fix it? Well, if the non-consensual searching ever breaks the only way will be, is to give consent ytdl-org/youtube-dl@14f29f0.
Unless there is a loophole. Either way, if fixed do close the issue?

jeroenev · 2021-04-07T09:17:41Z

Currently the 4.0 release seems to work again for me

dr460nf1r3 · 2021-04-07T09:25:20Z

I also did not experience issues after the fix was pushed, seems like it is indeed fixed

In the EU there exists a "General Data Protection Regulation" [1] aka GDPR (BTW: very user friendly!) which requires consent to tracking. To get the consent from the user, google-news requests are redirected to confirm and get a CONSENT Cookie from https://consent.google.de/s?continue=... This patch adds a CONSENT Cookie to the google-news request to avoid redirection. The behavior of the CONTENTS cookies over all google engines seems similar but the pattern is not yet fully clear to me, here are some random samples from my analysis .. Using common google search from different domains:: google.com: CONSENT=YES+cb.{{date}}-14-p0.de+FX+816 google.de: CONSENT=YES+cb.{{date}}-14-p0.de+FX+333 google.fr: CONSENT=YES+srp.gws-{{date}}-0-RC2.fr+FX+826 When searching about videos (google-videos):: google.es: CONSENT=YES+srp.gws-{{date}}-0-RC2.es+FX+076 google.de: CONSENT=YES+srp.gws-{{date}}-0-RC2.de+FX+171 Google news has only one domain for all languages:: news.google.com: CONSENT=YES+cb.{{date}}-14-p0.de+FX+816 Using google-scholar search from different domains:: scholar.google.de: CONSENT=YES+cb.{{date}}-14-p0.de+FX+333 scholar.google.fr: does not use such a cookie / did not ask the user scholar.google.es: does not use such a cookie / did not ask the user Interim summary: Pattern is unclear and I won't apply the CONSENT cookie to all google engines. More experience is need before we generalize the CONSENT cookies over all google engines. Related: - e9a6ab4 [fix] youtube - send CONSENT Cookie to not be redirected - benbusby/whoogle-search#311 - benbusby/whoogle-search#243 [1] https://en.wikipedia.org/wiki/General_Data_Protection_Regulation Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>

Suika added the bug Something isn't working label Mar 30, 2021

benbusby mentioned this issue Mar 30, 2021

[BUG] Google often ask for cookie approval #242

Closed

8 tasks

benbusby changed the title ~~[BUG] Cookie consent shows up every X searches~~ [BUG/FEATURE] Bypass EU cookie consent Mar 30, 2021

federicotorrielli mentioned this issue Mar 31, 2021

[BUG] Google cookie consent hijack every 3 page #248

Closed

8 tasks

benbusby added a commit that referenced this issue Mar 31, 2021

Hotfix: Set EU consent cookie to pending for all requests

fb863b5

See discussion on #243

bjornberfelo mentioned this issue Apr 1, 2021

Not able to search any more #251

Closed

Hunam6 mentioned this issue Apr 1, 2021

[BUG] "The page isn’t redirecting properly" after heroku quick deploy #252

Closed

6 tasks

trumad mentioned this issue Apr 1, 2021

[BUG] Whoogle returns a broken Page on search #253

Closed

8 tasks

ghost mentioned this issue Apr 2, 2021

[BUG] "Before you continue to Google Search" #257

Closed

benbusby added a commit that referenced this issue Apr 2, 2021

Hotfix: Set EU consent cookie to pending for all requests

fdd4ee5

See discussion on #243

benbusby mentioned this issue Apr 5, 2021

[BUG] New update not fixing old problem #262

Closed

benbusby closed this as completed Apr 7, 2021

Suika mentioned this issue May 14, 2021

[BUG] EU consent enforced, no more pending period #311

Closed

8 tasks

return42 mentioned this issue Jun 18, 2021

Fix google news searxng/searxng#156

Merged

opsdisk mentioned this issue Nov 13, 2021

Got no results opsdisk/yagooglesearch#5

Closed

rickykresslein mentioned this issue Aug 16, 2023

[BUG] Before u continue to Google #1053

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[BUG/FEATURE] Bypass EU cookie consent #243

[BUG/FEATURE] Bypass EU cookie consent #243

Suika commented Mar 30, 2021

readall commented Mar 30, 2021

benbusby commented Mar 30, 2021

Hunam6 commented Mar 30, 2021 •

edited

benbusby commented Mar 30, 2021

dr460nf1r3 commented Mar 30, 2021

Suika commented Mar 30, 2021

readall commented Mar 31, 2021

readall commented Mar 31, 2021

Suika commented Mar 31, 2021

readall commented Mar 31, 2021

Suika commented Mar 31, 2021 •

edited

benbusby commented Mar 31, 2021

federicotorrielli commented Apr 1, 2021

Hunam6 commented Apr 1, 2021

jeroenev commented Apr 1, 2021

federicotorrielli commented Apr 1, 2021 •

edited

Hunam6 commented Apr 1, 2021

gpopesc commented Apr 1, 2021

LeonMusCoden commented Apr 1, 2021

benbusby commented Apr 1, 2021

inlophe commented Apr 2, 2021

gpopesc commented Apr 2, 2021

Suika commented Apr 7, 2021

jeroenev commented Apr 7, 2021

dr460nf1r3 commented Apr 7, 2021

[BUG/FEATURE] Bypass EU cookie consent #243

[BUG/FEATURE] Bypass EU cookie consent #243

Comments

Suika commented Mar 30, 2021

readall commented Mar 30, 2021

benbusby commented Mar 30, 2021

Hunam6 commented Mar 30, 2021 • edited

benbusby commented Mar 30, 2021

dr460nf1r3 commented Mar 30, 2021

Suika commented Mar 30, 2021

readall commented Mar 31, 2021

readall commented Mar 31, 2021

Suika commented Mar 31, 2021

readall commented Mar 31, 2021

Suika commented Mar 31, 2021 • edited

benbusby commented Mar 31, 2021

federicotorrielli commented Apr 1, 2021

Hunam6 commented Apr 1, 2021

jeroenev commented Apr 1, 2021

federicotorrielli commented Apr 1, 2021 • edited

Hunam6 commented Apr 1, 2021

gpopesc commented Apr 1, 2021

LeonMusCoden commented Apr 1, 2021

benbusby commented Apr 1, 2021

inlophe commented Apr 2, 2021

gpopesc commented Apr 2, 2021

Suika commented Apr 7, 2021

jeroenev commented Apr 7, 2021

dr460nf1r3 commented Apr 7, 2021

Hunam6 commented Mar 30, 2021 •

edited

Suika commented Mar 31, 2021 •

edited

federicotorrielli commented Apr 1, 2021 •

edited