ver 3.8 - connection blocked due to unusual activity

[timeislove]

hello,

tried the snap app (V3.8). the 3.8 was just promoted from edge channel. the previous stable snap is 3.7.1

running snapd (2.38+) on xubuntu 18.04.2

regards

$ googler -d obama
[DEBUG] googler version 3.8
[DEBUG] Python version 3.5.2
[DEBUG] Connecting to new host www.google.com
[DEBUG] Fetching URL /search?ie=UTF-8&oe=UTF-8&q=obama&sei=TRYM7ltREemzXAAl09OLPw
[DEBUG] Cookie: 1P_JAR=2019-04-10-05
Traceback (most recent call last):
File "/snap/googler/300/usr/local/bin/googler", line 3257, in
main()
File "/snap/googler/300/usr/local/bin/googler", line 3246, in main
repl.cmdloop()
File "/snap/googler/300/usr/local/bin/googler", line 2732, in cmdloop
self.fetch_and_display()
File "/snap/googler/300/usr/local/bin/googler", line 2349, in enforced_method
method(self, *args, **kwargs)
File "/snap/googler/300/usr/local/bin/googler", line 2540, in fetch_and_display
self.fetch()
File "/snap/googler/300/usr/local/bin/googler", line 2349, in enforced_method
method(self, *args, **kwargs)
File "/snap/googler/300/usr/local/bin/googler", line 2447, in fetch
page = self._conn.fetch_page(self._google_url.relative())
File "/snap/googler/300/usr/local/bin/googler", line 1991, in fetch_page
raise GoogleConnectionError('Connection blocked due to unusual activity.')
main.GoogleConnectionError: Connection blocked due to unusual activity

[jarun]

Please take a look at #244.

[jarun]

For more info search for issues with the string "connection blocked"

[timeislove]

i use your ddgr on the same notebook around the same time under the same network environment. ddgr query is not blocked and successful.

the block issue happened after the snap app was promoted to ver 3.8. seems no problem with 3.7.1 (snap).

[jarun]

I am not sure if this is due to changes in googler or google's policy. I tested using the 3.8 release just now and it's working for me (results below).

Are you saying immediately after you get the error with 3.8, if you try 3.7.1 it works for you?

Do you have any means to try on a different network? Otherwise, can you automate this on Travis/CircleCI and test? With 3.8 our automated tests passed on both Travis and CircleCI.

$ /usr/local/bin/googler obama

 1.  The Office of Barack and Michelle Obama [barackobama.com]
     Welcome to the Office of Barack and Michelle Obama. We Love You Back. Play video. The Office of
     Barack and Michelle Obama · Obama Foundation. obama.

 2.  Barack Obama - Wikipedia [en.wikipedia.org]
     Barack Hussein Obama II (/bəˈrɑːk huːˈseɪn oʊˈbɑːmə/ ( listen); born August 4, 1961) is an American
     attorney and politician who served as the 44th president of the United States from 2009 to 2017.

 3.  Barack Obama (@BarackObama) | Twitter [twitter.com]
     The latest Tweets from Barack Obama (@BarackObama). Dad, husband, President, citizen. Washington,
     DC.

 4.  Barack Obama - Home | Facebook [www.facebook.com]
     Barack Obama, Washington, DC. 54M likes. Dad, husband, former President, citizen.

 5.  Barack Obama | Biography, Presidency, & Facts | Britannica.com [www.britannica.com]
     15 Mar 2019 ... did you know? When Obama traveled to Kenya in 2006, thousands of people waited to
     welcome him, lining the streets. The Obamas signed a ...

 6.  Ellen's Tribute to the Obamas - YouTube [www.youtube.com]
     19 Jan 2017 - 5 min - Uploaded by TheEllenShowTo commemorate President Obama's last day in office,
     Ellen took a look back at  some of her ...

 7.  Obama White House Archives [obamawhitehouse.archives.gov]
     See the President's daily schedule, explore behind-the-scenes photos from inside the White House,
     and find out all the ways you can engage with the most ...

 8.  Barack Obama - The New York Times [www.nytimes.com]
     News about Barack Obama. Commentary and archival information about Barack Obama from The New York
     Times.

 9.  Trump digs in on false claim that he stopped Obama's family ... [www.washingtonpost.com]
     1 hour ago ... “President Obama had child separation. Take a look. The press knows it, you know it,
     we all know it. I didn't have — I'm the one that stopped it.

 10. Barack Obama - Los Angeles Times [www.latimes.com]
     Obama senior adviser Valerie Jarrett shares her road from Chicago to the White ... Judge throws out
     Trump order and restores Obama-era drilling ban in Arctic.

[zmwangx]

Amusingly, on my Linode server (Ubuntu bionic), when I run googler directly from Ubuntu, the requests are perma blocked, but when I run googler tests from within an alpine slim-stretch container with Docker on the exact same host, the requests are not blocked. Should be the exact same IP address, the exact same request and the exact same headers. What a mystery.

[zmwangx]

Only difference I can think of: on Ubuntu bionic I have Python 3.6.7 with openssl 1.1.0g-2ubuntu4.3, on python:3.6-slim I have Python 3.6.8 with openssl 1.1.0j-1~deb9u1, so there could be a cipher suite difference.

[timeislove]

Only difference I can think of: on Ubuntu bionic I have Python 3.6.7 with openssl 1.1.0g-2ubuntu4.3, on python:3.6-slim I have Python 3.6.8 with openssl 1.1.0j-1~deb9u1, so there could be a cipher suite difference.

just a second though, would it be due to the 'behaviour' of the app sending queries or msg that google looks suspicious hence block the query?

i could google via firefox from the same notebook and the same network. i do not think the ip was blacklisted by google.

[jarun]

Yes, that's also possible. googler has it's own UA so it's possible to detect. Generally, it gets blocked if it's used for firing automated queries in succession. So please use it wisely.

[zmwangx]

As I said, googler running in Ubuntu and googler running inside a python:3.6-slim container on top of Ubuntu should be sending the exact same requests with the exact same headers (user agent included) so not sure what "behavior" is different.

googler is of course worlds apart from a JavaScript-enabled browser. Think of reCAPTCHA v3 where there's only a single checkbox, and the amount of completely transparent bot detection that happens behind the scenes.