New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Sometimes Google works and sometimes it doesn't? #1118

Open
irule2day opened this Issue Dec 16, 2017 · 38 comments

Comments

Projects
None yet
@irule2day

irule2day commented Dec 16, 2017

I have switched from one instance to another when Google search doesn't show up, but inevitably Google results do not show up at times for all instances I've tried. Is anyone else having the same problem? My main instance is searx.dk

@prolibre

This comment has been minimized.

prolibre commented Dec 16, 2017

Google certainly made some changes. For me google answers only one hour a day about... for the best days. I have three ip's and now they're all blocked. With my vpn I can still use google with CAPTCHA but it doesn't unlock searx (on the same ip).


Google a certainement fait des modifications. Pour moi google ne répond qu'une heure par jour environ... pour les meilleurs jours. J'ai trois ip et maintenant elles sont toutes bloquées. Avec mon vpn je peux encore utiliser google avec le CAPTCHA mais cela ne débloque pas searx (sur la même ip).

@irule2day

This comment has been minimized.

irule2day commented Dec 17, 2017

Ah yes I've also noticed they started blocking vpns for searching too with CAPTCHAs. Not a coder or anything, but it doesn't look like there is an easy solution for this.

@Pofilo

This comment has been minimized.

Collaborator

Pofilo commented Dec 18, 2017

#1089

Even with the VPN down for a long time, my IP was still black listed.
When you put the captcha on your browser, you are safe for some times but because they put a cookie ! That's why google is not answering to searx, they don't find a cookie so they ask for the captcha again.
And as the captcha given to searx is never filled, I supposed our IP will remain in the black list longer :/

@prolibre

This comment has been minimized.

prolibre commented Dec 20, 2017

Very good fix for me. Have a nice day.

@scarejar

This comment has been minimized.

scarejar commented Dec 25, 2017

My searx instances still have this problem. Is this something they'll have to manually do? Or is it an upcoming update that will fix it automatically ?

@asciimoo

This comment has been minimized.

Owner

asciimoo commented Dec 25, 2017

@scarejar update to the latest master

@genericsauce

This comment has been minimized.

genericsauce commented Jan 12, 2018

Seems like googles switched something around again, updated to latest and still complaining about captcha

@ghost

This comment has been minimized.

ghost commented Jan 30, 2018

Over the last few days I have been noticing this with my searx as well.

@Pofilo

This comment has been minimized.

Collaborator

Pofilo commented Jan 30, 2018

The problem is coming back for me too (I'm on last master commit).

@qutetemp

This comment has been minimized.

qutetemp commented Feb 3, 2018

There are a lot of instances which don't have google working. Might I suggest releasing a new version after something like this is fixed in order to ease up searx.me ?

@Dominion0815

This comment has been minimized.

Dominion0815 commented Feb 5, 2018

same problem here with 0.13.1:

ERROR:searx.search:engine google : exception : CAPTCHA required
Traceback (most recent call last):
  File "/usr/local/searx/searx/search.py", line 104, in search_one_request_safe
    search_results = search_one_request(engine, query, request_params)
  File "/usr/local/searx/searx/search.py", line 87, in search_one_request
    return engine.response(response)
  File "/usr/local/searx/searx/engines/google.py", line 217, in response
    raise RuntimeWarning(gettext('CAPTCHA required'))
RuntimeWarning: CAPTCHA required

@webworker01

This comment has been minimized.

webworker01 commented Feb 6, 2018

Is there possibly a way to pass the captcha through to the searx interface? If so would this be a problem with privacy?

PS Where can I find the log files?

@kettbi

This comment has been minimized.

kettbi commented Feb 7, 2018

Hi,
The default log files directory is /var/log/uwsgi/app/

@webworker01

This comment has been minimized.

webworker01 commented Feb 7, 2018

Thanks.. looks like it's captcha for me too.

@scroom

This comment has been minimized.

scroom commented Feb 10, 2018

Same problem here.

@asciimoo

This comment has been minimized.

Owner

asciimoo commented Feb 10, 2018

Everybody who has google errors, could you post what environment do you use?

  • Where is the instance being hosted (big datacenter/aws/digitalocean/pc/etc...)?
  • Is it a public instance?
  • Is there any other service running on the same node?
@scroom

This comment has been minimized.

scroom commented Feb 10, 2018

I'm running a VPS provided by Contabo.de. On the same server I'm running a mastodon instance. The searx instance is public.

@Dominion0815

This comment has been minimized.

Dominion0815 commented Feb 10, 2018

I'm running a Proxmox VM on my own dedicated server. And yes it is a public instance without other services.

@kettbi

This comment has been minimized.

kettbi commented Feb 12, 2018

  • I'm running a dedicated server from OVH
  • It's a public instance (about 20 users realy use it)
  • some other services are running on this server (apache with about 10-15 webservices : nextcloud, kanboard, tt-rss, humhub, ...), gitlab, openfire, ...
@dalf

This comment has been minimized.

Collaborator

dalf commented Feb 12, 2018

@Dominion0815

This comment has been minimized.

Dominion0815 commented Feb 12, 2018

Everything is fine, I'm not listed:

"POSSIBLY SAFE 0/96" and "IP Not Listed (Good!)"

@scroom

This comment has been minimized.

scroom commented Feb 12, 2018

@dalf Same here, everthing is fine:

POSSIBLY SAFE 0/96
IP Not Listed (Good!) -> on all listed blacklists

@dalf

This comment has been minimized.

Collaborator

dalf commented Feb 13, 2018

@Dominion0815 @scroom , another idea: could you try to use your server IP with a real browser (with Javascript) for a time?
most probably using an ssh sock proxy?
https://www.digitalocean.com/community/tutorials/how-to-route-web-traffic-securely-without-a-vpn-using-a-socks-tunnel

@genericsauce

This comment has been minimized.

genericsauce commented Feb 13, 2018

@dalf I just tried this, a google search doesn't prompt a captcha but still my searx instance does.

@Pofilo

This comment has been minimized.

Collaborator

Pofilo commented Feb 13, 2018

With my IP address and a real browser, Google asks me for a captcha (once because they put a cookie after that).

@Dominion0815

This comment has been minimized.

Dominion0815 commented Feb 15, 2018

at this moment I have 9 connections and google problems.
here is my lynx output:


____________________________________________________________________________________________________________________

   Submit
     ____________________________________________________________________________________________________________________

   About this page
   Our systems have detected unusual traffic from your computer network. This page checks to see if it's really you sending
   the requests, and not a robot. Why did this happen?
   This page appears when Google automatically detects requests coming from your computer network which appear to be in
   violation of the Terms of Service. The block will expire shortly after those requests stop. In the meantime, solving the
   above CAPTCHA will let you continue to use our services.
   This traffic may have been sent by malicious software, a browser plug-in, or a script that sends automated requests. If
   you share your network connection, ask your administrator for help — a different computer using the same IP address may
   be responsible. Learn more
   Sometimes you may be asked to solve the CAPTCHA if you are using advanced terms that robots are known to use, or sending
   requests very quickly.
   IP address: 62.xx.xx.xx
   Time: 2018-02-15T15:03:41Z
   URL: https://www.google.com/search?q=mettwurst
@webworker01

This comment has been minimized.

webworker01 commented Feb 15, 2018

I have a public searx instance that I'd like to keep public, but if the captcha thing keeps happening, I can't even use it myself!

I'm putting up cloudflare in front of my instance now.. What are the pros and cons of doing this in relation to privacy?

@Dominion0815

This comment has been minimized.

Dominion0815 commented Feb 15, 2018

my workaround is to activate more default search engines like "duckduckgo" and "startpage"

@webworker01

This comment has been minimized.

webworker01 commented Feb 15, 2018

@Dominion0815 yes this will be the next step if I can't make google reliable for everyone..

Edit: I took the step of changing default search engines to duckduckgo, startpage and bing. I want to keep Google on by default but as I get more traffic to my instance, I still kinda want google working for myself ;)

@webworker01

This comment has been minimized.

webworker01 commented Feb 19, 2018

Would it be possible to display the captcha somewhere on the web interface?

@ghost

This comment has been minimized.

ghost commented Feb 19, 2018

Still no solution to this problem?

@Dominion0815

This comment has been minimized.

Dominion0815 commented Feb 20, 2018

with 0.14.0 and filtron the google captcha problem still exists...

@ghost

This comment has been minimized.

ghost commented Feb 20, 2018

Yes @Dominion0815 , unfortunately... how can I add startpage and duckduckgo to my default search? At this time I have just bing which isn’t the best. And how can I remove google form the default search until it got fixed?

@Dominion0815

This comment has been minimized.

Dominion0815 commented Feb 20, 2018

in your settings.yml set "disabled" to "false" or delete the line.

example:

   - name : startpage
     engine : startpage
     shortcut : sp
     timeout : 6.0
     disabled : False
@ghost

This comment has been minimized.

ghost commented Feb 20, 2018

Great, thank you @Dominion0815!

I am left with one stupid question. To apply those settings I just have to run sudo /etc/init.d/uwsgi restart or sudo service uwsgi restart, is that correct? Mine is running on uwsgi and Apache.

Because if I am doing that my searx isn't starting anymore and so far the only solution for me was to reinstall it...

@dalf

This comment has been minimized.

Collaborator

dalf commented Mar 10, 2018

related to #729

@micah

This comment has been minimized.

micah commented Nov 20, 2018

I've got this problem. I'm running filtron and morty and the latest searx code. I've got ddg, startpage and bing enabled and I've tried to ssh tunnel with the IP and browse with it for a while. I've checked my IPs and they do not appear on RBLs or block lists. I've also tried to use elinks locally on the machine to use google... neither the local or the ssh tunnel are asked for a captcha, and the problem still persists.

Is there a way we could setup multiple IPv6 addresses on the machine and have searx rotate through them for the outgoing search results?

@micah

This comment has been minimized.

micah commented Nov 20, 2018

#1034 might fix this, but it has not received any response since it was submitted.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment