Google results not showing up

[Krando]

Version of SearXNG, commit number if you are using on master branch and stipulate if you forked SearXNG
2022.08.01-7c9c1124

How did you install SearXNG?
I installed directly by following the step-by-step installation tutorial on the documentation, I open up searx through searx/webapp.py.

A clear and concise description of what the bug is.
Everything was working fine until when Google suddenly stopped fetching results two days ago, all other engines work with no problems. Any search with Google returns "Sorry! we didn't find any results. Please use another query or search in more categories.".

This is strange particularly because 1) google autocomplete works, 2) there are no errors, 3) in the engines section, google shows that the response time is 2.0 and apparently no error is detected), 4) setting max timeout to 25 seconds didn't fix the problem, 5) turning on mobile_ui also didn't fix the problem.

How To Reproduce

Expected behavior
Google should return results.

Screenshots & Logs
DEBUG searx.webapp : starting webserver on 127.0.0.1:8888

• Serving Flask app 'webapp' (lazy loading)
• Environment: production
  WARNING: This is a development server. Do not use it in a production deployment.
  Use a production WSGI server instead.
• Debug mode: on
  INFO werkzeug : * Running on http://127.0.0.1:8888 (Press CTRL+C to quit)
  INFO werkzeug : * Restarting with stat
  INFO searx : load the user settings from /etc/searxng/settings.yml
  INFO searx : max_request_timeout=None
  INFO searx : version: 2022.08.01-7c9c1124
  INFO searx.shared : Use shared_simple implementation
  DEBUG searx.webapp : static directory is /usr/local/searxng/searxng-src/searx/static
  DEBUG searx.webapp : templates directory is /usr/local/searxng/searxng-src/searx/templates
  DEBUG searx.locales : locales_initialize: /usr/local/searxng/searxng-src/searx/translations
  DEBUG searx.plugins : searx.plugins.ahmia_filter: loaded
  DEBUG searx.plugins : searx.plugins.hash_plugin: loaded
  DEBUG searx.plugins : searx.plugins.hostname_replace: loaded
  DEBUG searx.plugins : searx.plugins.limiter: loaded
  DEBUG searx.plugins : searx.plugins.oa_doi_rewrite: loaded
  DEBUG searx.plugins : searx.plugins.search_on_category_select: loaded
  DEBUG searx.plugins : searx.plugins.self_info: loaded
  DEBUG searx.plugins : searx.plugins.tor_check: loaded
  DEBUG searx.plugins : searx.plugins.tracker_url_remover: loaded
  DEBUG searx.plugins : searx.plugins.vim_hotkeys: loaded
  DEBUG searx.engines.dailymotion : Initialized
  INFO searx : Send SIGUSR1 signal to pid 37461 to start the checker
  INFO searx : debug mode: checker is disabled
  DEBUG searx.webapp : starting webserver on 127.0.0.1:8888
  WARNING werkzeug : * Debugger is active!
  DEBUG searx.network.soundcloud : HTTP Request: GET https://soundcloud.com "HTTP/2 200 OK" (text/html)
  DEBUG searx.engines.wikidata : request time 0:00:00.751000
  DEBUG searx.engines.wikidata : Initialized
  DEBUG searx.network.z-library : HTTP Request: GET https://z-lib.org "HTTP/2 200 OK" (text/html; charset=UTF-8)
  DEBUG searx.engines.z-library : using base_url: https://pk1lib.org
  DEBUG searx.engines.z-library : Initialized
  DEBUG searx.network.soundcloud : HTTP Request: GET https://a-v2.sndcdn.com/assets/50-02cc92e2.js "HTTP/2 200 OK" (application/javascript)
  DEBUG searx.network.wolframalpha : HTTP Request: GET https://www.wolframalpha.com/input/api/v1/code?ts=9999999999999999999 "HTTP/1.1 200 OK"
  DEBUG charset_normalizer : Encoding detection: ascii is most likely the one.
  DEBUG searx.engines.wolframalpha : Initialized
  DEBUG searx.engines.soundcloud : Initialized
  DEBUG searx.webapp : http%3A//127.0.0.1%3A8888/ uses locale en
  INFO werkzeug : 127.0.0.1 - - [07/Aug/2022 22:17:58] "GET / HTTP/1.1" 200 -
  INFO werkzeug : 127.0.0.1 - - [07/Aug/2022 22:17:59] "GET /static/themes/simple/css/searxng.min.css HTTP/1.1" 304 -
  INFO werkzeug : 127.0.0.1 - - [07/Aug/2022 22:17:59] "GET /static/themes/simple/js/searxng.head.min.js HTTP/1.1" 304 -
  INFO werkzeug : 127.0.0.1 - - [07/Aug/2022 22:17:59] "GET /static/themes/simple/js/searxng.min.js HTTP/1.1" 304 -
  INFO werkzeug : 127.0.0.1 - - [07/Aug/2022 22:17:59] "GET /static/themes/simple/img/searxng.png HTTP/1.1" 304 -
  DEBUG searx.search : actual_timeout=25.0 (default_timeout=25.0, ?timeout_limit=None, max_request_timeout=None)
  DEBUG searx.network.google : HTTP Request: GET https://www.google.com/search?q=cats&hl=en&lr=lang_en&ie=utf8&oe=utf8&start=0&filter=0 "HTTP/2 200 OK" (text/html; charset=UTF-8)
  DEBUG searx.engines.google : did not find 'answer'
  DEBUG searx.webapp : http%3A//127.0.0.1%3A8888/search uses locale en
  INFO werkzeug : 127.0.0.1 - - [07/Aug/2022 22:18:15] "POST /search HTTP/1.1" 200 -
  INFO werkzeug : 127.0.0.1 - - [07/Aug/2022 22:18:15] "GET /static/themes/simple/css/searxng.min.css HTTP/1.1" 304 -
  INFO werkzeug : 127.0.0.1 - - [07/Aug/2022 22:18:15] "GET /static/themes/simple/js/searxng.head.min.js HTTP/1.1" 304 -
  INFO werkzeug : 127.0.0.1 - - [07/Aug/2022 22:18:15] "GET /static/themes/simple/js/searxng.min.js HTTP/1.1" 304 -

Additional context

[tiekoetter]

@Krando Sounds like the IP your instance uses to connect to google is blocked. Do you get a CAPTCHA when visiting google via the same IP?

[Krando]

@Krando Sounds like the IP your instance uses to connect to google is blocked. Do you get a CAPTCHA when visiting google via the same IP?

I don't get any sort of CAPTCHA when using Google through the browser and everything works fine there, the problem only concerns searxng+google.

For example, the link "https://www.google.com/search?q=cats&hl=en&lr=lang_en&ie=utf8&oe=utf8&start=0&filter=0" works fine on the browser and opens up without any CAPTCHA.

[Krando]

Another important to know is that my IP isn't static and changes often, so this problem is likely to do with my local searxng instance. Can someone help me figure out what the actual problem is? If other logs are needed, please let me know.

[return42]

Does use_mobile_ui: true help? See https://docs.searxng.org/src/searx.engines.google.html#module-searx.engines.google

[vojkovic]

I think something changed at Google that broke the functionality in searxng. Certain google searches on my instance are breaking Google as well. This persisted over three different IP addresses.

Does use_mobile_ui: true help? See https://docs.searxng.org/src/searx.engines.google.html#module-searx.engines.google

I think he answered this:

5. turning on mobile_ui also didn't fix the problem.

Interestingly, the instant answers are not affected by this and they show even when no other results are there.

[Krando]

Turning on use_mobile_ui unfortunately did not help either. I should also mention that Google no longer works at many instances listed at searx.space. At my own instance, certain queries like "football" show around 3 or 4 results but all other queries don't show any results, so what vojkovic said is probably right. Google likely made a change which is now messing up with searxng.

[return42]

Google likely made a change which is now messing up with searxng.

IDK but what I observed

• !go football :de works (my server's IP is located in DE)
• !go football :en no results
• !go football :fr works

May be this issue is related to the domian selection:

searxng/searx/engines/google.py

Line 209 in 925f3ff

ret_val['subdomain'] = 'www.' + google_domains.get(country.upper(), 'google.com')

I don't have the time right no to check / sorry.

[Krando]

!go football :ru
!go football :fr 

These two commands work for me and fetch results without any problems, but if I change the language of the engine to either Russian or French in order to force searxng to use google.ru or google.fr, it works for one or two searches and then doesn't work afterwards.

The debug output gives this:

DEBUG   searx.network.google          : HTTP Request: GET https://www.google.ru/search?q=football&hl=ru&lr=lang_ru&ie=utf8&oe=utf8&start=0&filter=0 "HTTP/2 200 OK" (text/html; charset=UTF-8)
DEBUG   searx.engines.google          : did not find 'answer'

But !go football :ru works. This is bizarre. Perhaps a clue to the actual problem is hidden somewhere in this.

[jtabeling]

Same issue here. No google search results are returned.

[raulpcs]

No google results for me since yesterday.
update: #1633 it also fixed it for me.

[raskitoma]

No google results for me since today

[Vic563]

Me either. No google results.

[blahgeek]

No google results for me since yesterday.
update: #1633 fixes for me

[Vic563]

@blahgeek how does one use the 1633 fix to correct the google issue?

[raulpcs]

@blahgeek how does one use the 1633 fix to correct the google issue?

You need to edit a file inside the docker file system to enter use:
docker exec -it searxng sh
Then go the file indicated on the files changed section:
searx/engines/google.py
then replace the problematic line using vi editor:
results_xpath = '//div[contains(@Class, "jtfYYd")]'
for
results_xpath = '//div[contains(@Class, "kvH3mc")]'
restart your container an then it should work.

[Vic563]

@raulpcs Thank you that worked. I can now see google results.

[raulpcs]

Google searches stopped working again for me.

[vojkovic]

Google searches stopped working again for me.

Try upgrading to the latest commit. #1643 fixed this issue.

[raulpcs]

Google searches stopped working again for me.

Try upgrading to the latest commit. #1643 fixed this issue.

Thanks, this commit fixed it.