Skip to content
This repository has been archived by the owner on Sep 7, 2023. It is now read-only.

Google never returns results: Sorry! we didn't find any results. Please use another query or search in more categories. #1748

Closed
ioogithub opened this issue Nov 22, 2019 · 11 comments · Fixed by #1749
Closed

Google never returns results: Sorry! we didn't find any results. Please use another query or search in more categories. #1748

ioogithub opened this issue Nov 22, 2019 · 11 comments · Fixed by #1749

Comments

@ioogithub
Copy link

ioogithub commented Nov 22, 2019 •
edited

Google is never returning any results. I always get "Sorry! we didn't find any results. Please use another query or search in more categories."

To troubleshoot, I set debug : True in settings.yml

I can see the request with no errors in the searx.log file:

DEBUG:urllib3.connectionpool:https://www.google.com:443 "GET /search?q=who+is+the+lead+developer+of+openssl&start=0&gws_rd=cr&gbv=1&lr=lang_en-GB&hl=en-GB&ei=x HTTP/1.1" 200 None
DEBUG:searx.search:actual_timeout=2.0 (default_timeout=2.0, ?timeout_limit=None, max_request_timeout=None)

so it isn't timing out or throwing an error.

If I take the same GET and try it with curl:

curl -A "Mozilla/5.0" "https://www.google.com/search?q=who+is+the+lead+developer+of+openssl&start=0&gws_rd=cr&gbv=1&lr=lang_en-GB&hl=en-GB&ei=x"

I get the complete webpage result and I can the search data:

<!doctype html><title>who is the lead developer of openssl - Google Search</title> ...

Why does searx not display the page and instead reports that it didn't find any results?
@unixfox
Copy link
Member

unixfox commented Nov 22, 2019 •
edited

Probably related to #1704 and #1596.

In summary Google is trying to push his new UI and Searx doesn't handle it, so I guess we will have to find another workaround to load the old UI or stop supporting Google until a new engine for google is developed.

EDIT: I tried to lower the Internet Explorer version to see if it would fallback to the old UI but it just doesn't want to, so I guess Google will be unusable until we have a new parser.

EDIT²: While searching for very old browsers, I found a working user agent that triggered the old Google Search UI: Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US) AppleWebKit/525.13 (KHTML, like Gecko) Version/3.0 Safari/525.13.
It's actually Safari 3.0 on Windows XP 😄.
You can see my workaround working on my Searx instance here: https://searx.be/
And if you want to test the workaround you just have to replace in the file searx/engines/google.py this line:

params['headers']['User-Agent'] = "Mozilla/5.0 (compatible; MSIE 9.0; Windows NT 6.1; Trident/5.0)"

with this one:

params['headers']['User-Agent'] = "Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US) AppleWebKit/525.13 (KHTML, like Gecko) Version/3.0 Safari/525.13"

@unixfox unixfox mentioned this issue Nov 22, 2019
Merged
@return42 return42 mentioned this issue Nov 23, 2019
Closed
@diegoroversi
Copy link

unifox fix works fine. Maybe we should made the useragent used for querying google a config parameter, so can be easily changed by the user.

@unixfox
Copy link
Member

unixfox commented Nov 24, 2019

Yeah why not but it's not that useful as a feature because most of the time if someone finds a new ideal user agent he will also know how to change the python code.
Moreover, by hardcoding the user agent we will be sure that if someone open a new issue related to Google the devs will know that in most case he is using the same hardcoded user agent and not a modified one.
In the end I don't really find the point of adding new code for that, I think it's better to focus on developing a new engine for Google Search than providing a way to fix a temporary workaround that will probably in one year or less stop working.

@lucky13820
Copy link
Contributor

@unixfox Hi, I used docker to setup my SearX instance, is there any way to change that python code with docker? Thanks.

@return42
Copy link
Contributor

return42 commented Nov 25, 2019 •
edited

@lucky13820 you can pull PRs, like I do. To pull PR #1749:

git remote add upstream https://github.com/asciimoo/searx.git
pull upstream refs/pull/1749/head

Here is another example where I pulled #1702 --> merge https://github.com/return42/searx/commit/dfe1f7752

@lucky13820
Copy link
Contributor

@return42 Thanks, I figured out.

@unixfox
Copy link
Member

unixfox commented Nov 25, 2019 •
edited

@lucky13820
You can use Docker volume and override the google.py by mounting it from the host filesystem.
If you are using docker-compose here is an example:

volumes: 
- './google.py:/usr/local/searx/searx/engines/google.py'

@unixfox
Copy link
Member

unixfox commented Dec 6, 2019

Google just forced the new UI for the old Safari user agent that got included in the patch.
I guess we will have to use a new parser for the new UI because I'm unable to find another user agent that could trigger the old UI again.

This was referenced Dec 6, 2019
Closed
Closed
@lucky13820
Copy link
Contributor

That workaround died fast. So should we open a new issue to track it? I am willing to help, but I don't know anything about Python. I only know HTML/CSS and some basic programming knowledge.

@virtadpt
Copy link

virtadpt commented Dec 6, 2019

Did anyone try one of the Lynx user agents to see what would happen? At this point it's worth a go.

@unixfox
Copy link
Member

unixfox commented Dec 6, 2019 •
edited

@virtadpt Yes I did here: #1749 (comment)

I bet there is no other user agent that could trigger again the old UI, that's why we need #1628 to get merged as soon as possible.

Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging a pull request may close this issue.

[fix] Force Google old UI with a new user agent
6 participants
@diegoroversi @return42 @virtadpt @unixfox @lucky13820 @ioogithub