Google rate limit kicking in

[mttkay]

While issuing a query, the app failed with the following error:

⭐  how2 -l android launch activity in new task                                                      
-You are doing too many requests to Google. You need to wait a bit before trying again.`

If the app is using an API key shared across all users, it might be hitting a rate limit earlier than expected?

[danyshaanan]

I think that there isn't a shared API key. This might be caused by someone in your network running lots of google api calls. (This probably makes more sense if you are in an office rather than at home).

[vasergen]

Have the same problem, after playing few minutes I always got an error "You are doing too many requests to Google. You need to wait a bit before trying again"

[mkohlmyr]

1. The how2 module does not use the Google Search API as it (to the best of my knowlege) has been deprecated in favour of Google Custom Search, which does not suit the needs of the project, not least of all because it imposes severe rate limits.
2. The error actually stems from node-google, the dependency how2 uses for scraping google results, which (as it should) respects the rate limiting Google imposes on what it believes to be (correctly in this case) automated traffic. This appears to be in the neighbourhood of four queries per minute.
3. A realistic (and useful!) fix for this issue would probably involve writing npm modules that implement the same interface as node-google (or a consistent adapter for existing modules) but for scraping bing and duck-duck-go. If searches were then scheduled round-robin from how2, it could perhaps triple the rate limit (depending on their respective limits). If there is sufficient interest in that solution, I'd be happy to take a look at it.

Hope that helps!

[nicandris]

I didn't even managed to use this once. I always get the rate limit msg

[santinic]

Yeah we need someone to write an api to scrape bing or duckduckgo

[mkohlmyr]

I'm happy to take a look at implementing that in the coming week, if you'd like.

[santinic]

@mkohlmyr awesome

[mkohlmyr]

Pretty sure I have ddg / bing / google basically working with an -e flag for search engine selection. If it sounds interesting I could also add support for an environment variable to set the default.

Let me know if you want any changes to https://github.com/mkohlmyr/how2/commit/4b0cf93f9e87634f125d28c658a313a7e1517e23 before I make a PR. Forgot about the tests!

https://github.com/mkohlmyr/how2/commit/4b0cf93f9e87634f125d28c658a313a7e1517e23
+
https://github.com/mkohlmyr/how2/commit/db290793ddf01d2e6729b566a387d82445cf1f53

Let me know if you want any changes.

[mkohlmyr]

Ping @santinic re commits above 📢

Happy to squash them if you feel that's cleaner, or if you want me to rename / restructure any of the changes.

[santinic]

@mkohlmyr sorry for the delay. It looks really good ! I have two questions:

• I seem to get fewer answers with google compared with the node-google, do you know why?
• With bing, I get this:

|Cannot connect to search engine.
Error: Failed to locate valid search results on page for query: [object Object]```

But, it's amazing :) thanks for your work!

[mkohlmyr]

• You're right, I didn't include the num querystring parameter for the google URL. What number would you like? node-google sets a default of 10 and a max of 100 I believe.
• I will take a look at that over the next few days, I did notice sometimes bing struggled to return answers for some queries.

I'll hit you back up by the end of the weekend probably with some updates.

[mkohlmyr]

So I have done some investigating.

Your error is probably a rate limiting page that I was unaware of. It would be easy to improve the printing of the parameters, I could also create a special error message for this case by looking at the source it brings back.

FWIW it also looks like Bing has a tendency to return questions that have been closed by the user, moderator or have no answers, all of which would cause the magic search to reply with "cannot find any reasonable answer for your query". e.g. how2 reverse proxy nginx -e bing.

I have not had time to implement the changes I mentioned, but I will hopefully get to it later this week (after wednesday). If you don't want to wait, feel free to adapt the code I posted!