-
Notifications
You must be signed in to change notification settings - Fork 1.7k
Scraping Google in the client side #1608
Comments
This seems to be quite difficult to implement and probably out of the scope of Searx. It would require to when the page load, the client do a request with javascript to fetch the Google results then generate a new HTML code and finally inject it into the page. I'm pretty sure it would add quite a lot of delay and complexity. The main issue is not the captcha of Google but it's bots that abuse the public Searx instances. If we get rid of these bots we wouldn't have any issue with Google like I already do on my instance : https://searx.be |
@unixfox |
@eggercomputerchaos I talked about it here: #1584 (comment) and here: #1034 (comment) |
@unixfox |
Would it be possible just solve the captcha on the client side? |
This code will scrape results from Google in client
|
@agnelvishal |
The integration is yet to be implemented |
ok, but in which file can I insert the code, or is it more complicated than I think? |
Where is the sense in? .. parsing in the WEB-client? |
My experience is that just a group of more than one user can result in CAPTCHA questions or even one havey user. Today it hits me at my instance https://darmarit.cloud/searx/ and I checked the apache log: I was the only user, no bots at this time (today). |
Update: now old google UI is completely dead #1748 (comment) |
This issue isn't really about the old UI but about parsing the (new) UI from the client side. |
Aaargh, my fail .. so many open issues / I got confused. Anyway parsing in the client is nonsense / the solution for the new UI is #1628 / or did I missed something .. again :) |
If scraping is done in the client side, that will prevent bot abuse. Because the bot will have to send request to Google search etc. |
@agnelvishal Is it possible to do it in the client side? I'm not sure whether there are same-origin policies limiting it. |
@agnelvishal we already know the benefits of parsing using the client side, I completely agree with you that it would be really great for Searx instances. @asciimoo @return42 I guess you can close this issue because it's technically not possible to do what have been requested. |
Since Google introduces captcha when there are too many requests from an IP ( #729), it might be better to scrape Google in the client side. This will also decrease server load.
The text was updated successfully, but these errors were encountered: