Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add pagination for Yep #1619

Closed
Krando opened this issue Aug 5, 2022 · 0 comments · Fixed by #1623
Closed

Add pagination for Yep #1619

Krando opened this issue Aug 5, 2022 · 0 comments · Fixed by #1623
Labels
area: engine new feature New feature or request

Comments

@Krando
Copy link

Krando commented Aug 5, 2022

Is your feature request related to a problem? Please describe.
After Bing, Yandex and Google, Yep is in my opinion the best independent search engine. Mojeek's results may sometimes be better ordered given certain queries but Yep's index is no doubt larger as it is powered by the data of Ahrefs which is one of the biggest SEO companies in the world and thus Yep has a huge backlink index. Engines like Mojeek and Gigablast take a lot of time to get the latest results but Yep fetches/indexes them pretty quickly. For copyrighted works or results that would normally be censored at Google or even Yandex, Yep is really great. Unfortunately, we only get around 10 results or something for most queries on SearXNG with Yep and the tremendous potential of this engine is therefore neglected. Adding pagination to Yep would improve SearXNG by a lot and you would notice it, believe me. Especially if you use Yep with the "query in the page's title" parameter enabled which helps sort out content unrelated to your search query.

Describe the solution you'd like
The next set of results is attained by clicking the "Show more" button. Yep uses some javascript trickery to fetch the results in a roundabout way using the API. You might be surprised to learn, however, that we can get 100 results in a single shot through a link as follows:

https://api.yep.com/fs/1/?client=web&limit=100&no_correct=false&q=searxng&type=web

If you open this click in Firefox, the browser will order the JSON source code for you with all the 100 links. If we set the 'limit' parameter to 10, we may get 10 results, if we set it to '20', we may get 20. And we can keep doing this until we reach 100 and then we won't see more results than 100 at that point. But if we use the "show more" button on the website, sometimes it's possible to get a lot more results than 100 but it depends. If there's a delay between each time we click the button, we can sometimes get more results than 100 but this is becoming increasingly rare according to my experience, perhaps someone else can confirm or check.

Nonetheless, even 100 results from Yep would be a great improvement to SearXNG and this should be added as soon as possible.

Here is the javascript file which the API uses for fetching the results when we click on the "show more button": https://cdn.yep.com/en/assets/js/62.1c84b76d.js

Describe alternatives you've considered
Nothing.

Additional context
Perhaps some Javascript master can come up with a creative trick or a solution to fetch more than 100 results using the API...

@return42 return42 added new feature New feature or request area: engine labels Aug 6, 2022
return42 added a commit to return42/searxng that referenced this issue Aug 6, 2022
yep.com is still in beta, the api.yep.com does not have paging support.  There
is only a 'limit' argument with a maximum of 100 results.

yep.com seems fast; there is nor need for a timeout of 12 sec.

The API returns JSON nevertheless what the HTTP header is, the "show more"
button on yep.com's web site does not set a special HTTP Accept header.

FYI: The index does not support languages, the WEB UI does not offer a language
selection of the results and the entire index seems in English.

Closes: searxng#1619
Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
area: engine new feature New feature or request
Projects
None yet
Development

Successfully merging a pull request may close this issue.

2 participants