Add support for Search Providers

[Jpxe]

Another redirect idea 😄
Edit: updated with search providers

Potential search providers:

These are all privacy focused.

• Searx
• DuckDuckGo
• Qwant
• MetaGer
• Swisscows
• Mojeek
• YaCy
• Startpage.com

Found on: privacytools.io & restoreprivacy.com

SearX

Searx is a privacy-respecting metasearch engine. Google Searches could redirect here (and potentially Bing, Yahoo, Yandex etc).

Source: GitHub
Documentation: doc
Instances: searx.space

There are also Meta-searx instances ("These are websites that source from other searx instances. These are useful if you can't decide which Searx instance to use"). More info is available on searx.space - click "About".

[Jpxe]

Some code that might help: https://github.com/allanburleson/duckduckgo-redirect (for DuckDuckGo, couldn't find any for Searx)

[SimonBrazell]

Do you think this would be worth the effort though, given most browsers support changing you search provider internally anyway?

[Jpxe]

Might not be. I'm thinking this would be useful for when you are on a non-google website and click on a Google search link. I'm not sure how many Google search URLs people come across. For me I guess it's more common than Gmaps embeds at least.

One somewhat common use case would be websites that use google as their site-specific search with a Google search box. I believe this is called Google Custom Search. Here's an example of this: https://phoronix.com

[SimonBrazell]

I don't know how common random Google search links are in the wild but I have certainly come across custom searches, can even think of this one I've used recently to test the feature on.

I've not had great experience with Searx though, maybe I wasn't using it right but I remember getting some errors rather than search results, tend to use DDG lately. Do you get reliable results using it?

[Jpxe]

I've used it from time to time, I currently use DDG. The problem seems to be that you have to change instance a lot if you want Google/Bing/Startpage results to be included, I'm guessing it's because they block them. The instance I used which worked great most of the time was https://search.privacytools.io/ (no google results at the moment). At the moment searx.info seems to be the best.

Another option could be to have DDG as an standard/optional instance

[SimonBrazell]

Changed the title to be more generic as whatever solution I go with will be search engine agnostic, allowing users to selected whichever provider they like.

[MalteKiefer]

Any update on this here?

[SimonBrazell]

No sorry, been focusing on other stuff atm, but I've left it open as I do still want to do this, I think initially just supporting searX as you suggest, but needs some investigation.

[tiotrom]

I can say SearX is amazing. One instance I use for the past months is https://searx.be/. Super fast. Much better results than any search engine. Also, there are many SearX instances. It really works. We even use it as the default search engine for our tromjaro.com Linux OS. I hope you consider adding it as a replacement for google. DDG is full of ads and goes down the path of google as I see it. SearX combines the searches from many sources and disables ads and trackers.

[SimonBrazell]

Hi @tiotrom I might give it another go, been liking DDG less and less lately, too often I find myself using the Google bang because I can't find the results I want, which defeats the purpose.

[tiotrom]

Could be a fantastic addition. We are testing this extension to see if it works well so maybe we add it as default to TROMjaro OS. We created a "trade-free" OS, meaning one where you should not trade your attention (ads), data, or currency to whatever you use. And since most people use google, instagram, youtube and the like, your extension can make it easier for that switch to a "trade-free" alternative that doesn't mine your data and such. So far we see that bibliogram doesn't really work, and yes we can change the instance manually but maybe would be a great idea to detect when an instance doesn't work and automagially switch to another one. Great job with this extension btw!

[SimonBrazell]

Sounds like a noble cause, there is a feature request to make this extension more outage resistant, similar to what you suggest, I haven't investigated a solution yet though, need to prioritise and start working through all these feature requests...

[tiotrom]

We have our own instance of SearX at https://tromland.org/searx/ and our friends from Nixnet.services searx.nixnet.services and Snopyta.org search.snopyta.org have their own too. We are extensively testing these instances. So far they work wonderfully. So yah, would be fantastic to add that option too.

[tiotrom]

I just have to insist with this :D - redirection to SearX is a wonderful idea. And there are so many SearX instances https://searx.space/ - I beg you add this one :). We have been testing this for a while now. We do research for books, we look for IT "fixes" all the time, media files, you name it. SearX works really great. :D

[tiotrom]

Here's how some are using SearX random instance for search https://searx.neocities.org/ and https://searx.neocities.org/changelog.html - could be helpful!

[SimonBrazell]

Got an email this morning from a developer asking for my help with their own extension based on this one that implements a search engine redirect, I submitted some code changes to get it working on Firefox, also asked them to consider creating a pull request to add this feature to Privacy Redirect.

[tiotrom]

Oh wow super cool! What I would recommend tho is to use SearX instances only, since the other ones like startpage or duckduckgo insert ads into the search results, making people click biased search results. in that sense they are not different from google. SearX has none of that.

[SimonBrazell]

Perhaps I'll make SearX the default then and leave the others there for the user to select if they wish.

[tiotrom]

Ok but I think when you insert ads into search results it is simply deceiving the users to click some websites. SearX uses all of the other search engines so I am not sure why you would need them...SearX uses DuckDuckGo, Startpage, Google, Bing and a lot more...and works amazingly great. We use them daily for our work. I personally suggest only leaving the SearX instances.

[SimonBrazell]

Good point, I more just wanted to leave them there so that people know they work if they happen to not want to use SearX for some reason.

[tiotrom]

I would be curious why they would not want to use SearX :D - but ofc, as you prefer. Just saying we run a SearX instance ourselves and for our custom operating system TROMjaro and all works super great. Plus, so many SearX instances out there...

[SimonBrazell]

Yeah I'm not sure why either but an example of this happening is available in the submitted source code, where the developer has given the following options for redirects:

  { link: "https://duckduckgo.com", q: "/" },
  { link: "https://startpage.com", q: "/search/" },
  { link: "https://www.qwant.com", q: "/" },
  { link: "https://www.mojeek.com", q: "/search" },

SearX isn't included in the options at all.

[SimonBrazell]

Can you provide a shortlist of instances to include?

[tiotrom]

I think people do not use SearX since changing your default search engine may be an inconvenient thing to do. I do research online for books and documentaries so I search a lot: from images to videos, news websites or research papers. All works fine really.

For a list, we use our own instance mainly https://tromland.org/searx/ and these are reliable too from our tests:

• https://search.snopyta.org/
• https://searx.info/
• https://searx.be/
• https://search.disroot.org/
• https://searx.tuxcloud.net/
• https://searx.ninja/
  Plus one that chooses at random from searx.space https://searx.neocities.org/

[SimonBrazell]

Good stuff, I'll include these in the list of suggestions and to simplify things I will probably not include the other providers as it's easier to keep the random pool list the same and I don't want to be redirecting people to providers randomly that may deceive users. Thanks for the heads up.

[tiotrom]

Awesome! This is super great that you include SearX! :) I am so happy!

[SimonBrazell]

The PR still needs some work though, so I'll need to spend some time on it when I next have some spare, but hopefully I'll get it working and released in the next couple of days.

[tiotrom]

You're awesome! Such an awesome extension! As I said we use it in TROMjaro as default!

[SimonBrazell]

Available in v1.1.44.

Thanks @ToughGuyKunio for the suggeston and @aapl-yumi for the contribution.

[tiotrom]

I am a bit sad that you did not include SearX which is the most private of those options and the most diverse and reliable. Was it difficult to integrate?

DuckDuckGo - fools people into clicking sponsored links (adverts) so it distorts people's views, similar to how facebook does. I also don't think their code is open source. Plus, since they make a profit out of ads, they may go down the slippery slope and in the future we may find out that they indeed tracked you and such. Remember the Brave Browser fiasco? Very "private" browser, injecting their affiliate links when people clicked certain urls.

Startpage - as far as I know it is owned by an ad-company...and as above, they inject ads into searches.

Qwant - same story as above, they insert ads.

Mojeek - good choice, they do not seem to inject ads.

SearX is all of the above, combined, without any ads, trackers, and so forth. The same way that you use Invidious for youtube, and not other platforms like dtube or lbry that are ad-based, you should use SearX if you want full privacy, since SearX has all that you need. SearX works 50 times better than Nitter, or Invidious, or Bibliogram. Meaning it is far more reliable. Just saying...

I don't want to bother you with this, but I do not understand why you didn't choose SearX as the only alternative, since there are so many instances out there. One is ours https://tromland.org/searx/

:(

[SimonBrazell]

Oh sorry, I can add SearX, I just merged what @aapl-yumi submitted and added any functionality that was missing, didn't touch the instances.

[SimonBrazell]

Found an issue with the latest build anyway so I'll add some SearX instances while I fix it.

[tiotrom]

Thanks a lot!

My only question now is, are the other ones relevant to keep? Since SearX uses duckduckgo, startpage, google, qwant or even mojeek + yahoo, bing and a lot more. Am thinking could even be easier from a code perspective + overall functionality since Privacy Redirect uses "instances" mostly, like nitter, or bibliogram and so forth. Anyways, up to you, and kudos for adding SearX. 👍

[seniorm0ment]

Did you implement the ability to handle Searx preferences across different Searx instances? Doesn't seem to be the case.
Is this something that could be looked into?

(Worth noting SearX instances can set different engines to support on a per instance basis)