Display which engines don't use HTTPS

[dalf]

The connection between the browser and searx can be encrypted, this visible on the browser.
But the connection between searx and a search engine can be in clear text without encryption. When it's possible searx uses https, but some search engines don't support HTTPS :

• invalid certificate (not self signed certificate)
• not HTTPS support

In other words, anyone on the wire can intercept the request.

Ideas :

• Display which search engines don't use HTTPS
• Have an option to disable these engines (a check box, something simple from the user point of view).

[pointhi]

probably we can improve that idea.

We could add an information node to every search-engine, which can contain some informations like:

• is this engine using https
• is this an open source engine (like yacy or nominatim)
• are we using the api
• short description about the engine
• website
• ...

Themes like oscar could display that informations using icons, or hover texts beside the allow/block button.

Based on that informations we could also create a plugin which automatically deactivate all non-https engines.

[Cqoicebordel]

I like that idea a lot !

[dalf]

One way to solve this privacy issue is to disable HTTP requests.

Can be implemented with a specific requests.adapters.HTTPAdapter that throws an exception when the get_connection method is called and use it in https://github.com/asciimoo/searx/blob/master/searx/poolrequests.py

This behavior could be enable with an user setting.

[dalf]

To bring back this issue from the dead, one idea could :

# to import
# results can be harmful, filter are not implemented
HARMFUL_CONTENT='harmful'
# results can be harmful, there is support for content filtering
SAFESEARCH_SUPPORT='support'
# results are for sure harmless
HARMLESS_CONTENT='harmless'

# for each engine
# description .. describes the engine
description={
  'categories': ['general'],
  'language_support': True,
  # should be safesearch_support ?
  'safesearch': SAFESEARCH_HARMFUL,
  # should be time_range_support ?
  'time_range': True,
  # should be paging_support ?
  'paging': True,
  # 'language': 'en',
  # informative
  'use_api': False,
  # forbid HTTP connection for this engine
  'allow_http': False,
  #how the user can visit the engine without using searx
  'url': 'https://www.bing.com/',
  'name': 'Bing',
}

[asciimoo]

@dalf i really like this idea. It would be nice to add these fields too:

• session_required (wolfram alpha, soundcloud, etc)
• multiple_requests (wikidata)
• special_search_syntax or query_example (e.g. currency convert has the format [Amount] [currency1] in [currency2]: 2 eur in huf)

[dalf]

• multiple_requests can be automatically updated
• special_search_syntax and query_example : 👍 related to [mod] fix #903 : add a function is_accepted to engine. #992 (regex or a method ?)

How about settings.yml ?

bing:
   description.use_api: True

?

[dalf]

This overly complicated over engineered script may help this issue: https://gist.github.com/dalf/3c3904699153a741f27842d8ea30b449
It parse, modifies and write all the engine source codes.

It still requires a manual checks of these fields (all False by default):

• allow_http
• session_required
• multiple_requests
• require_api_key

Sample output:

"""
 OpenStreetMap (Map)

 @website     https://openstreetmap.org/
 @provide-api yes (http://wiki.openstreetmap.org/wiki/Nominatim)

 @using-api   yes
 @results     JSON
 @stable      yes
 @parse       url, title
"""

from json import loads

# features
features = {
    "categories": ['map'],
    "paging": False,
    "language": False,
    "time_range": False,
    "safesearch": False,
    "allow_http": False,
    "session_required": False,
    "multiple_requests": False,
}

# metadata
metadata = {
    "website": "https://openstreetmap.org/",
    "use_api": True,
    "require_api_key": False,
}

# search-url
base_url = 'https://nominatim.openstreetmap.org/'
search_string = 'search/{query}?format=json&polygon_geojson=1&addressdetails=1'
result_base_url = 'https://openstreetmap.org/{osm_type}/{osm_id}'
...

[return42]

Correct me if I am wrong .. since f407dd8 we do not have any more http engines.

I close the issue, if I am wrong or overseen something, ask for reopen / thanks!

[dalf]

$ find -name '*.py' -exec -H "http:" {} \; | less

(okay = the path / reponds)

https://export.arxiv.org/ okay:
https://github.com/asciimoo/searx/blob/f0684a5bb5860c2b9caffefb47dc55781092819e/searx/engines/arxiv.py#L22

https okay:
https://github.com/asciimoo/searx/blob/f0684a5bb5860c2b9caffefb47dc55781092819e/searx/engines/gigablast.py#L60

http redirec to https://musicbrainz.org/ :
https://github.com/asciimoo/searx/blob/f0684a5bb5860c2b9caffefb47dc55781092819e/searx/engines/wikidata.py#L153-L156

https://www.acgsou.com/ okay :
https://github.com/asciimoo/searx/blob/f0684a5bb5860c2b9caffefb47dc55781092819e/searx/engines/acgsou.py#L22
https://github.com/asciimoo/searx/blob/f0684a5bb5860c2b9caffefb47dc55781092819e/searx/engines/acgsou.py#L44

http://api.mymemory.translated okay:
https://github.com/asciimoo/searx/blob/f0684a5bb5860c2b9caffefb47dc55781092819e/searx/engines/translated.py#L19-L20

Not working (redirect to https://wiki.dbpedia.org/lookup/ ):
https://github.com/asciimoo/searx/blob/f0684a5bb5860c2b9caffefb47dc55781092819e/searx/autocomplete.py#L119

Not working (redirect to https://seekstorm.com/ ):
https://github.com/asciimoo/searx/blob/f0684a5bb5860c2b9caffefb47dc55781092819e/searx/engines/faroo.py#L25

Most probably it is easier to upgrade all URL to https.

After that, perhaps it's time to forbid http connections ( https://github.com/asciimoo/searx/blob/master/searx/poolrequests.py#L69 )