[mod] by default allow only HTTPS, not HTTP

[dalf]

What does this PR do?

This PR disable HTTP by default, and allows only the HTTPS protocol.

The HTTP can be explicitly enable using enable_http: True.

Why is this change important?

The PR #2373 detects if engine use HTTP instead of HTTPS using the request function of each engine.

But some engine send HTTP(S) requests in the request function:

• searx/searx/engines/duckduckgo_images.py

  Line 40 in 0d8b369

  res = get(query_url, headers=headers)

• searx/searx/engines/seznam.py

  Line 26 in 0d8b369

  response_index = get(base_url, headers=params['headers'], raise_for_httperror=True)

Also, some other engines send additional request in the response function:

• searx/searx/engines/duckduckgo.py

  Line 92 in 0d8b369

  get(url_ping, headers=headers_ping)

• searx/searx/engines/pubmed.py

  Line 66 in 0d8b369

  search_results_xml = get(retrieve_url_encoded).content

This PR minimize the number of HTTP(S) requests each time searx starts (and tested too).

In addition it makes sure searx never sends HTTP request even if an engine sends more than one request (except if explicitly enabled).

How to test this PR locally?

• In settings.yml, set enable_http: False in the library genesis section.
• Search using this engine
• The response should be:
  
• the log:

ERROR:searx.search.processor.online:engine library genesis : requests exception(search duration : 0.08051466941833496 s, timeout: 7.0 s) : No connection adapters were found for 'http://libgen.rs/search.php?req=time'
Traceback (most recent call last):
  File "/home/alexandre/code/searx/searx/search/processors/online.py", line 143, in search
    search_results = self._search_basic(query, params)
  File "/home/alexandre/code/searx/searx/search/processors/online.py", line 123, in _search_basic
    response = self._send_http_request(params)
  File "/home/alexandre/code/searx/searx/search/processors/online.py", line 95, in _send_http_request
    response = req(params['url'], **request_args)
  File "/home/alexandre/code/searx/searx/poolrequests.py", line 209, in get
    return request('get', url, **kwargs)
  File "/home/alexandre/code/searx/searx/poolrequests.py", line 181, in request
    response = session.request(method=method, url=url, **kwargs)
  File "/home/alexandre/code/searx/local/py3/lib/python3.8/site-packages/requests/sessions.py", line 542, in request
    resp = self.send(prep, **send_kwargs)
  File "/home/alexandre/code/searx/local/py3/lib/python3.8/site-packages/requests/sessions.py", line 649, in send
    adapter = self.get_adapter(url=request.url)
  File "/home/alexandre/code/searx/local/py3/lib/python3.8/site-packages/requests/sessions.py", line 742, in get_adapter
    raise InvalidSchema("No connection adapters were found for {!r}".format(url))
requests.exceptions.InvalidSchema: No connection adapters were found for 'http://libgen.rs/search.php?req=time'

Author's checklist

Related issues

Related to

• Display if an engine does not support HTTPS requests #2373
• Display which engines don't use HTTPS #302 (comment)