LibreSSL is lacking host and IP verify parameters

[tiran]

LibreSSL is not OpenSSL 1.0.2 compatible. It's lacking X509 verify parameters to verify hostname or IP address during TLS handshake. Python 3.7 will require a fully OpenSSL 1.0.2 compatible API and the X509 verify parameter functions to work.

• X509_VERIFY_PARAM_set1_host
• X509_VERIFY_PARAM_set1_ip
• X509_VERIFY_PARAM_set1_ip_asc
• X509_VERIFY_PARAM_set_hostflags

https://www.openssl.org/docs/man1.0.2/crypto/X509_VERIFY_PARAM_set1_host.html

PS: You may want to start introducing OpenSSL 1.1.0 compatibility, too. Either Python 3.8 or 3.9 will require a fully OpenSSL 1.1 compatible API.

[4a6f656c]

Thanks for the heads up - LibreSSL was forked from OpenSSL 1.0.1 and we do not guarantee that we will fully implement later OpenSSL APIs. That said, the OpenSSL 1.0.2 functions that you mention have come up before and we'll take another look at those.

Re OpenSSL 1.1.x and Python 3.8/3.9, are you suggesting that Python is planning on dropping support for OpenSSL 1.0.x even though there is still no LTS release of OpenSSL 1.1 (the current release is only supported until 2018-08-31)? Presumably this is also likely to heading towards December 2019 (3.8) or June 2021 (3.9)?

[tiran]

CPython development has very little resources to work on ssl. Our current approach to verify hostnames and IP addresses has multiple issues. X509_VERIFY_PARAM_set1_host and related functions solve these issues with very little effort. I'm currently discussion whether it's more important to keep compatibility with LibreSSL or to make Python's TLS support more secure and standard conform.

OpenSSL 1.0.2 reaches EOL about the same time 3.9.0 is scheduled to release. We haven't decided a release date yet, the final will probably be released between December 2019 and February 2020. OpenSSL 1.0.2 supports ends 2019-12-31. Unless there are very compelling reasons, it doesn't make sense to hold back features and increase our maintenance burden.

I'm willing to keep the basic features of Python's ssl module compatible with LibreSSL -- if and only if LibreSSL is kept fully compatible with required OpenSSL APIs. Hostname verification is such a feature. Some advanced features like env var are already documented as not supported when Python is compiled with LibreSSL.

Could you please clarify your backwards compatibility statement on https://www.libressl.org/ ? It says

libssl: a TLS library, backwards-compatible with OpenSSL

but that's no longer true. LibreSSL's libssl is no longer compatible with any supported OpenSSL version.

[tiran]

After some discussion on the Python core developer list, I have decided to require OpenSSL 1.0.2 APIs. Every core developer in the discussion was in favor of using the new APIs. Python 3.7.0 beta1 will not by compatible with OpenSSL <= 1.0.1 or LibreSSL <= 2.6.4. Incompatibility with LibreSSL is only temporarily. I have implemented additional autoconf rules to test for required features and I'll keep all LibreSSL quirks. Python 3.7 will regain compatibility with LibreSSL as soon as you release a version which implements all required features.

OpenSSL recommends the API

The X509_VERIFY_PARAM_set1_host is OpenSSL's preferred way to validate host names. OpenSSL strongly recommends the API. From its man page

X509_VERIFY_PARAM_set1_host() sets the expected DNS hostname to name clearing any previously specified host name or names. If name is NULL, or empty the list of hostnames is cleared, and name checks are not performed on the peer certificate. If name is NUL-terminated, namelen may be zero, otherwise namelen must be set to the length of name. When a hostname is specified, certificate verification automatically invokes X509_check_host with flags equal to the flags argument given to X509_VERIFY_PARAM_set_hostflags() (default zero). Applications are strongly advised to use this interface in preference to explicitly calling X509_check_host, hostname checks are out of scope with the DANE-EE(3) certificate usage, and the internal check will be suppressed as appropriate when DANE support is added to OpenSSL.

X509_VERIFY_PARAM_set1_host resolves several Python issues

Python had multiple security issues just caused by incorrect or
buggy hostname matching. X509_VERIFY_PARAM_set1_host simplifies our code and resolves all outstanding issues.

• Until Python 3.2 and 2.7.9, the ssl module was not capable of
  performing host name matching. ssl.match_hostname() was
  introduced in 3.2.0 and later back-ported to 2.7.9.
• Issue #12000: Subject CN was ignored when a subject alternative
  name extension (SAN) was present without dNSName entries, thus
  violating RFC 2818.
• CVE-2013-2099: Multiple wildcard characters could be abused
  for Denial-of-Service attack in the re module.
• Issue #17997: RFC 2818 was superseded by RFC 6125, which no longer
  allows multiple wildcard characters. Wildcards are only supported
  in the left-most label.
• Issue #17997: ssl.match_hostname() did not implement partial
  wildcards of international domain names correctly.
• Issue #18709: The ssl module used an inappropriate OpenSSL function
  to convert host names from ASN.1 to strings. A host name with an
  embedded NULL byte could be abused to trick validation.
• Issue #17305: The ssl module does not handle IDNA 2008-encoded
  host names correctly. It converts from IDN A-label (ASCII
  compatible encoding) to IDN U-label (unicode) with Python's idna
  encoding, which is IDNA 2003-only.
• Issue #30141: The host name is not verified when a SSLSocket is
  created with do_handshake_on_connect=False and the application
  causes an implicit handshake w/o calling do_handshake() explicitly.
• A SSLSocket performs host name matching after the handshake and
  during the handshake. In case of an invalid host name, a client
  is suppose to abort the connection with appropriate TLS alert.
  This causes two problem. For one the server is not informed about
  a problem with the certificate. Also an invalid host name does not
  prevent the client from sending a TLS client authentication
  cert to a malicious server. The cert typically contains personal
  information like username and department.
• The server is not informed that the client has rejected the certificate.
  Standard-conform clients send a TLS ALERT with alert type BAD CERTIFICATE.

[yan12125]

@4a6f656c:

Looks like server name verification in LibreSSL is handled by libtls and servername is set via tls_connect_servername? It might be better to move SAN and CN handling (tls_check_name) from libtls to libssl so that users (e.g. CPython) can utilize it via existing SSL contexts instead of another tls context.

[4a6f656c]

@yan12125 libtls is a completely different beast - it has hostname verification enabled and required by default. You have to go out of your way to disable it, if you really want to do that. In comparison, as far as I know even with OpenSSL 1.1 you still have to explicitly request and enable hostname verification.

@tiran for a long time we've provided the X509_check_* functionality - it is just the X509_VERIFY_PARAM_* functions that were not currently available. While OpenSSL "strongly advise" to use this interface, there is no reason that code cannot implement the same functionality via the existing API (and I fully understand Python wanting to remove its own verification code) (and there appears to be about twice an many pieces of software using the X509_check_* interface than the X509_VERIFY_PARAM_* one).

I have just exposed the requested X509_VERIFY_PARAM_* functions, along with some others that are in the OpenSSL 1.0.2 API. If there is further functionality that you need, please let us know.

[Desertgeek]

I just ran into this problem after installing python3.9.1 on my Coral Dev Board. i created a virtual environment but now PIP doesn't work; I can't even upgrade it and I can't install any packages.
I got this message when building Python3.9.1:
"Could not build the ssl module!
Python requires an OpenSSL 1.0.2 or 1.1 compatible libssl with X509_VERIFY_PARAM_set1_host().
LibreSSL 2.6.4 and earlier do not provide the necessary APIs, #381"

Then after installing and creating the venv, I get this when trying to add packages:
"(.venv) mendel@xenial-tang:~$ python3.9 -m pip install --upgrade pip
WARNING: pip is configured with locations that require TLS/SSL, however the ssl module in Python is not available.
WARNING: Retrying (Retry(total=4, connect=None, read=None, redirect=None, status=None)) after connection broken by 'SSLError("Can't connect to HTTPS URL because the SSL module is not available.")': /simple/pip/
WARNING: Retrying (Retry(total=3, connect=None, read=None, redirect=None, status=None)) after connection broken by 'SSLError("Can't connect to HTTPS URL because the SSL module is not available.")': /simple/pip/
WARNING: Retrying (Retry(total=2, connect=None, read=None, redirect=None, status=None)) after connection broken by 'SSLError("Can't connect to HTTPS URL because the SSL module is not available.")': /simple/pip/
WARNING: Retrying (Retry(total=1, connect=None, read=None, redirect=None, status=None)) after connection broken by 'SSLError("Can't connect to HTTPS URL because the SSL module is not available.")': /simple/pip/
WARNING: Retrying (Retry(total=0, connect=None, read=None, redirect=None, status=None)) after connection broken by 'SSLError("Can't connect to HTTPS URL because the SSL module is not available.")': /simple/pip/
Could not fetch URL https://pypi.org/simple/pip/: There was a problem confirming the ssl certificate: HTTPSConnectionPool(host='pypi.org', port=443): Max retries exceeded with url: /simple/pip/ (Caused by SSLError("Can't connect to HTTPS URL because the SSL module is not available.")) - skipping
Requirement already up-to-date: pip in ./.venv/lib/python3.9/site-packages (20.2.3)
WARNING: You are using pip version 20.2.3; however, version 22.2.2 is available.
You should consider upgrading via the '/home/mendel/.venv/bin/python3.9 -m pip install --upgrade pip' command."

and:

"(.venv) mendel@xenial-tang:~$ pip install opencv-contrib-python
WARNING: pip is configured with locations that require TLS/SSL, however the ssl module in Python is not available.
WARNING: Retrying (Retry(total=4, connect=None, read=None, redirect=None, status=None)) after connection broken by 'SSLError("Can't connect to HTTPS URL because the SSL module is not available.")': /simple/opencv-contrib-python/
WARNING: Retrying (Retry(total=3, connect=None, read=None, redirect=None, status=None)) after connection broken by 'SSLError("Can't connect to HTTPS URL because the SSL module is not available.")': /simple/opencv-contrib-python/
WARNING: Retrying (Retry(total=2, connect=None, read=None, redirect=None, status=None)) after connection broken by 'SSLError("Can't connect to HTTPS URL because the SSL module is not available.")': /simple/opencv-contrib-python/
WARNING: Retrying (Retry(total=1, connect=None, read=None, redirect=None, status=None)) after connection broken by 'SSLError("Can't connect to HTTPS URL because the SSL module is not available.")': /simple/opencv-contrib-python/
WARNING: Retrying (Retry(total=0, connect=None, read=None, redirect=None, status=None)) after connection broken by 'SSLError("Can't connect to HTTPS URL because the SSL module is not available.")': /simple/opencv-contrib-python/
Could not fetch URL https://pypi.org/simple/opencv-contrib-python/: There was a problem confirming the ssl certificate: HTTPSConnectionPool(host='pypi.org', port=443): Max retries exceeded with url: /simple/opencv-contrib-python/ (Caused by SSLError("Can't connect to HTTPS URL because the SSL module is not available.")) - skipping
ERROR: Could not find a version that satisfies the requirement opencv-contrib-python (from versions: none)
ERROR: No matching distribution found for opencv-contrib-python"

Is there a fix for this?