hashlib: OpenSSL hash detection should obey security policy

[tiran]

BPO	40695
Nosy	@gpshead, @tiran, @ned-deily, @miss-islington
PRs	bpo-40695: Limit hashlib builtin hash fallback #20259 [3.9] bpo-40695: Limit hashlib builtin hash fallback (GH-20259) #20377

^{Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.}

Show more details

GitHub fields:

assignee = 'https://github.com/tiran'
closed_at = <Date 2021-05-22.09:17:30.971>
created_at = <Date 2020-05-20.08:07:57.604>
labels = ['type-bug', 'library', '3.9', '3.10']
title = 'hashlib: OpenSSL hash detection should obey security policy'
updated_at = <Date 2021-05-22.09:17:30.971>
user = 'https://github.com/tiran'

bugs.python.org fields:

activity = <Date 2021-05-22.09:17:30.971>
actor = 'christian.heimes'
assignee = 'christian.heimes'
closed = True
closed_date = <Date 2021-05-22.09:17:30.971>
closer = 'christian.heimes'
components = ['Library (Lib)']
creation = <Date 2020-05-20.08:07:57.604>
creator = 'christian.heimes'
dependencies = []
files = []
hgrepos = []
issue_num = 40695
keywords = ['patch']
message_count = 5.0
messages = ['369428', '369861', '369863', '394167', '394180']
nosy_count = 4.0
nosy_names = ['gregory.p.smith', 'christian.heimes', 'ned.deily', 'miss-islington']
pr_nums = ['20259', '20377']
priority = 'normal'
resolution = 'fixed'
stage = 'resolved'
status = 'closed'
superseder = None
type = 'behavior'
url = 'https://bugs.python.org/issue40695'
versions = ['Python 3.9', 'Python 3.10']

[tiran]

The hashlib module prefers hash implementations from OpenSSL. In case OpenSSL is not available or OpenSSL does not provide a hash algorithm, hashlib falls back to builtin implementations for MD5, SHA1, SHA2 family, SHA3/SHAKE family, and Blake2. The __get_openssl_constructor [1] function checks OpenSSL by retrieving the constructor and calling it. The calls fails if OpenSSL doesn't implement the EVP digest.

It also fails when the EVP digest is available but blocked by a security policy. In this case it falls back to the builtin implementation. If the builtin implementation has been removed by the package builder or --with-builtin-hashlib-hashes, then Python considers the hash algorithm as broken.

I propose to change the detection code so that Python uses OpenSSL implementation although it's blocked by the current system policy.

Current behavior:

$ rpm -qa openssl
openssl-1.1.1g-1.fc32.x86_64
$ /configure -C --with-builtin-hashlib-hashes=blake2
$ make -j4
$ ./python
>>> import hashlib
ERROR:root:code for hash md5 was not found.
Traceback (most recent call last):
  File "/root/cpython/Lib/hashlib.py", line 131, in __get_openssl_constructor
    f()
ValueError: [digital envelope routines: EVP_DigestInit_ex] disabled for FIPS

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/root/cpython/Lib/hashlib.py", line 251, in <module>
    globals()[__func_name] = __get_hash(__func_name)
  File "/root/cpython/Lib/hashlib.py", line 135, in __get_openssl_constructor
    return __get_builtin_constructor(name)
  File "/root/cpython/Lib/hashlib.py", line 118, in __get_builtin_constructor
    raise ValueError('unsupported hash type ' + name)
ValueError: unsupported hash type md5
>>> hashlib.md5()
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
AttributeError: module 'hashlib' has no attribute 'md5'

Proposed behavior:

$ ./python
>>> import hashlib
>>> hashlib.md5()
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
ValueError: [digital envelope routines: EVP_DigestInit_ex] disabled for FIPS
>>> hashlib.md5(usedforsecurity=False)
<md5 _hashlib.HASH object @ 0x7fb9d44b9b30>

Related issue:

bpo-9216 added the new hash constructor argument "usedforsecurity".
bpo-40637 added a new configure option --with-builtin-hashlib-hashes

[1]

cpython/Lib/hashlib.py

Lines 121 to 135 in 97fe9cf

	def __get_openssl_constructor(name):
	if name in __block_openssl_constructor:
	# Prefer our blake2 and sha3 implementation.
	return __get_builtin_constructor(name)
	try:
	# MD5, SHA1, and SHA2 are in all supported OpenSSL versions
	# SHA3/shake are available in OpenSSL 1.1.1+
	f = getattr(_hashlib, 'openssl_' + name)
	# Allow the C module to raise ValueError. The function will be
	# defined but the hash not actually available thanks to OpenSSL.
	f()
	# Use the C function directly (very fast)
	return f
	except (AttributeError, ValueError):
	return __get_builtin_constructor(name)

[miss-islington]

New changeset 4cc2f93 by Christian Heimes in branch 'master':
bpo-40695: Limit hashlib builtin hash fallback (GH-20259)
4cc2f93

[miss-islington]

New changeset 7015823 by Miss Islington (bot) in branch '3.9':
bpo-40695: Limit hashlib builtin hash fallback (GH-20259)
7015823

[ned-deily]

Is there anything more that needs to be done for this issue?

[tiran]

No, nothing left to do. Thanks for the ping!