Port _hashlib to OpenSSL 3.0.0

[tiran]

BPO	40479
Nosy	@gpshead, @tiran, @ZackerySpytz, @miss-islington
PRs	bpo-40479: Port hashlib to OpenSSL 3.0.0 #19878 bpo-40479: Fix hashlib issue with OpenSSL 3.0.0 #20107 bpo-40479: Test with latest OpenSSL versions #20108 [3.8] bpo-40479: Test with latest OpenSSL versions (GH-20108) #20111 [3.7] bpo-40479: Test with latest OpenSSL versions (GH-20108) #20112 bpo-40479: Fix hashlib's usedforsecurity for OpenSSL 3.0.0 (GH-30455) #30455 [3.10] bpo-40479: Fix hashlib's usedforsecurity for OpenSSL 3.0.0 (GH-30455) #30573 [3.9] bpo-40479: Fix hashlib's usedforsecurity for OpenSSL 3.0.0 (GH-30455) (GH-30574) #30574 bpo-40479: Fix typo, flag must be set for OpenSSL < 3.0.0 (GH-30584) #30584 [3.10] bpo-40479: Fix typo, flag must be set for OpenSSL < 3.0.0 (GH-30584) #30585 [3.9] bpo-40479: Fix typo, flag must be set for OpenSSL < 3.0.0 (GH-30584) #30586 bpo-40479: Fix undefined behavior in Modules/_hashopenssl.c #31153 [3.10] bpo-40479: Fix undefined behavior in Modules/_hashopenssl.c (GH-31153) #31183 [3.9] bpo-40479: Fix undefined behavior in Modules/_hashopenssl.c (GH-31153) #31184

^{Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.}

Show more details

GitHub fields:

assignee = 'https://github.com/tiran'
closed_at = None
created_at = <Date 2020-05-03.09:23:23.143>
labels = ['expert-SSL', '3.8', 'type-feature', '3.7', '3.9']
title = 'Port _hashlib to OpenSSL 3.0.0'
updated_at = <Date 2022-02-07.08:08:13.986>
user = 'https://github.com/tiran'

bugs.python.org fields:

activity = <Date 2022-02-07.08:08:13.986>
actor = 'miss-islington'
assignee = 'christian.heimes'
closed = False
closed_date = None
closer = None
components = ['SSL']
creation = <Date 2020-05-03.09:23:23.143>
creator = 'christian.heimes'
dependencies = []
files = []
hgrepos = []
issue_num = 40479
keywords = ['patch']
message_count = 16.0
messages = ['367962', '368957', '368958', '368961', '368962', '410097', '410471', '410479', '410481', '410523', '410525', '410526', '412614', '412715', '412716', '412717']
nosy_count = 4.0
nosy_names = ['gregory.p.smith', 'christian.heimes', 'ZackerySpytz', 'miss-islington']
pr_nums = ['19878', '20107', '20108', '20111', '20112', '30455', '30573', '30574', '30584', '30585', '30586', '31153', '31183', '31184']
priority = 'normal'
resolution = None
stage = 'patch review'
status = 'open'
superseder = None
type = 'enhancement'
url = 'https://bugs.python.org/issue40479'
versions = ['Python 3.7', 'Python 3.8', 'Python 3.9']

[tiran]

OpenSSL 3.0.0-alpha1 was releases about a week ago. OpenSSL 1.1.x APIs are still functional. However some APIs have been deprecated and FIPS (usedforsecurity flag) is no longer functional.

• One shot HMAC() is deprecated and should be replaced with EVP_MAC API calls
• ERR_func_error_string() is deprecated
• OpenSSL has introduced a new concept of crypto providers (OSSL_PROVIDER), library context (OPENSSL_CTX) and additional flags. A new function EVP_MD_fetch() has been introduced.
• FIPS support has been rewritten and is now shipped with OpenSSL 3.0.0. EVP_MD_CTX_FLAG_NON_FIPS_ALLOW is no longer supported. FIPS state is no longer part of EVP_MD_CTX but of EVP_MD.

See https://wiki.openssl.org/index.php/OpenSSL_3.0

[miss-islington]

New changeset 16d4e6f by Christian Heimes in branch 'master':
bpo-40479: Fix hashlib issue with OpenSSL 3.0.0 (GH-20107)
16d4e6f

[miss-islington]

New changeset 62d618c by Christian Heimes in branch 'master':
bpo-40479: Test with latest OpenSSL versions (GH-20108)
62d618c

[miss-islington]

New changeset 5e6b491 by Miss Islington (bot) in branch '3.7':
bpo-40479: Test with latest OpenSSL versions (GH-20108)
5e6b491

[miss-islington]

New changeset 5a06cf0 by Miss Islington (bot) in branch '3.8':
bpo-40479: Test with latest OpenSSL versions (GH-20108)
5a06cf0

[tiran]

Simple benchmark with openssl-3.0.0-5.el9.x86_64

With custom EVP_MD cache:
$ ./python -m timeit -s "from hashlib import md5" "md5(b'12345678', usedforsecurity=False).digest()"500000 loops, best of 5: 520 nsec per loop
$ ./python -m timeit -s "from hashlib import sha512" "sha512(b'12345678', usedforsecurity=False).digest()"
500000 loops, best of 5: 730 nsec per loop

Without EVP_MD cache:
$ ./python -m timeit -s "from hashlib import md5" "md5(b'', usedforsecurity=False).digest()"
500000 loops, best of 5: 807 nsec per loop
$ ./python -m timeit -s "from hashlib import sha512" "sha512(b'12345678', usedforsecurity=False).digest()"
200000 loops, best of 5: 1.03 usec per loop

[tiran]

New changeset 443b308 by Christian Heimes in branch 'main':
bpo-40479: Fix hashlib's usedforsecurity for OpenSSL 3.0.0 (GH-30455)
443b308

[tiran]

Something went wrong with Miss Islington.
3ce6945 / #74758 is backport to 3.10.

[tiran]

New changeset 4ddd5da by Christian Heimes in branch '3.9':
[3.9] bpo-40479: Fix hashlib's usedforsecurity for OpenSSL 3.0.0 (GH-30455) (GH-30574)
4ddd5da

[tiran]

New changeset 276c234 by Christian Heimes in branch 'main':
bpo-40479: Fix typo, flag must be set for OpenSSL < 3.0.0 (GH-30584)
276c234

[miss-islington]

New changeset 47422a8 by Miss Islington (bot) in branch '3.10':
[3.10] bpo-40479: Fix typo, flag must be set for OpenSSL < 3.0.0 (GH-30584) (GH-30585)
47422a8

[miss-islington]

New changeset 537f16a by Miss Islington (bot) in branch '3.9':
bpo-40479: Fix typo, flag must be set for OpenSSL < 3.0.0 (GH-30584)
537f16a

[ZackerySpytz]

There is a missing call to va_end() in 443b308 (which is undefined behavior). PR 31153 fixes this.

[miss-islington]

New changeset 59e004a by Zackery Spytz in branch 'main':
bpo-40479: Fix undefined behavior in Modules/_hashopenssl.c (GH-31153)
59e004a

[miss-islington]

New changeset 3ceff99 by Miss Islington (bot) in branch '3.10':
bpo-40479: Fix undefined behavior in Modules/_hashopenssl.c (GH-31153)
3ceff99

[miss-islington]

New changeset 0892a0e by Miss Islington (bot) in branch '3.9':
bpo-40479: Fix undefined behavior in Modules/_hashopenssl.c (GH-31153)
0892a0e

[kumaraditya303]

@tiran Can this be closed now ?

[gpshead]

closing as the bulk of this appears done but it isn't clear what 100% is. If there are remaining deprecated APIs in use we can open individual issues for those and clean them up as needed going forwards.