Investigate replacing SHA3 code with OpenSSL

[tiran]

BPO	37630
Nosy	@gpshead, @tiran, @stratakis, @miss-islington
PRs	bpo-37630: Use SHA3 and SHAKE XOF from OpenSSL #16049 bpo-37630: Fix spelling shake128 -> shake_128 (GH-20154) #20154 bpo-37630: Do not skip the sha3 tests in case of missing builtin sha3 module #20986

^{Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.}

Show more details

GitHub fields:

assignee = 'https://github.com/tiran'
closed_at = None
created_at = <Date 2019-07-19.11:30:56.450>
labels = ['extension-modules', 'type-bug', '3.9']
title = 'Investigate replacing SHA3 code with OpenSSL'
updated_at = <Date 2021-04-17.21:27:15.432>
user = 'https://github.com/tiran'

bugs.python.org fields:

activity = <Date 2021-04-17.21:27:15.432>
actor = 'miss-islington'
assignee = 'christian.heimes'
closed = False
closed_date = None
closer = None
components = ['Extension Modules']
creation = <Date 2019-07-19.11:30:56.450>
creator = 'christian.heimes'
dependencies = []
files = []
hgrepos = []
issue_num = 37630
keywords = ['patch']
message_count = 7.0
messages = ['348165', '369066', '369067', '369138', '369715', '369741', '391313']
nosy_count = 4.0
nosy_names = ['gregory.p.smith', 'christian.heimes', 'cstratak', 'miss-islington']
pr_nums = ['16049', '20154', '20986']
priority = 'normal'
resolution = None
stage = 'patch review'
status = 'open'
superseder = None
type = 'behavior'
url = 'https://bugs.python.org/issue37630'
versions = ['Python 3.9']

[tiran]

Recent OpenSSL comes with SHA3. Now that Python is going to drop support for old OpenSSL, we can consider to use OpenSSL's SHA3 and drop the reference implementation from Python.

For variable length SHAKE API, OpenSSL added EVP_MD_CTRL_XOF_LEN and EVP_DigestFinalXOF().

[miss-islington]

New changeset d5b3f6b by Christian Heimes in branch 'master':
bpo-37630: Use SHA3 and SHAKE XOF from OpenSSL (GH-16049)
d5b3f6b

[tiran]

I'll add a whatsnew later.

[tiran]

New changeset 62ecd8a by Christian Heimes in branch 'master':
bpo-37630: Fix spelling shake128 -> shake_128 (GH-20154)
62ecd8a

[tiran]

OpenSSL's SHA-3 implementation is a tiny bit faster than our builtin copy of SHA-3.

builtin SHA-3 with PGO

$ python3 -m timeit -s "from _sha3 import sha3_256; d = b'12345678' * 1000" "sha3_256(d)"
10000 loops, best of 5: 20.3 usec per loop

builtin SHA-3 without PGO

$ ./python -m timeit -s "from _sha3 import sha3_256; d = b'12345678' * 1000" "sha3_256(d)"
10000 loops, best of 5: 21.1 usec per loop

OpenSSL SHA-3

$ ./python -m timeit -s "from _hashlib import openssl_sha3_256 as sha3_256; d = b'12345678' * 1000" "sha3_256(d)"
20000 loops, best of 5: 19.1 usec per loop

OpenSSL's Blake2 implementation is also a tiny bit faster. (b.copy().update() because the _hashlib module doesn't have fast constructor yet)

$ python3 -m timeit -s "from _blake2 import blake2b; b = blake2b(); d = b'12345678' * 1000" "b.copy().update(d)"
50000 loops, best of 5: 9.67 usec per loop
$ python3 -m timeit -s "from _hashlib import new; b = new('blake2b512'); d = b'12345678' * 1000" "b.copy().update(d)"
50000 loops, best of 5: 8.87 usec per loop

[tiran]

LibreSSL does neither include SHA3/SHAKE family nor Blake2. Feature requests have been open for 1.5 to almost four years. The first reply on each feature request don't come as a surprise to me...

libressl/portable#199
libressl/portable#455

[miss-islington]

New changeset 6857198 by stratakis in branch 'master':
bpo-37630: Do not skip the sha3 tests in case of missing builtin sha3 module (GH-20986)
6857198

[gpshead]

OpenSSL doesn't have to provide sha3. LibreSSL noted above does not. BoringSSL does not. You can compile without it or disable it. etc.

We've switched our built-in implementation used when OpenSSL does not offer it to use a verified HACL* sha3 implementation instead of the 3.11 tiny_sha3 one which replaced the <=3.10 reference implementation sha3.