Prefer libb2 over vendored copy of blake2

[tiran]

BPO	47095
Nosy	@gpshead, @tiran
PRs	bpo-47095: Use libb2 to provide blake2 implementation (GH-32059) #32059

^{Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.}

Show more details

GitHub fields:

assignee = None
closed_at = <Date 2022-03-26.20:05:29.265>
created_at = <Date 2022-03-22.19:51:17.380>
labels = ['extension-modules', 'type-bug', '3.11']
title = 'Prefer libb2 over vendored copy of blake2'
updated_at = <Date 2022-03-26.20:05:29.265>
user = 'https://github.com/tiran'

bugs.python.org fields:

activity = <Date 2022-03-26.20:05:29.265>
actor = 'christian.heimes'
assignee = 'none'
closed = True
closed_date = <Date 2022-03-26.20:05:29.265>
closer = 'christian.heimes'
components = ['Extension Modules']
creation = <Date 2022-03-22.19:51:17.380>
creator = 'christian.heimes'
dependencies = []
files = []
hgrepos = []
issue_num = 47095
keywords = ['patch']
message_count = 3.0
messages = ['415807', '415833', '416085']
nosy_count = 2.0
nosy_names = ['gregory.p.smith', 'christian.heimes']
pr_nums = ['32059']
priority = 'normal'
resolution = 'fixed'
stage = 'resolved'
status = 'closed'
superseder = None
type = 'behavior'
url = 'https://bugs.python.org/issue47095'
versions = ['Python 3.11']

[tiran]

Python's blake2 implementation provides hashing, MAC (key, salt, personalization), variable length output, and tree hashing [1]. All features except for tree hashing are provided by OpenSSL 3.0.0 and newer [2]. It is unlikely that OpenSSL will get tree hashing any time soon, if all. [3]

I would like to remove our vendored copy of blake2 eventually and just rely on OpenSSL. Therefore I propose to deprecate tree hashing feature so we can drop it in Python 3.13. The tree hashing parameters are: fanout, depth, leaf_size, node_offset, node_depth, inner_size, last_node

Note: OpenSSL 3.0 might impose additional restrictions on the parameter. It might be possible that OpenSSL does not support salt and personalization (OSSL_MAC_PARAM_CUSTOM) without a MAC key.

Alternatively we could replace our copy of blake2 and depend on libb2 from https://blake2.net/. libb2 is available in Fedora.

[1] https://docs.python.org/3/library/hashlib.html#hashlib.blake2b
[2] https://www.openssl.org/docs/manmaster/man7/EVP_MAC-BLAKE2.html
[3] openssl/openssl#980

[gpshead]

In the short term we should prefer libb2 linkage when available.

As for deprecation, it'd be useful to research how often the options going away are used in code in PyPI packages and in Github repos to understand the deprecation impact.

The PyPI landscape for blake2 modules is not great because we've had it in hashlib for a while. One of those, or a new one, would need to be created by someone who needs the non openssl features.

ultimate goal: simplify what's in Modules/_blake2/impl/ if it cannot be removed. Use an external library for the implementation when possible (and in all our binary releases. Those are better maintained to take advantage of performance or hw features over time.

[tiran]

New changeset b16b6bb by Christian Heimes in branch 'main':
bpo-47095: Use libb2 to provide blake2 implementation (GH-32059)
b16b6bb

[zooba]

Is there a configure option to prefer the vendored copy? Or static linking? (This change has broken some of my semi-portable builds, because libb2 isn't installed by default but it gets pulled in by the build tools, so it's on the build machines.)

[tiran]

Untested: You should be able to statically link with libb2 using something like LIBB2_LIBS="-L/path/to/libdir -l:libb2.a -Wl,--exclude-libs,libb2.a

[zooba]

Thanks! I used LIBB2_LIBS="-L$(pkg-config libb2 --variable=libdir) -l:libb2.a -Wl,--exclude-libs,libb2.a" and it's worked just fine (well, with one targeted test, going to be another hour or so before I can do a "real" test).