hashlib leak on import

[thehesiod]

BPO	33526
Nosy	@gpshead, @tiran, @encukou, @thehesiod

^{Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.}

Show more details

GitHub fields:

assignee = None
closed_at = <Date 2018-05-15.21:51:22.205>
created_at = <Date 2018-05-15.20:49:31.860>
labels = ['extension-modules', 'performance']
title = 'hashlib leak on import'
updated_at = <Date 2018-05-15.21:51:22.203>
user = 'https://github.com/thehesiod'

bugs.python.org fields:

activity = <Date 2018-05-15.21:51:22.203>
actor = 'thehesiod'
assignee = 'none'
closed = True
closed_date = <Date 2018-05-15.21:51:22.205>
closer = 'thehesiod'
components = ['Extension Modules']
creation = <Date 2018-05-15.20:49:31.860>
creator = 'thehesiod'
dependencies = []
files = []
hgrepos = []
issue_num = 33526
keywords = []
message_count = 3.0
messages = ['316723', '316737', '316738']
nosy_count = 4.0
nosy_names = ['gregory.p.smith', 'christian.heimes', 'petr.viktorin', 'thehesiod']
pr_nums = []
priority = 'normal'
resolution = None
stage = 'resolved'
status = 'closed'
superseder = None
type = 'resource usage'
url = 'https://bugs.python.org/issue33526'
versions = ['Python 3.6']

[thehesiod]

I'm seeing a lot of leaks via valgrind against the hashlib module. It appears that it's calling OpenSSL_add_all_digests(); on init, and never calling the corresponding EVP_Cleanup on free: https://www.openssl.org/docs/man1.1.0/crypto/OpenSSL_add_all_digests.html. I see a ton of leaks like the following:

==27765== 24 bytes in 1 blocks are still reachable in loss record 13 of 10,294
==27765== at 0x4C28C7B: malloc (vg_replace_malloc.c:299)
==27765== by 0xA92E337: CRYPTO_malloc (in /usr/lib64/libcrypto.so.1.0.2k)
==27765== by 0xA9E325A: lh_insert (in /usr/lib64/libcrypto.so.1.0.2k)
==27765== by 0xA93103E: OBJ_NAME_add (in /usr/lib64/libcrypto.so.1.0.2k)
==27765== by 0xA9F3559: OpenSSL_add_all_digests (in /usr/lib64/libcrypto.so.1.0.2k)
==27765== by 0xA44CF02: PyInit__hashlib (_hashopenssl.c:998)
==27765== by 0x506E627: _PyImport_LoadDynamicModuleWithSpec (importdl.c:154)
==27765== by 0x506DBA7: _imp_create_dynamic_impl (import.c:2008)
==27765== by 0x5067A2A: _imp_create_dynamic (import.c.h:289)
==27765== by 0x4F3061A: PyCFunction_Call (methodobject.c:114)
==27765== by 0x503E10C: do_call_core (ceval.c:5074)
==27765== by 0x5035F30: _PyEval_EvalFrameDefault (ceval.c:3377)
==27765== by 0x502280F: PyEval_EvalFrameEx (ceval.c:718)
==27765== by 0x503A944: _PyEval_EvalCodeWithName (ceval.c:4139)
==27765== by 0x503DA4D: fast_function (ceval.c:4950)
==27765== by 0x503D3FC: call_function (ceval.c:4830)
==27765== by 0x5035563: _PyEval_EvalFrameDefault (ceval.c:3295)
==27765== by 0x502280F: PyEval_EvalFrameEx (ceval.c:718)
==27765== by 0x503D70D: _PyFunction_FastCall (ceval.c:4891)
==27765== by 0x503D922: fast_function (ceval.c:4926)
==27765== by 0x503D3FC: call_function (ceval.c:4830)
==27765== by 0x5035563: _PyEval_EvalFrameDefault (ceval.c:3295)
==27765== by 0x502280F: PyEval_EvalFrameEx (ceval.c:718)
==27765== by 0x503D70D: _PyFunction_FastCall (ceval.c:4891)
==27765== by 0x503D922: fast_function (ceval.c:4926)
==27765== by 0x503D3FC: call_function (ceval.c:4830)
==27765== by 0x5035563: _PyEval_EvalFrameDefault (ceval.c:3295)
==27765== by 0x502280F: PyEval_EvalFrameEx (ceval.c:718)
==27765== by 0x503D70D: _PyFunction_FastCall (ceval.c:4891)
==27765== by 0x503D922: fast_function (ceval.c:4926)
==27765== by 0x503D3FC: call_function (ceval.c:4830)
==27765== by 0x5035563: _PyEval_EvalFrameDefault (ceval.c:3295)
==27765== by 0x502280F: PyEval_EvalFrameEx (ceval.c:718)
==27765== by 0x503D70D: _PyFunction_FastCall (ceval.c:4891)
==27765== by 0x503D922: fast_function (ceval.c:4926)

I'm not exactly sure how this is happening yet (I know the code I use does a __import__ and uses multiple threads). It sounds like this call should be ref-counted or perhaps only done once for the life of the application.

[tiran]

Thanks for notifying us. There isn't much we can do here.

Python doesn't support unloading of C extension modules. The state will persist until the process ends. It doesn't make much sense to call EVP_cleanup() on shutdown, too. It will just slow down shutdown. The memory is freed anyway once the process has terminated.

[thehesiod]

closing as I'm not quite sure this is right