Memory leak in OpenSSL thread state

[tiran]

BPO	28588
Nosy	@tiran

^{Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.}

Show more details

GitHub fields:

assignee = 'https://github.com/tiran'
closed_at = <Date 2017-09-07.20:39:37.305>
created_at = <Date 2016-11-02.12:01:29.938>
labels = ['extension-modules', 'expert-SSL', '3.7', 'performance']
title = 'Memory leak in OpenSSL thread state'
updated_at = <Date 2017-09-07.20:39:37.303>
user = 'https://github.com/tiran'

bugs.python.org fields:

activity = <Date 2017-09-07.20:39:37.303>
actor = 'christian.heimes'
assignee = 'christian.heimes'
closed = True
closed_date = <Date 2017-09-07.20:39:37.305>
closer = 'christian.heimes'
components = ['Extension Modules', 'SSL']
creation = <Date 2016-11-02.12:01:29.938>
creator = 'christian.heimes'
dependencies = []
files = []
hgrepos = []
issue_num = 28588
keywords = []
message_count = 2.0
messages = ['279922', '301626']
nosy_count = 1.0
nosy_names = ['christian.heimes']
pr_nums = []
priority = 'normal'
resolution = 'wont fix'
stage = 'resolved'
status = 'closed'
superseder = None
type = 'resource usage'
url = 'https://bugs.python.org/issue28588'
versions = ['Python 2.7', 'Python 3.6', 'Python 3.7']

[tiran]

Quote from curl/curl#964

---
I wrote to Matt Caswell from openssl.org about this memleah, and he answered:
OpenSSL maintains a separate error queue for each thread. On each queue there can be
multiple errors. ERR_get_state() does not add any errors to the queue it
merely returns the ERR_STATE (i.e. the queue) for the current thread.
If the current thread has no queue then ERR_get_state() will create one.

ERR_clear_error() removes all the errors that are on the current
thread's queue. It does not deallocate the current thread's queue.

ERR_remove_thread_state() deallocates the specified thread's queue.

The mem leaks you are seeing are almost certainly because the queues for
your app's threads have not been deallocated.
---

The memory leak only affects OpenSSL 1.0.2 and older. OpenSSL 1.1.0 takes care of threading, locking and thread local resources itself.

[tiran]

It's rather painful to fix the issue. Since the memory leak only affects users that create and destroy a lot of threads and the bug has been addressed by OpenSSL 1.1.0, I won't fix it.

If users are running into the issue, they should rather update to more recent OpenSSL versions. The 1.1.0 series uses proper thread local storage.