hashlib memory leak

[AdrianTeng]

BPO	20967
Nosy	@jcea, @vstinner, @AdrianTeng
Files	memoryleak_min.py: demo for memory leak in linux

^{Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.}

Show more details

GitHub fields:

assignee = None
closed_at = <Date 2014-03-19.14:18:12.955>
created_at = <Date 2014-03-18.14:03:55.796>
labels = ['library', 'performance']
title = 'hashlib memory leak'
updated_at = <Date 2014-03-19.14:18:12.954>
user = 'https://github.com/AdrianTeng'

bugs.python.org fields:

activity = <Date 2014-03-19.14:18:12.954>
actor = 'brett.cannon'
assignee = 'none'
closed = True
closed_date = <Date 2014-03-19.14:18:12.955>
closer = 'brett.cannon'
components = ['Library (Lib)']
creation = <Date 2014-03-18.14:03:55.796>
creator = 'ateng'
dependencies = []
files = ['34489']
hgrepos = []
issue_num = 20967
keywords = []
message_count = 5.0
messages = ['213961', '213977', '213978', '213981', '214094']
nosy_count = 3.0
nosy_names = ['jcea', 'vstinner', 'ateng']
pr_nums = []
priority = 'normal'
resolution = 'duplicate'
stage = None
status = 'closed'
superseder = None
type = 'resource usage'
url = 'https://bugs.python.org/issue20967'
versions = ['Python 2.7']

[AdrianTeng]

A particular usage pattern of hashlib will cause a memory leak.

This leaks:
import hashlib
import sys

if __name__ == '__main__':
    data_sha1 = "hello world"
    data_md5 = "hello world"
    for i in xrange(int(1e6)):
        hashlib.sha1(data_sha1)
        hashlib.md5(data_md5)

        if i % 1000 == 0:
            print sys.getrefcount(data_sha1), ",", sys.getrefcount(data_md5)

---

this doesn't leak:

import hashlib
import sys


if __name__ == '__main__':
    data_sha1 = "hello world"
    data_md5 = "hello world"
    for i in xrange(int(1e6)):
        sha1 = hashlib.sha1()
        sha1.update(data_sha1)
		md5 = hashlib.md5()
		md5.update(data_md5)
        
        if i % 1000 == 0:
            print sys.getrefcount(data_sha1), ", ", sys.getrefcount(data_md5)

See attached for leak memory profiling in linux

[vstinner]

I'm unable to reproduce your issue with Python 2.7.5:

$ python memoryleak_min.py 
[256720896.0, 15740928.0, 139264.0]
[256724992.0, 15962112.0, 139264.0]
[256724992.0, 15966208.0, 139264.0]
[256724992.0, 15966208.0, 139264.0]
(...)
[256724992.0, 15966208.0, 139264.0]

$ python2.7 <your example>
5 , 5
5 , 5
(...)
5 , 5

What is your exact Python version? What is your OS? OS version?

[AdrianTeng]

Python 2.7.3, Red Hat Enterprise Linux Server release 5.5, with kernal 2.6.18-308.el5

[vstinner]

"Python 2.7.3, Red Hat Enterprise Linux Server release 5.5, with kernal 2.6.18-308.el5"

So your version is older than mine. Let me check Misc/NEWS... Oh...

"Issue bpo-15219: Fix a reference leak when hashlib.new() is called with invalid parameters."

in the "What's New in Python 2.7.4 release candidate 1" section.

Can you please try a more recent Python 2.7 version to check if it's the same bug?

[AdrianTeng]

Yup. Tested on 2.7.5 and it doesn't leak. I guess this is a duplicate of bpo-15219.
Cheers!