Memory leak in dict with shared keys

[vstinner]

BPO	31317
Nosy	@rhettinger, @pitrou, @vstinner, @methane, @serhiy-storchaka, @zhangyangyu
Superseder	bpo-31217: test_code leaked [1, 1, 1] memory blocks on x86 Gentoo Refleaks 3.6/3.x
Files	test_leak.py

^{Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.}

Show more details

GitHub fields:

assignee = None
closed_at = <Date 2017-08-31.22:23:00.056>
created_at = <Date 2017-08-31.20:18:11.860>
labels = ['3.7', 'performance']
title = 'Memory leak in dict with shared keys'
updated_at = <Date 2017-08-31.22:23:00.054>
user = 'https://github.com/vstinner'

bugs.python.org fields:

activity = <Date 2017-08-31.22:23:00.054>
actor = 'vstinner'
assignee = 'none'
closed = True
closed_date = <Date 2017-08-31.22:23:00.056>
closer = 'vstinner'
components = []
creation = <Date 2017-08-31.20:18:11.860>
creator = 'vstinner'
dependencies = []
files = ['47111']
hgrepos = []
issue_num = 31317
keywords = []
message_count = 8.0
messages = ['301058', '301060', '301062', '301063', '301064', '301065', '301067', '301068']
nosy_count = 6.0
nosy_names = ['rhettinger', 'pitrou', 'vstinner', 'methane', 'serhiy.storchaka', 'xiang.zhang']
pr_nums = []
priority = 'normal'
resolution = 'duplicate'
stage = 'resolved'
status = 'closed'
superseder = '31217'
type = 'resource usage'
url = 'https://bugs.python.org/issue31317'
versions = ['Python 3.7']

[vstinner]

Copy attached test_leak.py to Lib/test/ and run:

haypo@selma$ ./python -m test -R 3:30 test_leak

Run tests in parallel using 1 child processes
0:00:00 load avg: 0.67 [1/1/1] test_leak failed
beginning 33 repetitions
123456789012345678901234567890123
.................................
test_leak leaked [1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1] memory blocks, sum=30

1 test failed:
test_leak

Total duration: 682 ms
Tests result: FAILURE

It seems like Thread2 constructor creates a dictionary on "self.hashvalue = (...)". It's a dictionary sharing keys, but it seems like the dict table memory is not released whne the bug occurs.

Sadly, I'm only able to reproduce the bug on a 32-bit build. Since the bug depends on exact hash values, maybe the bug exists in 64-bit builds, but is less likely?

I'm not 100% sure if the bug is related to dict or set.

The bug comes from bpo-31217.

[vstinner]

If you cannot reproduce the bug on 32-bit build, try to set "FIXED_HASH = 0" in test_leak.py, run "./python -m test -R 3:10 test_leak" multiple times until it fails, then copy the last value of the printed list into hash_values in test_leak.py.

[vstinner]

I spent a lot of time in gdb, but I'm not sure anymore that the issue is related to dict shared keys.

[pitrou]

It's a really weird bug you found. It depends on the exact values passed to -R:

$ ./python -m test -R 3:6 test_haypoleak
Run tests sequentially
0:00:00 load avg: 0.39 [1/1] test_haypoleak
beginning 9 repetitions
123456789
.........
test_haypoleak leaked [1, 1, 1, 1, 1, 1] memory blocks, sum=6
test_haypoleak failed

$ ./python -m test -R 2:6 test_haypoleak
Run tests sequentially
0:00:00 load avg: 0.36 [1/1] test_haypoleak
beginning 8 repetitions
12345678
........
1 test OK.

[pitrou]

Actually, it's not the bug that is weird, but the heuristics in -R seem completely weird now. If I print the alloc and rc deltas:

$ ./python -m test -R 3:6 test_haypoleak
Run tests sequentially
0:00:00 load avg: 0.09 [1/1] test_haypoleak
beginning 9 repetitions
123456789
.........
alloc: [0, 0, 0, 1, 1, 1, 1, 1, 1], rc: [0, 0, 0, 0, 0, 0, 0, 0, 0]
test_haypoleak leaked [1, 1, 1, 1, 1, 1] memory blocks, sum=6
test_haypoleak failed

$ ./python -m test -R 2:6 test_haypoleak
Run tests sequentially
0:00:00 load avg: 0.10 [1/1] test_haypoleak
beginning 8 repetitions
12345678
........
alloc: [0, 0, 0, 1, 1, 1, 1, 1], rc: [0, 0, 0, 0, 0, 0, 0, 0]
1 test OK.

[pitrou]

s/completely weird/broken/

$ ./python -m test -R 2:20 test_haypoleak
Run tests sequentially
0:00:00 load avg: 0.10 [1/1] test_haypoleak
beginning 22 repetitions
1234567890123456789012
......................
alloc: [0, 0, 0, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1], rc: [0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0]
1 test OK.

[pitrou]

To come back to this bug, it still fails if add a __slots__, so I don't think it's related to shared key dictionaries.

[vstinner]

Antoine: "Actually, it's not the bug that is weird, but the heuristics in -R seem completely weird now. If I print the alloc and rc deltas:"

I changed it recently to reduce false alarms: see bpo-30776.

Antoine: "To come back to this bug, it still fails if add a __slots__, so I don't think it's related to shared key dictionaries."

Oh ok, in that case, I will move back to bpo-31217. I close this issue as a duplicate.