test_bigmem.test_from_2G_generator uses more memory than expected

[pitrou]

BPO	5438
Nosy	@Yhg1s, @pitrou, @serhiy-storchaka, @zooba, @llllllllll, @miss-islington
PRs	bpo-5438: Update memory requirements and optimize test_bigmem.py. #11123 [3.7] bpo-5438: Update memory requirements and optimize test_bigmem.py. (GH-11123) #11259
Files	bigmem.patch bigmem-update.patch bigmem_usage_440G.png

^{Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.}

Show more details

GitHub fields:

assignee = 'https://github.com/Yhg1s'
closed_at = <Date 2019-08-02.22:25:50.886>
created_at = <Date 2009-03-08.01:25:13.696>
labels = ['3.8', '3.7', 'tests', 'performance']
title = 'test_bigmem.test_from_2G_generator uses more memory than expected'
updated_at = <Date 2019-08-02.22:25:50.880>
user = 'https://github.com/pitrou'

bugs.python.org fields:

activity = <Date 2019-08-02.22:25:50.880>
actor = 'steve.dower'
assignee = 'twouters'
closed = True
closed_date = <Date 2019-08-02.22:25:50.886>
closer = 'steve.dower'
components = ['Tests']
creation = <Date 2009-03-08.01:25:13.696>
creator = 'pitrou'
dependencies = []
files = ['38945', '38952', '38996']
hgrepos = []
issue_num = 5438
keywords = ['patch']
message_count = 22.0
messages = ['83295', '109768', '132074', '240713', '240728', '240842', '240848', '240870', '240884', '240887', '240913', '240922', '240936', '240944', '240953', '240960', '240968', '241147', '241939', '331660', '332191', '332194']
nosy_count = 7.0
nosy_names = ['twouters', 'pitrou', 'nadeem.vawda', 'serhiy.storchaka', 'steve.dower', 'llllllllll', 'miss-islington']
pr_nums = ['11123', '11259']
priority = 'normal'
resolution = 'fixed'
stage = 'resolved'
status = 'closed'
superseder = None
type = 'resource usage'
url = 'https://bugs.python.org/issue5438'
versions = ['Python 3.7', 'Python 3.8']

[pitrou]

It seems test_from_2G_generator doesn't respect its announced memory
consumption. I launched test_bigmem with "-M 12G" on a 16GB machine, and
the interpreter process got killed by the OOM killer (Out Of Memory)
while running this test case.

[BreamoreBoy]

Is this problem repeatable? I haven't got a big enough box on which to try this out.

[Yhg1s]

I'm not surprised this test takes up more memory... It claims to take up 9 times the test size in memory, with a minimum useful test size of 1G. With a limit of 12G that would make the test size about 1.33G. However, it then proceeds to build a tuple of 159 million distinct integers and stuffs them into a tuple. Each integer is at least 24 bytes long, plus the 8 of the pointer in the tuple, making the total memory use over 32 times the test size. I'll fix the test.

[llllllllll]

I am now computing the size based on the system's int and pointer size.

[llllllllll]

I have updated the test to make another test that does not have a __len__.

[llllllllll]

Steve, we talked about the possibility of having you run some of these tests on a machine with a large amount of ram; I am adding you to the nosy list.

[serhiy-storchaka]

How long time the test run (if there is enough memory)? May be we should decrease memory requirements by using shared values?

[llllllllll]

I am not sure yet; I don't have access to a machine that can run the test.

[zooba]

I'll soon have Windows and Ubuntu machines with 448GB RAM each - anything special I should test here?

[llllllllll]

The tests need to be run with higher memory caps, this can be set with the -M flag, like "python -m test -M 64G test_bigmem". I am under the impression that these tests don't get run all the time so we might want to run that suite.

[zooba]

It's running fine, but not really exercising the limit. I ran with -M 440G and we haven't gone past 38GB usage yet. I'll post a timeline plot when it finishes (on Windows - not sure how to collect the data but I'm also running on Ubuntu), but it looks like we may need to tweak the tests to go closer to the limit.

[zooba]

I'm up to test_from_2G_generator. It says the peak will be 36GB. Memory grew in chunks up to about 40GB and has been stuck there for 10 minutes.