add to dict fails after 1,000,000 items on py 2.7.5

[miltmobley]

BPO	19443
Nosy	@tim-one, @rhettinger, @pitrou, @vstinner

^{Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.}

Show more details

GitHub fields:

assignee = None
closed_at = <Date 2013-10-31.20:40:57.864>
created_at = <Date 2013-10-30.04:43:03.321>
labels = ['interpreter-core', 'type-bug', 'invalid']
title = 'add to dict fails after 1,000,000 items on py 2.7.5'
updated_at = <Date 2013-10-31.20:40:57.863>
user = 'https://bugs.python.org/miltmobley'

bugs.python.org fields:

activity = <Date 2013-10-31.20:40:57.863>
actor = 'benjamin.peterson'
assignee = 'none'
closed = True
closed_date = <Date 2013-10-31.20:40:57.864>
closer = 'benjamin.peterson'
components = ['Interpreter Core']
creation = <Date 2013-10-30.04:43:03.321>
creator = 'miltmobley'
dependencies = []
files = []
hgrepos = []
issue_num = 19443
keywords = []
message_count = 9.0
messages = ['201705', '201706', '201707', '201708', '201731', '201767', '201768', '201769', '201847']
nosy_count = 7.0
nosy_names = ['tim.peters', 'rhettinger', 'pitrou', 'vstinner', 'mrabarnett', 'miltmobley', 'Elizacat']
pr_nums = []
priority = 'normal'
resolution = 'not a bug'
stage = None
status = 'closed'
superseder = None
type = 'behavior'
url = 'https://bugs.python.org/issue19443'
versions = ['Python 2.7']

[miltmobley]

d, i = {}, 0
while (i < 10000000):
n = i + 1
d[n] = n
i += 1

On Py 2.7.5 (windows7, x64, 4GB ram) this program slowed down obviously after passing 1,000,000 adds and never completed or raised an exception.
Windows performance monitor showed it was only claiming about 1 MB memory per second, and stopped claiming more memory at about 646MB. The only other large memory consumer in the system was JRE using about 550 MB on behalf of Eclipse.

[Elizacat]

Your test case, nor the one I wrote, trigger Python 2.7.5 nor 3.3.2 on Linux 64-bit:

try: xrange
except: xrange = range

d = {}
r = 10000000
ctr = 0
for n in xrange(r):
   d[n] = n
   ctr += 1

assert len(d) == r
assert ctr == r

[rhettinger]

Works for me -- on 64-bit OS X 10.9 running Python 2.7.5

[tim-one]

Just another data point: runs fine on Vista, 32-bit box, Python 2.7.5. Python is consuming about 320MB when the dict is done building.

[vstinner]

It works for me on Linux 64-bit:

$ python
Python 2.7.3 (default, Aug  9 2012, 17:23:57) 
[GCC 4.7.1 20120720 (Red Hat 4.7.1-5)] on linux2
Type "help", "copyright", "credits" or "license" for more information.
>>> d, i = {}, 0
>>> while (i < 10000000):
...     n =  i + 1
...     d[n] = n
...     i += 1
... 
>>> import os
>>> os.system("grep VmRSS /proc/%s/status" % os.getpid())
VmRSS:	  637984 kB
0

"On Py 2.7.5 (windows7, x64, 4GB ram) this program slowed down obviously after passing 1,000,000 adds and never completed or raised an exception."

What is the exception? How much free memory do you have? If Python has not enough memory, Windows will probably starts to move memory to the disk and the system will becomes slower and slower.

[miltmobley]

I followed the suggestion of email responders to use xrange instead of while, and observed that 32-bit Suse Linux got past 44,000,000 adds before exiting with "Memory Error", while 64-bit Windows 7 slowed down markedly after 22,000,000 adds and was unusable after 44,000,000 adds. However, the program did not stop or raise an exception, which is a concern. The size of the dict was 1.6 GB at that level. My current suspicion is that Windows is not doing a good job of pushing memory already allocated by the process to the virtual file system as the process continues to request more memory. But in my opinion Python should be able to detect failure to complete an allocation request on Windows, and raise an appropriate exception, as it does for Linux.

[mrabarnett]

Works for me: Python 2.7.5, 64-bit, Windows 8.1

[pitrou]

But in my opinion Python should be able to detect failure to complete an allocation request on Windows

Which failure? You're telling us it doesn't fail, it just becomes slow.
(by the way, have you checked whether your machine is swapping when that happens?)

[miltmobley]

I now believe the problem of slow execution is caused by performance of the Windows 7 page file, and not by a Python bug. Others reported that similar tests worked on Windows 8.1 and various Linux systems. So I request to close or withdraw the Python "bug".