MemoryError when working on large dictionaries on Windows 2003 Server

[amdescombes]

BPO	1270
Nosy	@loewis
Files	unnamed unnamed unnamed unnamed unnamed

^{Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.}

Show more details

GitHub fields:

assignee = None
closed_at = <Date 2007-10-12.10:20:56.551>
created_at = <Date 2007-10-12.10:02:24.144>
labels = ['interpreter-core', 'invalid', 'performance']
title = 'MemoryError when working on large dictionaries on Windows 2003 Server'
updated_at = <Date 2007-10-12.12:53:48.966>
user = 'https://bugs.python.org/amdescombes'

bugs.python.org fields:

activity = <Date 2007-10-12.12:53:48.966>
actor = 'amdescombes'
assignee = 'none'
closed = True
closed_date = <Date 2007-10-12.10:20:56.551>
closer = 'loewis'
components = ['Interpreter Core']
creation = <Date 2007-10-12.10:02:24.144>
creator = 'amdescombes'
dependencies = []
files = ['8519', '8520', '8521', '8522', '8523']
hgrepos = []
issue_num = 1270
keywords = []
message_count = 10.0
messages = ['56361', '56362', '56363', '56364', '56365', '56366', '56367', '56368', '56369', '56371']
nosy_count = 2.0
nosy_names = ['loewis', 'amdescombes']
pr_nums = []
priority = 'normal'
resolution = 'not a bug'
stage = None
status = 'closed'
superseder = None
type = 'resource usage'
url = 'https://bugs.python.org/issue1270'
versions = ['Python 2.5']

[amdescombes]

Hi everybody,

I am using Python 2.5.1.
I am creating a rather large dictionnary, with more than 8 million
entries, memory usage should be about 1.5GB. If I run the program on
Windows XP all runs fine, if I run the program on Windows Server 2003 I
start getting MemoryError exceptions around the 1GB boundary.
Is there a known issue with Windows 2003 Server?

Thanks,

André M. Descombes

[loewis]

Please understand that the purpose of this bug tracker is to track bug
reports and patches to Python, not a means of getting help in using
Python. If you think you have found a bug, please submit a bug report
that allows us to reproduce the bug. If you merely need help, please use
some of the Python community channels, such as python-list@python.org.

As for your question: I'm not aware of an issue with Windows 2003 server
that could explain that behavior. Most likely, you get the MemoryError
because you really ran out of memory. Try increasing the page file, and
put more main memory into the machine.

[amdescombes]

Martin,

it is really a bug, as I have tried on a machine with Windows 2003 server
and 4GBs of ram and it still doesn't run, and it does run on xp with only
1GB of ram.
Does python.exe have then IMAGE_FILE_LARGE_ADDRESS_AWARE bit set, that could
explain the behaviour it its not set.

Thanks,

Andre M. Descombes

On 10/12/07, Martin v. Löwis <report@bugs.python.org> wrote:

Martin v. Löwis added the comment:

Please understand that the purpose of this bug tracker is to track bug
reports and patches to Python, not a means of getting help in using
Python. If you think you have found a bug, please submit a bug report
that allows us to reproduce the bug. If you merely need help, please use
some of the Python community channels, such as python-list@python.org.

As for your question: I'm not aware of an issue with Windows 2003 server
that could explain that behavior. Most likely, you get the MemoryError
because you really ran out of memory. Try increasing the page file, and
put more main memory into the machine.

----------
nosy: +loewis
resolution: -> invalid

---

Tracker <report@bugs.python.org>
<http://bugs.python.org/issue1270\>

---

[loewis]

No, Python does not have IMAGE_FILE_LARGE_ADDRESS_AWARE set; that's not
a bug, though.

In any case, the lack of this flag cannot explain why your application
works on Windows XP and not on Windows 2003.

[amdescombes]

Apparently, XP is more lax when it comes to checking this flag on the
programs it runs then Windows 2003 Server. I am currently doing a test where
I have manually added the flag to python to see if this makes the problem go
away. I will post my results as soon as I have them.

Andre M. Descombes

[amdescombes]

I set the IMAGE_FILE_LARGE_ADDRESS_AWARE flag on python and I still get the
memoryerror exceptions. When they start happening there is still more than
1GB of available memory!

[loewis]

I set the IMAGE_FILE_LARGE_ADDRESS_AWARE flag on python and I still get the
memoryerror exceptions. When they start happening there is still more than
1GB of available memory!

Just in case it isn't clear: I still don't see an indication of a bug in
Python here. If Python reports MemoryError, it likely means that the
system has refused to provide Python with more memory, whether or not
it has memory available for other purposes.

Martin

[amdescombes]

Ok, so where would you recommend I move this discussion to?

Andre

On 10/12/07, Martin v. Löwis <report@bugs.python.org> wrote:

Martin v. Löwis added the comment:

> I set the IMAGE_FILE_LARGE_ADDRESS_AWARE flag on python and I still get
the
> memoryerror exceptions. When they start happening there is still more
than
> 1GB of available memory!

Just in case it isn't clear: I still don't see an indication of a bug in
Python here. If Python reports MemoryError, it likely means that the
system has refused to provide Python with more memory, whether or not
it has memory available for other purposes.

Martin

---

Tracker <report@bugs.python.org>
<http://bugs.python.org/issue1270\>

---

[loewis]

I personally don't think this problem can be resolved by discussion.
Instead, you have to use a debugger, debug your application, Python, and
Windows, to find out the true cause of the problem. Only then discussion
can be fruitful.

If you cannot do these steps yourself, consider hiring somebody for you
who can.

If you still want to discuss, try python-list.

[amdescombes]

Thanks Martin,

I think you are right, perhaps someone on the python-list will have seen and
solved the problem before.

Andre