Python should use 3GB Address Space on Windows

[gfe]

BPO	1449496
Nosy	@tim-one, @loewis, @josiahcarlson, @amauryfa, @pitrou, @tjguk, @briancurtin
Files	Issue1449496.diff

^{Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.}

Show more details

GitHub fields:

assignee = None
closed_at = <Date 2010-12-17.00:44:29.617>
created_at = <Date 2006-03-14.11:04:09.000>
labels = ['type-feature', 'OS-windows']
title = 'Python should use 3GB Address Space on Windows'
updated_at = <Date 2011-05-19.21:39:05.700>
user = 'https://bugs.python.org/gfe'

bugs.python.org fields:

activity = <Date 2011-05-19.21:39:05.700>
actor = 'Chad.Austin'
assignee = 'none'
closed = True
closed_date = <Date 2010-12-17.00:44:29.617>
closer = 'loewis'
components = ['Windows']
creation = <Date 2006-03-14.11:04:09.000>
creator = 'gfe'
dependencies = []
files = ['20083']
hgrepos = []
issue_num = 1449496
keywords = ['patch']
message_count = 14.0
messages = ['61220', '61221', '61222', '61223', '61224', '112751', '124161', '124169', '124198', '124200', '124262', '124263', '124270', '136337']
nosy_count = 10.0
nosy_names = ['tim.peters', 'loewis', 'josiahcarlson', 'amaury.forgeotdarc', 'pitrou', 'gfe', 'tim.golden', 'brian.curtin', 'Jason.Scheirer', 'Chad.Austin']
pr_nums = []
priority = 'normal'
resolution = 'wont fix'
stage = None
status = 'closed'
superseder = None
type = 'enhancement'
url = 'https://bugs.python.org/issue1449496'
versions = ['Python 3.2']

[gfe]

Python runs fine using a 3GB address space on 32bit
Windows.

To take advantage of this feature, Python should be
linked using the /LARGEADDRESSAWARE flag of the
linker.

For details, please refer to:

http://msdn.microsoft.com/library/en-
us/memory/base/4gt_ram_tuning.asp

As most Windows users just use pre-built executables,
this would be a worthwhile change.

Best regards,
Martin Gfeller

[josiahcarlson]

Logged In: YES
user_id=341410

Microsoft claims that the switch does not negatively affect
users on systems without large amounts of memory, and the
extra gig of memory on platforms which could use it would be
nice.

Can you offer a patch?

[tim-one]

Logged In: YES
user_id=31435

MS also says "However, the file cache, paged pool, and
non-paged pool are smaller, which can adversely affect
applications with heavy networking or I/O", so the rationale
for making this change is clear as mud. That is, it's not a
pure win: some apps win at the expense of others.

[gfe]

Logged In: YES
user_id=884167

Tim, the /LARGEADDRESSAWARE flag just specifies whether
python.exe is enabled for #GB addresses, i.e. doesn't use
dubious pointer manipulations that make it fail with a 3GB
address space.

It does not cause Windows to provide the 3GB address space,
nor to reduce cache and pool size - that is controlled
separately by the /3GB boot switch (and various registry
settings for finer control of cache and pool sizes).

In other words, it is an enabling declaration, not a
control parameter.

Josiah, I don't compile on Windows, hence cannot provide a
patch. What I did is applying the flag to the executable by
using editbin.exe.

Brgds, Martin

[tim-one]

Logged In: YES
user_id=31435

Thanks, Martin. Raymond Chen notes that some C code can
fail when this is enabled:

http://blogs.msdn.com/oldnewthing/archive/2004/08/12/213468.aspx

I doubt that core Python's C has 2GB assumptions, but
extension modules might. Provided that whoever cares enough
to write the patch (sorry, I don't) points to that info (in
the NEWS file would be good enough), this is fine by me.

[briancurtin]

Unless anyone plans on a patch I say we let this go. As of 2.5 we provide x64 installers, and with most users running on 64-bit OS'es I would say it's better to suggest they use a 64-bit compiled Python to obtain an even larger addressable space.

Of course, not all users are on 64-bit, but it'll take some work and tests for this to be considered for 3.2.

[JasonScheirer]

The patch is a simple tweak to the Visual Studio project's compiler settings. Seems quite innocuous, nothing broke in the Python harness or our internal harnesses (at ESRI) by enabling this.

[loewis]

I'm rejecting this as "won't fix". Users having the underlying problem (i.e. need more than 2GB of heap) really should switch to a 64-bit installation. The risk of something breaking is just not worth it.

That testing showed no issues is not convincing at all. Existing test suites will likely not exercise critical code paths wrt. this change, which are issues caused by some objects potentially exceeding 2GB in 32-bit mode. This might not even be Python core objects, but may be objects in an extension module.

[gfe]

Martin, we're running with this for years and with many extensions modules, without an issue. What is 64-bit safe should be 32-bit safe, not only 31-bit safe. But you're right, this is not a proof, and we have switched to 64-bit ourselves.

[amauryfa]

What is 64-bit safe should be 32-bit safe, not only 31-bit safe

Not here. Python uses "signed size_t" for various lengths and sizes.
On win32 this only gives you 31 bits...

[JasonScheirer]

I would like to see this reopened: we have a very large class of users that are not ready to entirely port to 64-bit and need this now.

When running a Python script from within a desktop application (which embeds Python and has /LARGEADDRESSAWARE set) or outside in Python.exe (which does not) results in the outputs from the tools calling out to these 32 bit libraries to produce different outputs because the amount of data they are able to allocate and process at the same time. The same Python script that gives correct output on a large dataset in this desktop app will not yield the same quality of results when run overnight as a stand-alone Python script. Essentially this discounts Python scripts as an option for automation in this case.

I understand that yes, applications still cannot allocate more CONTIGUOUS memory, but this is not a regression if it continues to be so. Allowing the process to allocate more TOTAL memory is a net benefit, especially on on complex software systems that can't simply be rebuilt in 64 bit mode due to third-party restraints (tied to a specific library version, lack of access to source, lengthy software approvals processes, etc.)

[pitrou]

When running a Python script from within a desktop application (which
embeds Python and has /LARGEADDRESSAWARE set) [...]. Essentially this
discounts Python scripts as an option for automation in this case.

Well, if you already embed Python in your application, you can probably also embed it in a simple console application which will enable automation, can't you?

[loewis]

I would like to see this reopened: we have a very large class of
users that are not ready to entirely port to 64-bit and need this
now.

And I remain -1 to such requests. You can appeal to that by writing a
PEP, or finding a committer who is willing to make this change.

Alternatively, just provide these users with binaries that you built
yourself, or hire somebody to build such binaries for you.

I understand that yes, applications still cannot allocate more
CONTIGUOUS memory, but this is not a regression if it continues to be
so.

It would be a regression if Python started crashing because of that
change, which is my concern.

[ChadAustin]

IMVU's downloadable client is built atop CPython. 80% of our users use 32-bit Windows and 20% use 64-bit Windows. We do not intend to provide 64-bit builds of our application for many of the same reasons sketched out in http://blogs.msdn.com/b/ricom/archive/2009/06/10/visual-studio-why-is-there-no-64-bit-version.aspx

Process address space is one of our key bottlenecks: limiting the amount of cache we can mmap in, various runtime heap reserves, etc. For those 20% of users running 64-bit Windows, /LARGEADDRESSAWARE would give 32-bit Python access to 4 GB of address space, which would certainly help them.

The other reason not to use 64-bit binaries is that they consume significantly more memory and cache, especially because Python is so pointer-heavy.

In the meantime, we can use editbin /LARGEADDRESSAWARE to modify the shipped .exes.