locale can be imported at startup but relies on too many library modules

[pitrou]

BPO	9548
Nosy	@malemburg, @warsaw, @brettcannon, @rhettinger, @mdickinson, @ncoghlan, @orsenthil, @pitrou, @vstinner, @tiran, @benjaminp, @merwok, @florentx, @ericsnowcurrently
Files	bootlocale.patch bootlocale2.patch bootlocale3.patch

^{Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.}

Show more details

GitHub fields:

assignee = None
closed_at = <Date 2013-10-12.16:05:51.286>
created_at = <Date 2010-08-09.19:09:29.665>
labels = ['type-bug', 'library']
title = 'locale can be imported at startup but relies on too many\tlibrary modules'
updated_at = <Date 2013-10-12.16:05:51.284>
user = 'https://github.com/pitrou'

bugs.python.org fields:

activity = <Date 2013-10-12.16:05:51.284>
actor = 'pitrou'
assignee = 'none'
closed = True
closed_date = <Date 2013-10-12.16:05:51.286>
closer = 'pitrou'
components = ['Library (Lib)']
creation = <Date 2010-08-09.19:09:29.665>
creator = 'pitrou'
dependencies = []
files = ['18455', '32025', '32050']
hgrepos = []
issue_num = 9548
keywords = ['patch']
message_count = 29.0
messages = ['113457', '113459', '113462', '113466', '113476', '113481', '113488', '113492', '113499', '113510', '113511', '113516', '113591', '113596', '113878', '113880', '113912', '113936', '160940', '199346', '199364', '199367', '199507', '199510', '199511', '199512', '199513', '199575', '199581']
nosy_count = 16.0
nosy_names = ['lemburg', 'barry', 'brett.cannon', 'rhettinger', 'mark.dickinson', 'ncoghlan', 'orsenthil', 'pitrou', 'vstinner', 'christian.heimes', 'benjamin.peterson', 'eric.araujo', 'Arfrever', 'flox', 'python-dev', 'eric.snow']
pr_nums = []
priority = 'normal'
resolution = 'fixed'
stage = 'resolved'
status = 'closed'
superseder = None
type = 'behavior'
url = 'https://bugs.python.org/issue9548'
versions = ['Python 3.4']

[pitrou]

Consider the two following commands, from an SVN working copy:

$ ./python -c 'import heapq; print(heapq.heapify)'
<built-in function heapify>

$ cat | ./python -c 'import heapq; print(heapq.heapify)'
<function heapify at 0x7f5d456025a0>

In the second case, the "from _heapq import *" in heapq fails silently. The reason was a bit difficult to find, but it goes as following:

• when run from a non-pty, initializing the standard streams imports Lib/locale.py
• Lib/locale.py import collections which import heapq
• heapq tries to import _heapq but, at this point, the build dir (such as "build/lib.linux-x86_64-3.2/") hasn't been added to sys.path, since site.py does it: the import fails
• heapq silences the ImportError, and this early loaded copy of the module is then made available to all subsequent consumers of heapq, without C accelerators

The issue might seem rather exotic, but it is a more general symptom of the problem that importing Lib/locale.py when initializing std{in,out,err} brings too many import dependencies.

A possible solution is to define a subset of locale.py that is used for bootstrap and rely on as little modules as possible.

[pitrou]

Attached patch fixes the issue by creating a separate "_bootlocale" module, used at bootstrap, which doesn't import collections, functools and friends.

[pitrou]

Updated patch also replaces imports of locale in site.py (since it can be imported early on startup).

[pitrou]

Chances are that the patch will also fix the test_heapq failures on some buildbots (see e.g. http://www.python.org/dev/buildbot/3.x/builders/x86%20Tiger%203.x/builds/759/steps/test/logs/stdio ).

[malemburg]

Antoine Pitrou wrote:

Antoine Pitrou <pitrou@free.fr> added the comment:

Attached patch fixes the issue by creating a separate "_bootlocale" module, used at bootstrap, which doesn't import collections, functools and friends.

I don't think that's the right way forward.

It's much easier to either remove the need to import those
extra modules or defer their import to actual use within
a function.

Both collections and functools are used in two distinct places.
The re module use could also be deferred to actual use.

It would be worthwhile adding a note to the top of the module
mentioning the fact that the module is loaded early on by Python
and to warn about use of non-builtin module imports.

[pitrou]

I don't think that's the right way forward.

It's much easier to either remove the need to import those
extra modules or defer their import to actual use within
a function.

The latter has various issues, such as being overly tedious and
potentially risk (because of the import lock).
Furthermore, functools.wraps is used as a decorator at the toplevel of
locale.py, which makes it currently unremovable.

The former is easier said than done, and we may end up writing poor
man's reimplementations of some stdlib functions.

It would be worthwhile adding a note to the top of the module
mentioning the fact that the module is loaded early on by Python
and to warn about use of non-builtin module imports.

Use of non-builtin stdlib modules is legitimate for many functions which
aren't used in the startup process anyway. The rationale behind the
patch is precisely to define a subset that is needed at startup and
shouldn't rely on stdlib facilities. The rest of the locale module is
allowed to use whatever stdlib facilities it desires, which is always
much more practical.

(note, this is already done elsewhere, for example _abcoll.py or
_weakrefset.py)

[malemburg]

Antoine Pitrou wrote:

Antoine Pitrou <pitrou@free.fr> added the comment:

> I don't think that's the right way forward.
>
> It's much easier to either remove the need to import those
> extra modules or defer their import to actual use within
> a function.

The latter has various issues, such as being overly tedious and
potentially risk (because of the import lock).

Really ? collections is only used in format_string and can easily
be imported there. Note that putting imports into functions is a technique
to remove problems with import locks - it doesn't create any.

Furthermore, functools.wraps is used as a decorator at the toplevel of
locale.py, which makes it currently unremovable.

If you look at the code, that decorator is not needed at all. Furthermore
the whole complication was just introduced to make some tests easier
to write.

The former is easier said than done, and we may end up writing poor
man's reimplementations of some stdlib functions.

Please be more specific. The modules that have an impact
on import time are collections, functools and re.

> It would be worthwhile adding a note to the top of the module
> mentioning the fact that the module is loaded early on by Python
> and to warn about use of non-builtin module imports.

Use of non-builtin stdlib modules is legitimate for many functions which
aren't used in the startup process anyway. The rationale behind the
patch is precisely to define a subset that is needed at startup and
shouldn't rely on stdlib facilities. The rest of the locale module is
allowed to use whatever stdlib facilities it desires, which is always
much more practical.

I said "warn about the use of non-builtin modules", not disallow
their use. AFAIK, the purpose of the exercise is to reduce the
number of non-builtin modules being loaded early on during
Python startup.

(note, this is already done elsewhere, for example _abcoll.py or
_weakrefset.py)

Those are special cases since they are needed for ABCs. locale
is not special enough to warrant such a disruption in module
organization and it's not needed either (see above).

[pitrou]

I said "warn about the use of non-builtin modules", not disallow
their use. AFAIK, the purpose of the exercise is to reduce the
number of non-builtin modules being loaded early on during
Python startup.

The purpose of the exercise is to avoid, as much as possible, the
numerous issues that have already been encountered because of the
startup procedure relying on locale in various conditions for
initialization of the standard streams.

You are proposing that we commit a one-time fix to fix the current
situation. Recent history shows that similar problems will arise again,
and waste useful developer time for committing urgency fixes in order to
fix the build (not to mention that the symptoms are sometimes difficult
to diagnose, such as in this very issue). I am proposing a broader
change which prevents, as much as possible, similar problems from
reappearing and will relieve us from a dispensable burden.

> (note, this is already done elsewhere, for example _abcoll.py or
> _weakrefset.py)

Those are special cases since they are needed for ABCs. locale
is not special enough to warrant such a disruption in module
organization and it's not needed either (see above).

"special enough" is in the eye of the beholder.

[brettcannon]

Just to comment on the import lock issue when importing from within a function, it is a problem. If an import triggers the launching of a thread which calls these functions that have an import inside of them, deadlock will occur. This has been reported as an issue before so people do it (unfortunately).

Importing within functions is something to be avoided when possible.

[vstinner]

heapq tries to import _heapq but, at this point, the build dir
(such as "build/lib.linux-x86_64-3.2/") hasn't been added to sys.path

The problem only exists for developers, not for an installation copy of Python?

Another approach is to call required site code earlier (eg. rewrite it in C and execute it before loading the locale module).

[pitrou]

> heapq tries to import _heapq but, at this point, the build dir
> (such as "build/lib.linux-x86_64-3.2/") hasn't been added to sys.path

The problem only exists for developers, not for an installation copy
of Python?

This particular problem indeed (for developers and for buildbots - see
the weird test_heapq failures on some OS X buildbots).

Another approach is to call required site code earlier (eg. rewrite it
in C and execute it before loading the locale module).

Indeed, but since it calls sysconfig.get_platform(), I'm not sure how
much code would need to be rewritten in C.

[vstinner]

Indeed, but since it calls sysconfig.get_platform(), I'm not sure how
much code would need to be rewritten in C.

Oh, the function is prefixed by the following comment:

# XXX This should not be part of site.py, since it is needed even when
# using the -S option for Python.  See http://www.python.org/sf/586680
def addbuilddir():

Issue bpo-586680 was closed as wontfix.

--

Oh yes, sysconfig.get_platform() is complex :-/

Brett wrote "If we think once can reliably add the directory based purely on whether it starts with "build/lib.", and ..." (msg73411).