pkgutil doesn't understand case-senseless filesystems

[freddrake]

BPO	935117
Nosy	@tim-one, @freddrake, @pitrou, @ericvsmith, @tiran, @tarekziade, @merwok
Files	case_ok.py: pure-Python case_ok() function

^{Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.}

Show more details

GitHub fields:

assignee = 'https://github.com/freddrake'
closed_at = None
created_at = <Date 2004-04-14.18:35:19.000>
labels = ['type-feature', 'library']
title = "pkgutil doesn't understand case-senseless filesystems"
updated_at = <Date 2013-06-19.18:46:44.208>
user = 'https://github.com/freddrake'

bugs.python.org fields:

activity = <Date 2013-06-19.18:46:44.208>
actor = 'Ankur.Ankan'
assignee = 'fdrake'
closed = False
closed_date = None
closer = None
components = ['Library (Lib)']
creation = <Date 2004-04-14.18:35:19.000>
creator = 'fdrake'
dependencies = []
files = ['1251']
hgrepos = []
issue_num = 935117
keywords = []
message_count = 7.0
messages = ['20509', '20511', '20512', '59204', '82040', '82068', '185584']
nosy_count = 9.0
nosy_names = ['tim.peters', 'fdrake', 'pitrou', 'eric.smith', 'christian.heimes', 'tarek', 'eric.araujo', 'Anthony.Kong', 'Ankur.Ankan']
pr_nums = []
priority = 'normal'
resolution = None
stage = 'test needed'
status = 'open'
superseder = None
type = 'enhancement'
url = 'https://bugs.python.org/issue935117'
versions = ['Python 3.3']

[freddrake]

The pkgutil.extend_path() function doesn't understand
case-senseless filesystems the way Python's import
does, but it should. On a case-insensitive filesystem,
a directory that matches the package name in spelling
but not case can be mistakenly added to a package by
extend_path(); this can cause differently-named
packages to be mistakenly merged and allow unrelated
code to shadow code in a secondary component of a
package (where secondary means something other than the
first directory found that matches the package).

Consider this tree in a filesystem:

d1/
foo/
__init__.py # this calls pkgutil.extend_path()
module.py # imports module "foo.something"
d2/
Foo/
__init__.py # an unrelated package
something.py
d3/
foo/
__init__.py
something.py

sys.path contains d1/, d2/, d3/, in that order.

After the call to extend_path() in d1/foo/init.py,
foo.__path__ will contain d1/foo/, d2/Foo/, d3/foo/ (in
that order), and foo/module.py will get
d2/Foo/something.py when it imports foo.something.
What's intended is that it get d3/foo/something.py; on
a case-sensitive filesystem, that's what would have
happened.

pkgutil.extend_path() should exercise the same care and
check the same constraints as Python's import.

[tim-one]

Logged In: YES
user_id=31435

Attaching a pure-Python case_ok() "building block" that
should be adequate for doing the hard part of this. It's
trickier than first appears because Python's case-sensitive
imports are not sensitive to the case of extensions
(.py, .pyo, ...) on case-insensitive filesystems.

[freddrake]

Logged In: YES
user_id=3066

Assigned to me since I intend to make this work in the end.

[tiran]

Fred, do you still want to fix the bug?

[pitrou]

Perhaps it's just me, but Fred's use case looks horrible.
If you have three packages named "foo", "Foo" and "foo", all of which
reachable through sys.path on a case-insensitive filesystem, I'm not
sure how to expect Python to make sense out of the situation.
IMHO, the only reasonable solution is to fix the package names...

[freddrake]

Antoine: I agree programmers shouldn't try to create situations like this.

Consider however an application assembled using a build tool like
zc.buildout, which installs each package into a separate installation
location (based on setuptools and easy_install). If the application
uses several packages from the "foo" namespace (to keep with the
original names from the report), there will be directories matching the
first and third packages from the example.

Now, further development on the application may cause a 3rd-party
package named Foo to be used. This name is not the application
programmer's to select, and the volume of code from the "foo" namespace
may be too large to modify; it may represent 3rd party code (for
example, the "zope" namespace package contains many widely used
examples; if used at all, many are likely to be used).

So I think the original use case stands, and may even be more important
than ever given the widespread use of "namespace" packages.

This also suggests that developers should check for the existence of
packages of similar name before releasing code, in order to avoid this
situation.

(I'm disassociating msg20510, since roundup doesn't screw up the tree in
the original report, so that message no longer adds anything useful.)

[merwok]

With the advent of implicit namespace packages in 3.3, I don’t know if anything in pkgutil will be done for this use case.

(BTW is it normal that pkgutil.extend_path is not at least doc-deprecated?)