Fix docs about module search order

[dmugtasimov]

BPO	16891
Nosy	@ncoghlan, @bitdancer, @serhiy-storchaka

^{Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.}

Show more details

GitHub fields:

assignee = None
closed_at = <Date 2020-05-31.13:52:03.149>
created_at = <Date 2013-01-08.08:14:31.356>
labels = ['type-bug', 'docs']
title = 'Fix docs about module search order'
updated_at = <Date 2020-05-31.13:52:03.148>
user = 'https://bugs.python.org/dmugtasimov'

bugs.python.org fields:

activity = <Date 2020-05-31.13:52:03.148>
actor = 'serhiy.storchaka'
assignee = 'docs@python'
closed = True
closed_date = <Date 2020-05-31.13:52:03.149>
closer = 'serhiy.storchaka'
components = ['Documentation']
creation = <Date 2013-01-08.08:14:31.356>
creator = 'dmugtasimov'
dependencies = []
files = []
hgrepos = []
issue_num = 16891
keywords = []
message_count = 9.0
messages = ['179321', '179322', '179353', '179355', '179357', '179399', '179405', '179422', '370460']
nosy_count = 5.0
nosy_names = ['ncoghlan', 'r.david.murray', 'docs@python', 'serhiy.storchaka', 'dmugtasimov']
pr_nums = []
priority = 'normal'
resolution = 'out of date'
stage = 'resolved'
status = 'closed'
superseder = None
type = 'behavior'
url = 'https://bugs.python.org/issue16891'
versions = ['Python 2.7']

[dmugtasimov]

http://docs.python.org/2/tutorial/modules.html should be rewritten.
AS IS
6.1.2. The Module Search Path

When a module named spam is imported, the interpreter first searches for a built-in module with that name. If not found, it then searches for a file named spam.py in a list of directories given by the variable sys.path. sys.path is initialized from these locations:

TO BE
6.1.2. The Module Search Path

When a module named spam is imported, the interpreter first searches for a built-in module with that name. If not found, it looks in the containing package (the package of which the current module is a submodule). If not found, it then searches for a file named spam.py in a list of directories given by the variable sys.path. sys.path is initialized from these locations:

------
Note that now "6.1.2. The Module Search Path" and "6.4.2. Intra-package References" are contradictary since in 6.4.2 it is said: "In fact, such references are so common that the import statement first looks in the containing package before looking in the standard module search path.", but this is not reflected in 6.1.2.

------
EXAMPLE (for more information see http://stackoverflow.com/questions/14183541/why-python-finds-module-instead-of-package-if-they-have-the-same-name#comment19687166_14183541 ):
/home/dmugtasimov/tmp/name-res3/xyz
__init__.py
a.py
b.py
t.py
xyz.py

Files init.py, b.py and xyz.py are empty
File a.py:

import os, sys
ROOT_DIRECTORY = os.path.abspath(os.path.join(os.path.dirname(__file__), '..'))
if not sys.path or ROOT_DIRECTORY not in sys.path:
    print 'sys.path is modified in a.py'
    sys.path.insert(0, ROOT_DIRECTORY)
else:
    print 'sys.path is NOT modified in a.py'

print 'sys.path:', sys.path
print 'BEFORE import xyz.b'
import xyz.b
print 'AFTER import xyz.b'

File t.py:

import os, sys
ROOT_DIRECTORY = os.path.abspath(os.path.join(os.path.dirname(__file__), '..'))
if not sys.path or ROOT_DIRECTORY not in sys.path:
    print 'sys.path is modified in t.py'
    sys.path.insert(0, ROOT_DIRECTORY)
else:
    print 'sys.path is NOT modified in t.py'

import xyz.a

Run:

python a.py

Output:

sys.path is modified in a.py
sys.path: ['/home/dmugtasimov/tmp/name-res3', '/home/dmugtasimov/tmp/name-res3/xyz',
 '/usr/local/lib/python2.7/dist-packages/tornado-2.3-py2.7.egg',
 '/home/dmugtasimov/tmp/name-res3/xyz', '/usr/lib/python2.7',
 '/usr/lib/python2.7/plat-linux2', '/usr/lib/python2.7/lib-tk',
 '/usr/lib/python2.7/lib-old', '/usr/lib/python2.7/lib-dynload',
 '/usr/local/lib/python2.7/dist-packages',
 '/usr/local/lib/python2.7/dist-packages/setuptools-0.6c11-py2.7.egg-info',
 '/usr/lib/python2.7/dist-packages',
 '/usr/lib/python2.7/dist-packages/PIL',
 '/usr/lib/python2.7/dist-packages/gst-0.10',
 '/usr/lib/python2.7/dist-packages/gtk-2.0',
 '/usr/lib/python2.7/dist-packages/ubuntu-sso-client']
BEFORE import xyz.b
AFTER import xyz.b

Run:

python -vv a.py

Output:

    import xyz # directory /home/dmugtasimov/tmp/name-res3/xyz
    # trying /home/dmugtasimov/tmp/name-res3/xyz/__init__.so
    # trying /home/dmugtasimov/tmp/name-res3/xyz/__init__module.so
    # trying /home/dmugtasimov/tmp/name-res3/xyz/__init__.py
    # /home/dmugtasimov/tmp/name-res3/xyz/__init__.pyc matches /home/dmugtasimov/tmp/name-res3/xyz/__init__.py
    import xyz # precompiled from /home/dmugtasimov/tmp/name-res3/xyz/__init__.pyc
    # trying /home/dmugtasimov/tmp/name-res3/xyz/b.so
    # trying /home/dmugtasimov/tmp/name-res3/xyz/bmodule.so
    # trying /home/dmugtasimov/tmp/name-res3/xyz/b.py
    # /home/dmugtasimov/tmp/name-res3/xyz/b.pyc matches /home/dmugtasimov/tmp/name-res3/xyz/b.py
    import xyz.b # precompiled from /home/dmugtasimov/tmp/name-res3/xyz/b.pyc

Run:

python t.py

Output:

    sys.path is modified in t.py
    sys.path is NOT modified in a.py
    sys.path: ['/home/dmugtasimov/tmp/name-res3', '/home/dmugtasimov/tmp/name-res3/xyz',
     '/usr/local/lib/python2.7/dist-packages/tornado-2.3-py2.7.egg',
     '/home/dmugtasimov/tmp/name-res3/xyz', '/usr/lib/python2.7',
     '/usr/lib/python2.7/plat-linux2', '/usr/lib/python2.7/lib-tk',
     '/usr/lib/python2.7/lib-old', '/usr/lib/python2.7/lib-dynload',
     '/usr/local/lib/python2.7/dist-packages',
     '/usr/local/lib/python2.7/dist-packages/setuptools-0.6c11-py2.7.egg-info',
     '/usr/lib/python2.7/dist-packages',
     '/usr/lib/python2.7/dist-packages/PIL',
     '/usr/lib/python2.7/dist-packages/gst-0.10',
     '/usr/lib/python2.7/dist-packages/gtk-2.0',
     '/usr/lib/python2.7/dist-packages/ubuntu-sso-client']
    BEFORE import xyz.b
    Traceback (most recent call last):
      File "t.py", line 9, in <module>
        import xyz.a
      File "/home/dmugtasimov/tmp/name-res3/xyz/a.py", line 11, in <module>
        import xyz.b
    ImportError: No module named b

Run:

python -vv t.py

Output:

    import xyz # directory /home/dmugtasimov/tmp/name-res3/xyz
    # trying /home/dmugtasimov/tmp/name-res3/xyz/__init__.so
    # trying /home/dmugtasimov/tmp/name-res3/xyz/__init__module.so
    # trying /home/dmugtasimov/tmp/name-res3/xyz/__init__.py
    # /home/dmugtasimov/tmp/name-res3/xyz/__init__.pyc matches /home/dmugtasimov/tmp/name-res3/xyz/__init__.py
    import xyz # precompiled from /home/dmugtasimov/tmp/name-res3/xyz/__init__.pyc
    # trying /home/dmugtasimov/tmp/name-res3/xyz/a.so
    # trying /home/dmugtasimov/tmp/name-res3/xyz/amodule.so
    # trying /home/dmugtasimov/tmp/name-res3/xyz/a.py
    # /home/dmugtasimov/tmp/name-res3/xyz/a.pyc matches /home/dmugtasimov/tmp/name-res3/xyz/a.py
    import xyz.a # precompiled from /home/dmugtasimov/tmp/name-res3/xyz/a.pyc
    # trying /home/dmugtasimov/tmp/name-res3/xyz/os.so
    # trying /home/dmugtasimov/tmp/name-res3/xyz/osmodule.so
    # trying /home/dmugtasimov/tmp/name-res3/xyz/os.py
    # trying /home/dmugtasimov/tmp/name-res3/xyz/os.pyc
    # trying /home/dmugtasimov/tmp/name-res3/xyz/sys.so
    # trying /home/dmugtasimov/tmp/name-res3/xyz/sysmodule.so
    # trying /home/dmugtasimov/tmp/name-res3/xyz/sys.py
    # trying /home/dmugtasimov/tmp/name-res3/xyz/sys.pyc
    # trying /home/dmugtasimov/tmp/name-res3/xyz/xyz.so
    # trying /home/dmugtasimov/tmp/name-res3/xyz/xyzmodule.so
    # trying /home/dmugtasimov/tmp/name-res3/xyz/xyz.py
    # /home/dmugtasimov/tmp/name-res3/xyz/xyz.pyc matches /home/dmugtasimov/tmp/name-res3/xyz/xyz.py
    import xyz.xyz # precompiled from /home/dmugtasimov/tmp/name-res3/xyz/xyz.pyc
    #   clear[2] __file__
    #   clear[2] __package__
    #   clear[2] sys
    #   clear[2] ROOT_DIRECTORY
    #   clear[2] __name__
    #   clear[2] os
    sys.path is modified in t.py
    sys.path is NOT modified in a.py
    sys.path: ['/home/dmugtasimov/tmp/name-res3', '/home/dmugtasimov/tmp/name-res3/xyz',
     '/usr/local/lib/python2.7/dist-packages/tornado-2.3-py2.7.egg',
     '/home/dmugtasimov/tmp/name-res3/xyz', '/usr/lib/python2.7',
     '/usr/lib/python2.7/plat-linux2', '/usr/lib/python2.7/lib-tk',
     '/usr/lib/python2.7/lib-old', '/usr/lib/python2.7/lib-dynload',
     '/usr/local/lib/python2.7/dist-packages',
     '/usr/local/lib/python2.7/dist-packages/setuptools-0.6c11-py2.7.egg-info',
     '/usr/lib/python2.7/dist-packages',
     '/usr/lib/python2.7/dist-packages/PIL',
     '/usr/lib/python2.7/dist-packages/gst-0.10',
     '/usr/lib/python2.7/dist-packages/gtk-2.0',
     '/usr/lib/python2.7/dist-packages/ubuntu-sso-client']
    BEFORE import xyz.b
    Traceback (most recent call last):
      File "t.py", line 9, in <module>
        import xyz.a
      File "/home/dmugtasimov/tmp/name-res3/xyz/a.py", line 11, in <module>
        import xyz.b
    ImportError: No module named b

As you see sys.path is the same for both cases:

sys.path: ['/home/dmugtasimov/tmp/name-res3', '/home/dmugtasimov/tmp/name-res3/xyz', '/usr/local/lib/python2.7/dist-packages/tornado-2.3-py2.7.egg', '/home/dmugtasimov/tmp/name-res3/xyz', '/usr/lib/python2.7', '/usr/lib/python2.7/plat-linux2', '/usr/lib/python2.7/lib-tk', '/usr/lib/python2.7/lib-old', '/usr/lib/python2.7/lib-dynload', '/usr/local/lib/python2.7/dist-packages', '/usr/local/lib/python2.7/dist-packages/setuptools-0.6c11-py2.7.egg-info', '/usr/lib/python2.7/dist-packages', '/usr/lib/python2.7/dist-packages/PIL', '/usr/lib/python2.7/dist-packages/gst-0.10', '/usr/lib/python2.7/dist-packages/gtk-2.0', '/usr/lib/python2.7/dist-packages/ubuntu-sso-client']

But the behaviour is different. For a.py python searches for package xyz first, and them for module b in it:

    import xyz # directory /home/dmugtasimov/tmp/name-res3/xyz
    # trying /home/dmugtasimov/tmp/name-res3/xyz/__init__.so
    # trying /home/dmugtasimov/tmp/name-res3/xyz/__init__module.so
    # trying /home/dmugtasimov/tmp/name-res3/xyz/__init__.py
    # /home/dmugtasimov/tmp/name-res3/xyz/__init__.pyc matches /home/dmugtasimov/tmp/name-res3/xyz/__init__.py
    import xyz # precompiled from /home/dmugtasimov/tmp/name-res3/xyz/__init__.pyc
    # trying /home/dmugtasimov/tmp/name-res3/xyz/b.so
    # trying /home/dmugtasimov/tmp/name-res3/xyz/bmodule.so
    # trying /home/dmugtasimov/tmp/name-res3/xyz/b.py
    # /home/dmugtasimov/tmp/name-res3/xyz/b.pyc matches /home/dmugtasimov/tmp/name-res3/xyz/b.py
    import xyz.b # precompiled from /home/dmugtasimov/tmp/name-res3/xyz/b.pyc

In other words:

Search PACKAGE xyz in directory sys.path[0] -> FOUND
Search module b in PACKAGE xyz -> FOUND
Continue execution

For t.py it searches for moduel xyz in the same directory as a.py itself and then fails to find module b in module xyz:

    # trying /home/dmugtasimov/tmp/name-res3/xyz/xyz.so
    # trying /home/dmugtasimov/tmp/name-res3/xyz/xyzmodule.so
    # trying /home/dmugtasimov/tmp/name-res3/xyz/xyz.py
    # /home/dmugtasimov/tmp/name-res3/xyz/xyz.pyc matches /home/dmugtasimov/tmp/name-res3/xyz/xyz.py
    import xyz.xyz # precompiled from /home/dmugtasimov/tmp/name-res3/xyz/xyz.pyc

In other words:

Search MODULE xyz in directory in the same directory as a.py (or sys.path[1] ?) -> FOUND
Search MODULE b in MODULE xyz -> NOT FOUND
ImportError

So it looks like if "import xyz.b" bahaves different depending on how a.py was initially loaded as a script or imported from another module.

[dmugtasimov]

UPDATE:
CHANGE
http://stackoverflow.com/questions/14183541/why-python-finds-module-instead-of-package-if-they-have-the-same-name#comment19687166_14183541

TO
http://stackoverflow.com/questions/14183541/why-python-finds-module-instead-of-package-if-they-have-the-same-name

Because the whole question and replies are important.

[dmugtasimov]

As I investigate it a little closer it seems to me that it is not a documentation issue, but an implementation issue.

http://docs.python.org/2/reference/simple_stmts.html#import
"A package can contain other packages and modules while modules cannot contain other modules or packages."

The only why to import name from module is
from xyz import b
where "b" is name defined inside xyz module.

Issuing
import xyz.b
means that "b" is module or package.

Therefore xyz cannot be a module, since "...modules cannot contain other modules or packages." This means that xyz is package.

The problem is that it is considered as module for case:
python t.py

[bitdancer]

So it looks like if "import xyz.b" bahaves different depending on how
a.py was initially loaded as a script or imported from another module.

There are several differences between importing a module and running a script, one of which is what is on sys.path. You constructed your example to mask the path difference.

You are getting hit by the difference between absolute and relative imports. How implicit relative imports behave are one of the other things that are different between running a file as a script and importing it. This kind of confusion is one of the reasons implicit relative imports were dropped in Python3.

If you add

  from __future__ import absolute_import

to the top of your a and t files, t will no longer produce an import error.

It could be that the documentation could be improved, but I'm not sure it is worth the effort for 2.7. If there are statements in the 3.x docs that are incorrect now that implicit relative imports are gone, those would definitely be worth fixing.

[dmugtasimov]

A lot of people are still using python 2.7, even 2.6. For me it would be a nice fix in docs since I spent a plenty of time, trying to figure out what is going on.

In my previous comment I also pointed out that implementation probably should be fixed too, since interpreter tries to import module from module, which should not be possible according to docs. It may be an issue for Python 3 too, but I did not check.

[ncoghlan]

The docs sometimes try to draw a sharp distinction between modules and
packages, but it's essentially a lie - a package is really just a module
with a __path__ attribute, and if you know what you are doing, you make
even an ordinary module behave like a package (e.g. os.path)

Until 3.3, the import system had too many internal inconsistencies for sane
documentation. That's fixed in 3.3, but no comprehensive docs will be
happening for earlier versions.

[ncoghlan]

That said, the originally proposed docs change looks like a solid improvement to me.

It's still a lie, but an appropriate one for the tutorial.

[dmugtasimov]

Further investigation led me to the conclusion that "TO BE" should look like this:

6.1.2. The Module Search Path

When a module named spam is imported, the interpreter first searches in the containing package (the package of which the current module is a submodule) if applicable (a module is not required to be a submodule of a package). If not found or module is not a part of any package it searches for a built-in module with that name. If not found, it then searches for a file named spam.py in a list of directories given by the variable sys.path. sys.path is initialized from these locations:

[serhiy-storchaka]

Python 2.7 is no longer supported.