Eliminate implicit __main__ relative imports

[ncoghlan]

BPO	29929
Nosy	@warsaw, @ronaldoussoren, @ncoghlan, @Rosuav

^{Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.}

Show more details

GitHub fields:

assignee = None
closed_at = None
created_at = <Date 2017-03-28.09:22:30.381>
labels = ['interpreter-core', 'type-feature', '3.7']
title = 'Eliminate implicit __main__ relative imports'
updated_at = <Date 2017-06-06.19:54:51.571>
user = 'https://github.com/ncoghlan'

bugs.python.org fields:

activity = <Date 2017-06-06.19:54:51.571>
actor = 'ronaldoussoren'
assignee = 'none'
closed = False
closed_date = None
closer = None
components = ['Interpreter Core']
creation = <Date 2017-03-28.09:22:30.381>
creator = 'ncoghlan'
dependencies = []
files = []
hgrepos = []
issue_num = 29929
keywords = []
message_count = 7.0
messages = ['290689', '290691', '290693', '295077', '295175', '295228', '295298']
nosy_count = 4.0
nosy_names = ['barry', 'ronaldoussoren', 'ncoghlan', 'Rosuav']
pr_nums = []
priority = 'normal'
resolution = None
stage = None
status = 'open'
superseder = None
type = 'enhancement'
url = 'https://bugs.python.org/issue29929'
versions = ['Python 3.7']

[ncoghlan]

In just the last 24 hours, I've run across two cases where the default "the script directory is on sys.path" behaviour confused even experienced programmers:

1. a GitHub engineer thought the Python version in their Git-for-Windows bundle was broken because "from random import randint" failed (from a script called "random.py"

2. a Red Hat engineer was thoroughly confused when their systemd.py script was executed a second time when an unhandled exception was raised (Fedora's system Python is integrated with the ABRT crash reporter, and the except hook implementation does "from systemd import journal" while dealing with an unhandled exception)

This isn't a new problem, we've known for a long time that people are regularly confused by this, and it earned a mention as one of my "Traps for the Unwary in Python's Import System": http://python-notes.curiousefficiency.org/en/latest/python_concepts/import_traps.html#the-name-shadowing-trap

However, what's changed is that for the first time I think I see a potential way out of this: rather than injecting the script directory as sys.path[0], we could set it as "__main__.__path__ = [<the-script-dir>]".

Cross-version compatible code would then be written as:

    if "__path__" in globals():
        from . import relative_module_name
    else:
        import relative_module_name

This approach would effectively be a continuation of PEP-328 (which eliminated implicit relative imports from within packages) and PEP-366 (which allowed implicit relative imports from modules executed with the '-m' switch).

[ncoghlan]

A key enabler for actually pursuing this idea would be coming up with a feasible way of emitting a deprecation warning for code that relied on the old implicit relative imports. A reasonable fast check for that would be to:

1. Start populating a private sys._main_path_entry in the sys module in addition to including it in both __main__.__path__ and sys.path

2. During the deprecation period, emit a warning when an import is satisfied from the sys._main_path_entry directory and the fully qualified module name *doesn't* start with "__main__."

3. After the deprecation period, stop populating sys.path[0], stop setting sys._main_path_entry, and stop emitting the deprecation warning

There's still plenty of details to be worked out before this idea could become reality (especially in terms of how it relates to module execution with the -m switch), but both the idea and a managed migration away from the status quo seem like they should be feasible.

[ncoghlan]

In formulating a post to import-sig about this, I realised it made more sense to describe it in terms of the goal (eliminating implicit __main__ relative imports) rather than one possible technique for achieving that goal.

[ncoghlan]

See https://mail.python.org/pipermail/import-sig/2017-March/001068.html for the above-mentioned import-sig post (the design in that email isn't the same as the one described above, but later in the thread I decided the design suggested here is likely to be less confusing overall)

[ronaldoussoren]

A disadvantage of requiring "from . import ..." to import modules next to the script is that this requires a different mechanism before and after installation of a script.

That is, before installation the additional modules are next to the script ("from . import helper" and after installation the additional modules are in site-packages while the script itself is in the bin directory ("import helper").

[ncoghlan]

Ronald: that depends somewhat on how the installation is handled. For example, for entrypoints-style scripts, the entirety of __main__ is auto-generated anyway, and anyone using "./setup.py develop" or "pip install -e ." to add a suitable sys.path entry during development will still be able to do "import helper" regardless of what happens to sys.path by default.

However, Guido has stated he doesn't like the idea of requiring beginners to learn the "from . import helper" construct, so I think that's enough to kill that particular proposed solution to the problem described in my opening message.

Another solution proposed on python-ideas was to move the script directory to the *end* of sys.path rather than having it at the beginning, but the problem with that is that it not only cripples our ability to add new modules to the standard library, but it also greatly increases the odds of additions to site-packages by redistributors breaking end user scripts. As things currently stand, the name shadowing caused by such additions only needs to be resolved by projects that actually want to access the standard library or redistributor provided module. By contrast, if the script directory were added at the end of sys.path, then those scripts would outright break as they'd start getting the newly added module rather than the peer module they were expecting.

Eliminating both of those more general approaches pretty much leaves us with the one more narrowly focused option: when an import candidate matches __main__.__spec__.origin, we start ignoring it with a silent-by-default ImportWarning and move on. To avoid breaking imports when using the -m switch we'd likely need to accept PEP-499 as well, but I was generally inclined to do that anyway (I just hadn't gotten around to offering to be BDFL-Delegate for that PEP as the runpy module maintainer yet)

[ronaldoussoren]

If only someone had access to the time machine keys to fix this 20 year ago :-(. Anything beyond that last option (recognising that the script tries to import itself under another name) is bound to run into odd issues or backward compatibility concerns.

Just recognising a reimport of __main__ should avoid a lot of confusion though, from what I've seen in discussions most cases of unintentional shadowing of the stdlib is caused by folks naming a exploratory script the same as a stdlib module (e.g. naming a script "socket.py" when experimenting with sockets).

W.r.t. "from . import ..." and scripts: installing using entry points isn't a problem, but installing using plain distutils still is as is the even more low-tech option of just copying files to the right location (maybe using a Makefile). But that issue is moot now that Guido has stated he doesn't like the idea.