Enhancing '-m' to support packages (PEP 338)

[ncoghlan]

BPO	1043356
Nosy	@rhettinger, @ncoghlan
Files	execmodule.py: Python code to execute a module as a script PEP_338.diff: C source diff for PEP 338 pep338.zip: Simple package for feature demonstration

^{Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.}

Show more details

GitHub fields:

assignee = None
closed_at = <Date 2006-02-11.08:51:23.000>
created_at = <Date 2004-10-08.21:44:01.000>
labels = []
title = "Enhancing '-m' to support packages (PEP 338)"
updated_at = <Date 2006-02-11.08:51:23.000>
user = 'https://github.com/ncoghlan'

bugs.python.org fields:

activity = <Date 2006-02-11.08:51:23.000>
actor = 'ncoghlan'
assignee = 'none'
closed = True
closed_date = None
closer = None
components = ['None']
creation = <Date 2004-10-08.21:44:01.000>
creator = 'ncoghlan'
dependencies = []
files = ['6287', '6288', '6289']
hgrepos = []
issue_num = 1043356
keywords = ['patch']
message_count = 9.0
messages = ['47031', '47032', '47033', '47034', '47035', '47036', '47037', '47038', '47039']
nosy_count = 2.0
nosy_names = ['rhettinger', 'ncoghlan']
pr_nums = []
priority = 'normal'
resolution = None
stage = None
status = 'closed'
superseder = None
type = None
url = 'https://bugs.python.org/issue1043356'
versions = ['Python 2.5']

[ncoghlan]

Thomas Heller posted "python -m pychecker.checker
<script>' as an example where it is worthwhile to allow
execution of a script inside a package. This seems like
a compelling use case to me (making pychecker easy to
run is enough of a PITA that I confess I often don't
run it), and I'm sure there are other examples.

The only semantics that I can see working for packages
are the following:

python -m pychecker.checker

is equivalent to:

python execchecker.py

where execchecker.py is:

import sys, imp
import pychecker
info = imp.find_module("checker", pychecker.__path__)
sys.argv[0] = info[1]
execfile(info[0])

We *have* to do the import of the requested script's
containing package - otherwise, we can't obey Python's
packaging rules with respect to the use of __path__.

I will attach two versions.

The first will, when it fails to find a module for
itself, switch to executing something like "-m
execmodule". 'execmodule.py' would be a new stdlib
module to handle the above behaviour (initially not
providing any useful interfaces when directly
imported). Guido was certainly right when he said this
will be easier to enhance if written in Python - I
think this way (going to Python when it isn't a
top-level module) gives us the best of both worlds.

The second version will be pure C. The reason I'm going
to do that, is I expect the idea of adding a standard
library module for this to be controversial, especially
given the proximity to 2.4beta1. (I hope I'm wrong,
since I think the Python-C hybrid is a more flexible
solution, which can also be leveraged by other
interpreter implementations)

[ncoghlan]

Logged In: YES
user_id=1038590

execmodule.py attached.

You can experiment using "python -m execmodule <script>",
since I haven't written the C code to automatically hook
into this yet.

"from execmodule import execmodule" also has intriguing
possibilities. . .

[ncoghlan]

Logged In: YES
user_id=1038590

run_package_Py.diff attached, and a new version of execmodule.py

If it fails to find the module requested with '-m', it
reinterprets the command line from "python -m <script>" to
"python -m execmodule <script>".

One obscure issue is that a command line like "python
-mpychecker.checker" will overwrite the C-level argv[0] with
the path to execmodule.py, whereas "python -m
pychecker.checker" will leave it alone (the path to
execmodule.py is stored in argv[1] in the second case).

This may interact oddly with anyone using Py_GetArgcArgv to
get the *original* command line arguments.

One possibility is to just leave junk in sys.argv[0] when
running execmodule automatically (execmodule just deletes it
anyway).

I'm out of time for today, so I doubt I'll be able to get to
a C version before the beta release.

[rhettinger]

Logged In: YES
user_id=80475

This should be taken back to python-dev. Guido had strong
reservations about even the simple version of the patch. It
is rapidly losing its innocence.

Also, it's a little late in the game. The beta is due out on
Tuesday (leaving no time for thorough review or opportunity
for fixups if necessary).

The mention of "intriguing possibilities" will likely not help
rally support.

[ncoghlan]

Logged In: YES
user_id=1038590

I agreee with Raymond - bumping the group to Python 2.5.

I'll put execmodule.py in the cookbook as a useful 2.4
recipe (getting the failure modes right was surprisingly
tricky, and I can see people wanting this capability)

At the moment, making this part of the standard library
would be understandably controversial. It will probably be
better to wait and see what the wider (i.e. non-python-dev)
community thinks of the basic version if '-m' before making
things too complex.

The intriguing possibilities of execmodule are simply that
it makes "execmodule('some_pkg.some_module') and
"execmodule('somemodule') available at the Python level. The
notion of using the Python namespace instead of the
filesystem (via execfile) to execute subscripts is an
interesting one.

[ncoghlan]

Logged In: YES
user_id=1038590

execmodule has been posted to the cookbook:
http://aspn.activestate.com/ASPN/Cookbook/Python/Recipe/307772

I've also updated the attached version of execmodule.py to
match what I posted to the cookbook. The main difference is
to increase the similarity of the interface to that for
execfile - the dict arguments are called globals and locals,
and the module is correctly executed in the caller's
namespace if no globals are provided.

[ncoghlan]

Logged In: YES
user_id=1038590

I've put a PEP together for adding this in 2.5 - a draft was
posted to python-dev, but I'm still waiting for the PEP number.

Two points from the PEP draft:

1. The implementation will stay as a Python/C hybrid. It's
   more useful that way (e.g. "scripts that run scripts" like
   pychecker.checker or profile can easily be given '-m' style
   arguments of their own)

2. The C portion of the implementation needs to change so
   that argv[0] is never altered. The easiest way to do that is
   to leave junk in sys.argv[0] when execmodule.py gets invoked
   implicitly. I don't have a functional build machine at the
   moment, which is why I haven't updated the patch.

The draft can be found here:
http://mail.python.org/pipermail/python-dev/2004-November/049837.html

[ncoghlan]

Logged In: YES
user_id=1038590

The PEP now has a number (PEP-338) which I've added to the
title of the tracker item.

I've also attached an updated patch which matches the
behaviour described in the PEP.

Finally, I added a small archive with a simple package structure
to demonstrate the feature.

To try it out:

1. Download execmodule.py to your build environment's Lib
   directory
2. Apply the patch from the src directory with "patch -p0 -i
   PEP_338.diff" & rebuild
3. Extract the archive so it makes a "PEP-338" package inside
   the Lib directory.

Then, "./python -m PEP-338.hi" and "./python -m
PEP-338.subpkg.hi" should print out short messages along with
the contents of sys.argv.

[ncoghlan]

Logged In: YES
user_id=1038590

Closing this patch and submitting a new one after major
update to the PEP.