Provide CPython command line functionality via runpy module

[ncoghlan]

BPO	6816
Nosy	@ncoghlan, @pitrou, @ericsnowcurrently

^{Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.}

Show more details

GitHub fields:

assignee = 'https://github.com/ncoghlan'
closed_at = <Date 2009-11-16.06:58:45.790>
created_at = <Date 2009-09-01.10:56:37.757>
labels = ['type-feature']
title = 'Provide CPython command line functionality via runpy module'
updated_at = <Date 2011-11-29.06:34:28.324>
user = 'https://github.com/ncoghlan'

bugs.python.org fields:

activity = <Date 2011-11-29.06:34:28.324>
actor = 'eric.snow'
assignee = 'ncoghlan'
closed = True
closed_date = <Date 2009-11-16.06:58:45.790>
closer = 'ncoghlan'
components = []
creation = <Date 2009-09-01.10:56:37.757>
creator = 'ncoghlan'
dependencies = []
files = []
hgrepos = []
issue_num = 6816
keywords = []
message_count = 14.0
messages = ['92140', '95275', '95280', '95311', '95312', '95315', '95316', '95321', '95323', '95407', '95418', '95420', '95427', '95457']
nosy_count = 4.0
nosy_names = ['ncoghlan', 'pitrou', 'fwierzbicki', 'eric.snow']
pr_nums = []
priority = 'normal'
resolution = 'accepted'
stage = None
status = 'closed'
superseder = None
type = 'enhancement'
url = 'https://bugs.python.org/issue6816'
versions = ['Python 2.7', 'Python 3.2']

[ncoghlan]

Currently, the runpy._run_module_as_main() function allows a launch
script to mimic Python's -m switch fairly well.

This RFE suggests making it possible to mimic other command line
behaviour of the CPython interpreter in a documented fashion:

run_module_as_main - document and make public the existing
_run_module_as_main function (exposes -m style functionality)

run_path_as_main - accept a filesystem path and execute it as per the
command line rules for executing named scripts, zipfiles and directories
(exposes normal script execution functionality)

run_file_as_main - accept a file-like object and execute it as per the
command line rules for executing stdin (exposes '-' style functionality)

run_code_as_main - accept a string or code object and execute it
(exposes -c style functionality)

The runpy module would also be updated to expose these via its command
line to ensure they achieve their stated goal of allowing a launch
script to mimic the native interpreter behaviour. Due to the legacy of
implementing -m style execution by default, the module command line will
expose filesystem path execution through a '-f' switch.

[ncoghlan]

Descoped idea to just provide runpy.run_path (filesystem path equivalent
of runpy.run_module)

[ncoghlan]

2.7: r76286
3.2 merge is still a work in progress

[bitdancer]

The windows buildbots are reporting an odd error in test_runpy. I can't
see by visual inspection why the test is failing:

http://www.python.org/dev/buildbot/stable/builders/x86%20XP-4%20trunk/builds/2615/steps/test/logs/stdio

[pitrou]

The test is probably failing because the pattern is interpreted as a regex:
>>> re.match(r'\t', r'\t') is None
True

There is another problem, which is that test_runpy (or runpy itself) is
now leaking references:

test_runpy
beginning 6 repetitions
123456
......
test_runpy leaked [56, 56, 56] references, sum=168

[ncoghlan]

The windows errors are most likely due to the fact that I'm not doubling
the backslashes in the file paths before passing them to
assertRaisesRegex() as the regular expression pattern.

The reference leak is also interesting, since there isn't any new C code
here. Although I do add quite a few new entries to the path importer
cache as part of the run_path tests - that might be enough to explain it.

[ncoghlan]

Hmm, definitely not the path importer cache - regrtest already takes
that into account by reverting it to its original state before checking
the reference counts.

[ncoghlan]

There's another cache in zipimport that wasn't being cleared when
hunting refleaks. I've updated that on the trunk and will include it in
the forward port to 3k (along with a fix for the Windows path separator
problem).

[ncoghlan]

Added to Py3k in r76324 (including the fixes made on 2.x after the
original checkin)

[pitrou]

Another probably related buildbot failure (again, lack of escaping the
regex pattern):

======================================================================
ERROR: test_directory_error (test.test_runpy.RunPathTest)
----------------------------------------------------------------------

Traceback (most recent call last):
  File
"/Users/buildbot/buildarea/3.x.heller-x86-osx5/build/Lib/runpy.py", line
125, in _get_main_module_details
    return _get_module_details(main_name)
  File
"/Users/buildbot/buildarea/3.x.heller-x86-osx5/build/Lib/runpy.py", line
115, in _get_module_details
    raise ImportError("No code object available for %s" % mod_name)
ImportError: No code object available for __main__

During handling of the above exception, another exception occurred:

ImportError: can't find '__main__' module in
'/private/var/folders/hm/hmcPs0tuELuc-92a0F0LIk+++TM/-Tmp-/tmpp8zORb'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File
"/Users/buildbot/buildarea/3.x.heller-x86-osx5/build/Lib/test/test_runpy.py",
line 334, in test_directory_error
    self._check_import_error(script_dir, msg)
  File
"/Users/buildbot/buildarea/3.x.heller-x86-osx5/build/Lib/test/test_runpy.py",
line 295, in _check_import_error
    self.assertRaisesRegexp(ImportError, msg, run_path, script_name)
  File
"/Users/buildbot/buildarea/3.x.heller-x86-osx5/build/Lib/unittest/case.py",
line 869, in assertRaisesRegexp
    callable_obj(*args, **kwargs)
  File
"/Users/buildbot/buildarea/3.x.heller-x86-osx5/build/Lib/unittest/case.py",
line 126, in __exit__
    expected_regexp = re.compile(expected_regexp)
  File "/Users/buildbot/buildarea/3.x.heller-x86-osx5/build/Lib/re.py",
line 205, in compile
    return _compile(pattern, flags)
  File "/Users/buildbot/buildarea/3.x.heller-x86-osx5/build/Lib/re.py",
line 273, in _compile
    p = sre_compile.compile(pattern, flags)
  File
"/Users/buildbot/buildarea/3.x.heller-x86-osx5/build/Lib/sre_compile.py", line
491, in compile
    p = sre_parse.parse(p, flags)
  File
"/Users/buildbot/buildarea/3.x.heller-x86-osx5/build/Lib/sre_parse.py",
line 692, in parse
    p = _parse_sub(source, pattern, 0)
  File
"/Users/buildbot/buildarea/3.x.heller-x86-osx5/build/Lib/sre_parse.py",
line 315, in _parse_sub
    itemsappend(_parse(source, state))
  File
"/Users/buildbot/buildarea/3.x.heller-x86-osx5/build/Lib/sre_parse.py",
line 522, in _parse
    raise error("multiple repeat")
sre_constants.error: multiple repeat

======================================================================
ERROR: test_zipfile_error (test.test_runpy.RunPathTest)
----------------------------------------------------------------------

Traceback (most recent call last):
  File
"/Users/buildbot/buildarea/3.x.heller-x86-osx5/build/Lib/runpy.py", line
125, in _get_main_module_details
    return _get_module_details(main_name)
  File
"/Users/buildbot/buildarea/3.x.heller-x86-osx5/build/Lib/runpy.py", line
115, in _get_module_details
    raise ImportError("No code object available for %s" % mod_name)
ImportError: No code object available for __main__

During handling of the above exception, another exception occurred:

ImportError: can't find '__main__' module in
'/private/var/folders/hm/hmcPs0tuELuc-92a0F0LIk+++TM/-Tmp-/tmpOprB3K/test_zip.zip'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File
"/Users/buildbot/buildarea/3.x.heller-x86-osx5/build/Lib/test/test_runpy.py",
line 357, in test_zipfile_error
    self._check_import_error(zip_name, msg)
  File
"/Users/buildbot/buildarea/3.x.heller-x86-osx5/build/Lib/test/test_runpy.py",
line 295, in _check_import_error
    self.assertRaisesRegexp(ImportError, msg, run_path, script_name)
  File
"/Users/buildbot/buildarea/3.x.heller-x86-osx5/build/Lib/unittest/case.py",
line 869, in assertRaisesRegexp
    callable_obj(*args, **kwargs)
  File
"/Users/buildbot/buildarea/3.x.heller-x86-osx5/build/Lib/unittest/case.py",
line 126, in __exit__
    expected_regexp = re.compile(expected_regexp)
  File "/Users/buildbot/buildarea/3.x.heller-x86-osx5/build/Lib/re.py",
line 205, in compile
    return _compile(pattern, flags)
  File "/Users/buildbot/buildarea/3.x.heller-x86-osx5/build/Lib/re.py",
line 273, in _compile
    p = sre_compile.compile(pattern, flags)
  File
"/Users/buildbot/buildarea/3.x.heller-x86-osx5/build/Lib/sre_compile.py", line
491, in compile
    p = sre_parse.parse(p, flags)
  File
"/Users/buildbot/buildarea/3.x.heller-x86-osx5/build/Lib/sre_parse.py",
line 692, in parse
    p = _parse_sub(source, pattern, 0)
  File
"/Users/buildbot/buildarea/3.x.heller-x86-osx5/build/Lib/sre_parse.py",
line 315, in _parse_sub
    itemsappend(_parse(source, state))
  File
"/Users/buildbot/buildarea/3.x.heller-x86-osx5/build/Lib/sre_parse.py",
line 522, in _parse
    raise error("multiple repeat")
sre_constants.error: multiple repeat

---

Ran 16 tests in 32.210s

[ncoghlan]

Hmm, I think I need to use a more robust regex escaping approach... I
never knew the OS level temp dir creation could get that creative.

[ncoghlan]

I fixed the tests to use the proper escaping function from the re module
instead of directly doubling backslashes, so that error shouldn't happen
any more.

(Was it one of the unstable buildbots that picked this up? I didn't see
anything come through on the checkins list)

[pitrou]

(Was it one of the unstable buildbots that picked this up? I didn't see
anything come through on the checkins list)

Buildbot failures have stopped being e-mailed long ago it seems.
(ah, you really thought our buildbots were all green in all that
time? :P)

[ncoghlan]

Antoine Pitrou wrote:

Antoine Pitrou <pitrou@free.fr> added the comment:

> (Was it one of the unstable buildbots that picked this up? I didn't see
> anything come through on the checkins list)

Buildbot failures have stopped being e-mailed long ago it seems.
(ah, you really thought our buildbots were all green in all that
time? :P)

I knew the stable list was a lot shorter than the full list, so I
thought it was remotely possible :)