compileall.compile_dir(..., ddir="<prefix>") omits the intermediate package paths when prepending the prefix

[gpshead]

BPO	39769
Nosy	@gpshead
PRs	bpo-39769: Fix compileall ddir for subpkgs. #18676 [3.8] bpo-39769: Fix compileall ddir for subpkgs. (GH-18676) #18718 [3.7] bpo-39769: Fix compileall ddir for subpkgs. (GH-18676) (GH-18718) #18725

^{Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.}

Show more details

GitHub fields:

assignee = None
closed_at = <Date 2020-03-01.19:07:52.955>
created_at = <Date 2020-02-27.05:18:05.962>
labels = ['3.7', '3.8', 'type-bug', 'library', '3.9']
title = 'compileall.compile_dir(..., ddir="<prefix>") omits the intermediate package paths when prepending the prefix'
updated_at = <Date 2020-03-01.19:07:52.954>
user = 'https://github.com/gpshead'

bugs.python.org fields:

activity = <Date 2020-03-01.19:07:52.954>
actor = 'gregory.p.smith'
assignee = 'none'
closed = True
closed_date = <Date 2020-03-01.19:07:52.955>
closer = 'gregory.p.smith'
components = ['Library (Lib)']
creation = <Date 2020-02-27.05:18:05.962>
creator = 'gregory.p.smith'
dependencies = []
files = []
hgrepos = []
issue_num = 39769
keywords = ['patch']
message_count = 11.0
messages = ['362767', '362768', '362782', '362785', '362787', '362788', '362956', '362961', '363077', '363078', '363079']
nosy_count = 1.0
nosy_names = ['gregory.p.smith']
pr_nums = ['18676', '18718', '18725']
priority = 'normal'
resolution = 'fixed'
stage = 'commit review'
status = 'closed'
superseder = None
type = 'behavior'
url = 'https://bugs.python.org/issue39769'
versions = ['Python 3.5', 'Python 3.6', 'Python 3.7', 'Python 3.8', 'Python 3.9']

[gpshead]

Easiest to demonstrate as such:

#!/bin/bash

mkdir bug
touch bug/__init__.py
mkdir bug/foo
touch bug/foo/__init__.py
touch bug/foo/bar.py

python3 -m compileall -d "<prefix>" bug
python2 -m compileall -d "<prefix>" bug

echo "prefix embedded in PY3 pyc code object for lib.foo.bar:"
strings bug/foo/__pycache__/bar.cpython-3*.pyc | grep prefix
echo "prefix embedded in PY2 pyc code object for lib.foo.bar:"
strings bug/foo/bar.pyc | grep prefix

Run that script and you'll see:
Listing 'bug'...
Compiling 'bug/init.py'...
Listing 'bug/foo'...
Compiling 'bug/foo/init.py'...
Compiling 'bug/foo/bar.py'...
Listing bug ...
Compiling bug/init.py ...
Listing bug/pycache ...
Listing bug/foo ...
Compiling bug/foo/init.py ...
Listing bug/foo/pycache ...
Compiling bug/foo/bar.py ...
prefix embedded in PY3 pyc code object for lib.foo.bar:
<prefix>/bar.py
prefix embedded in PY2 pyc code object for lib.foo.bar:
<prefix>/foo/bar.pyt

Notice that the Python 3 pyc file contains a code.co_filename of "<prefix>/bar.py" instead of the correct value (that Python 2 inserts) of "<prefix>/foo/bar.py".

[gpshead]

This has probably been happening for many 3 releases. I noticed it on 3.6 when investigating a problem with wrong paths in tracebacks coming from code which we used compileall.compile_dir(src_root, quiet=1, ddir="<prefix">) on.

I'm guessing ddir= (-d on the cli) isn't widely used?

[gpshead]

Ironically 3.9a4 gains some compileall enhancements that could be used as an awkward workaround from https://bugs.python.org/issue38112:

python 2.7 compile_dir(d, ddir="<foo>") behavior can be had with
python 3.9a4 compile_dir(d, prependdir="<foo>", stripdir=d)

I still intend to fix the actual ddir behavior as it is wrong.

[gpshead]

The regression was caused by the parallelization enhancement implementation from https://bugs.python.org/issue16104

[gpshead]

meaning this regression was introduced in 3.5.

[gpshead]

belated postmortem: if there had been tests against the filename encoded in the pycs living in subdirs as generated by compile_dir, this regression would not have happened.

[gpshead]

New changeset 0267335 by Gregory P. Smith in branch 'master':
bpo-39769: Fix compileall ddir for subpkgs. (GH-18676)
0267335

[gpshead]

keeping this open while i investigate if fixing 3.8 and 3.7 is feasible. if nothing else i'll try to add a note to the docs about the issue in 3.5-3.8 with a code sample suggested workaround.

[gpshead]

New changeset ce720d3 by Gregory P. Smith in branch '3.8':
bpo-39769: Fix compileall ddir for subpkgs. (GH-18676) (GH-18718)
ce720d3

[gpshead]

New changeset 7c64726 by Gregory P. Smith in branch '3.7':
[3.7] bpo-39769: Fix compileall ddir for subpkgs. (GH-18676) (GH-18718) (GH-18725)
7c64726

[gpshead]

if anyone needs this on their older 3.6 or 3.5 trees, the 3.7/3.8 patch is a trivial backport.