Compileall improvements #82293

encukou · 2019-09-11T14:01:20Z

BPO	38112
Nosy	@gpshead, @vstinner, @encukou, @frenzymadness, @pablogsal
PRs	bpo-38112: Compileall improvements #16012 bpo-38112: compileall: Skip long path path on Windows if the path can't be created #16414 bpo-38112: compileall tests: Use shorter name for long_path test #16419 bpo-38112: Document that compileall.compile_[dir,file] also accept multiple opt levels #20174

^{Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.}

Show more details

GitHub fields:

assignee = None
closed_at = <Date 2019-10-15.13:40:03.440>
created_at = <Date 2019-09-11.14:01:19.510>
labels = ['expert-installation']
title = 'Compileall improvements'
updated_at = <Date 2020-05-18.13:23:44.140>
user = 'https://github.com/encukou'

bugs.python.org fields:

activity = <Date 2020-05-18.13:23:44.140>
actor = 'vstinner'
assignee = 'none'
closed = True
closed_date = <Date 2019-10-15.13:40:03.440>
closer = 'vstinner'
components = ['Installation']
creation = <Date 2019-09-11.14:01:19.510>
creator = 'petr.viktorin'
dependencies = []
files = []
hgrepos = []
issue_num = 38112
keywords = ['patch']
message_count = 11.0
messages = ['351887', '353266', '353271', '353272', '353275', '353286', '353319', '354610', '354728', '362783', '369215']
nosy_count = 5.0
nosy_names = ['gregory.p.smith', 'vstinner', 'petr.viktorin', 'frenzy', 'pablogsal']
pr_nums = ['16012', '16414', '16419', '20174']
priority = 'normal'
resolution = 'fixed'
stage = 'resolved'
status = 'closed'
superseder = None
type = None
url = 'https://bugs.python.org/issue38112'
versions = []

encukou · 2019-09-11T14:01:19Z

Hello,
In Fedora, we've long used a Bash script to compile modules to bytecode, as the compileall module was historically unsuitable.
Recently, Lumír re-wrote our enhancements to compileall, and published the result as compileall2 on PyPI. We'd like to get the improvements back into the standard library so everyone can benefit.
The improvements are:

Raise the default recursion limit
Control of the path baked into the *.pyc file (-s to strip a prefix; -p to prepend a new one) to support different build/install locations.
Compiling for multiple optimization levels at once (by specifying -o multiple times)

A PR should follow shortly.

encukou · 2019-09-26T06:28:44Z

New changeset 8e7bb99 by Petr Viktorin (Lumír 'Frenzy' Balhar) in branch 'master':
bpo-38112: Compileall improvements (GH-16012)
8e7bb99

vstinner · 2019-09-26T07:12:58Z

test_compileall failed on AMD64 Windows8.1 Non-Debug 3.x:
https://buildbot.python.org/all/#/builders/12/builds/3257

Example:

ERROR: test_compile_dir_pathlike (test.test_compileall.CompileallTestsWithSourceEpoch)
----------------------------------------------------------------------

Traceback (most recent call last):
  File "D:\buildarea\3.x.ware-win81-release.nondebug\build\lib\test\test_compileall.py", line 48, in setUp
    os.makedirs(self.long_path)
  File "D:\buildarea\3.x.ware-win81-release.nondebug\build\lib\os.py", line 211, in makedirs
    makedirs(head, exist_ok=exist_ok)
  File "D:\buildarea\3.x.ware-win81-release.nondebug\build\lib\os.py", line 211, in makedirs
    makedirs(head, exist_ok=exist_ok)
  File "D:\buildarea\3.x.ware-win81-release.nondebug\build\lib\os.py", line 211, in makedirs
    makedirs(head, exist_ok=exist_ok)
  [Previous line repeated 18 more times]
  File "D:\buildarea\3.x.ware-win81-release.nondebug\build\lib\os.py", line 221, in makedirs
    mkdir(name, mode)
FileNotFoundError: [WinError 206] The filename or extension is too long: 'D:\\temp\\tmpcgsk1z3q\\long\\1\\2\\3\\4\\5\\6\\7\\8\\9\\10\\11\\12\\13\\14\\15\\16\\17\\18\\19\\20\\21\\22\\23\\24\\25\\26\\27\\28\\29\\30\\31\\32\\33\\34\\35\\36\\37\\38\\39\\40\\41\\42\\43\\44\\45\\46\\47\\48\\49\\50\\51\\52\\53\\54\\55\\56\\57\\58\\59\\60\\61\\62\\63\\64\\65\\66\\67\\68\\69\\70\\71\\72\\73\\74\\75\\76\\77\\78'

Failures:

ERROR: test_compile_dir_pathlike (test.test_compileall.CompileallTestsWithSourceEpoch)
ERROR: test_compile_file_pathlike (test.test_compileall.CompileallTestsWithSourceEpoch)
ERROR: test_compile_file_pathlike_ddir (test.test_compileall.CompileallTestsWithSourceEpoch)
ERROR: test_compile_files (test.test_compileall.CompileallTestsWithSourceEpoch)
ERROR: test_compile_missing_multiprocessing (test.test_compileall.CompileallTestsWithSourceEpoch)
ERROR: test_compile_one_worker (test.test_compileall.CompileallTestsWithSourceEpoch)
ERROR: test_compile_path (test.test_compileall.CompileallTestsWithSourceEpoch)
ERROR: test_compile_pool_called (test.test_compileall.CompileallTestsWithSourceEpoch)
ERROR: test_compile_workers_cpu_count (test.test_compileall.CompileallTestsWithSourceEpoch)
ERROR: test_compile_workers_non_positive (test.test_compileall.CompileallTestsWithSourceEpoch)
ERROR: test_ignore_symlink_destination (test.test_compileall.CompileallTestsWithSourceEpoch)
ERROR: test_magic_number (test.test_compileall.CompileallTestsWithSourceEpoch)
ERROR: test_mtime (test.test_compileall.CompileallTestsWithSourceEpoch)
ERROR: test_multiple_optimization_levels (test.test_compileall.CompileallTestsWithSourceEpoch)
ERROR: test_no_pycache_in_non_package (test.test_compileall.CompileallTestsWithSourceEpoch)
ERROR: test_optimize (test.test_compileall.CompileallTestsWithSourceEpoch)
ERROR: test_prepend_only (test.test_compileall.CompileallTestsWithSourceEpoch)
ERROR: test_strip_and_prepend (test.test_compileall.CompileallTestsWithSourceEpoch)
ERROR: test_strip_only (test.test_compileall.CompileallTestsWithSourceEpoch)
ERROR: test_strip_prepend_and_ddir (test.test_compileall.CompileallTestsWithSourceEpoch)
ERROR: test_compile_dir_pathlike (test.test_compileall.CompileallTestsWithoutSourceEpoch)
ERROR: test_compile_file_pathlike (test.test_compileall.CompileallTestsWithoutSourceEpoch)
ERROR: test_compile_file_pathlike_ddir (test.test_compileall.CompileallTestsWithoutSourceEpoch)
ERROR: test_compile_files (test.test_compileall.CompileallTestsWithoutSourceEpoch)
ERROR: test_compile_missing_multiprocessing (test.test_compileall.CompileallTestsWithoutSourceEpoch)
ERROR: test_compile_one_worker (test.test_compileall.CompileallTestsWithoutSourceEpoch)
ERROR: test_compile_path (test.test_compileall.CompileallTestsWithoutSourceEpoch)
ERROR: test_compile_pool_called (test.test_compileall.CompileallTestsWithoutSourceEpoch)
ERROR: test_compile_workers_cpu_count (test.test_compileall.CompileallTestsWithoutSourceEpoch)
ERROR: test_compile_workers_non_positive (test.test_compileall.CompileallTestsWithoutSourceEpoch)
ERROR: test_ignore_symlink_destination (test.test_compileall.CompileallTestsWithoutSourceEpoch)
ERROR: test_magic_number (test.test_compileall.CompileallTestsWithoutSourceEpoch)
ERROR: test_mtime (test.test_compileall.CompileallTestsWithoutSourceEpoch)
ERROR: test_multiple_optimization_levels (test.test_compileall.CompileallTestsWithoutSourceEpoch)
ERROR: test_no_pycache_in_non_package (test.test_compileall.CompileallTestsWithoutSourceEpoch)
ERROR: test_optimize (test.test_compileall.CompileallTestsWithoutSourceEpoch)
ERROR: test_prepend_only (test.test_compileall.CompileallTestsWithoutSourceEpoch)
ERROR: test_strip_and_prepend (test.test_compileall.CompileallTestsWithoutSourceEpoch)
ERROR: test_strip_only (test.test_compileall.CompileallTestsWithoutSourceEpoch)
ERROR: test_strip_prepend_and_ddir (test.test_compileall.CompileallTestsWithoutSourceEpoch)
FAIL: test_no_args_respects_force_flag (test.test_compileall.CommandLineTestsWithSourceEpoch)
Re-running failed tests in verbose mode
Re-running test_tarfile in verbose mode
Re-running test_compileall in verbose mode
ERROR: test_compile_dir_pathlike (test.test_compileall.CompileallTestsWithSourceEpoch)
ERROR: test_compile_file_pathlike (test.test_compileall.CompileallTestsWithSourceEpoch)
ERROR: test_compile_file_pathlike_ddir (test.test_compileall.CompileallTestsWithSourceEpoch)
ERROR: test_compile_files (test.test_compileall.CompileallTestsWithSourceEpoch)
ERROR: test_compile_missing_multiprocessing (test.test_compileall.CompileallTestsWithSourceEpoch)
ERROR: test_compile_one_worker (test.test_compileall.CompileallTestsWithSourceEpoch)
ERROR: test_compile_path (test.test_compileall.CompileallTestsWithSourceEpoch)
ERROR: test_compile_pool_called (test.test_compileall.CompileallTestsWithSourceEpoch)
ERROR: test_compile_workers_cpu_count (test.test_compileall.CompileallTestsWithSourceEpoch)
ERROR: test_compile_workers_non_positive (test.test_compileall.CompileallTestsWithSourceEpoch)
ERROR: test_ignore_symlink_destination (test.test_compileall.CompileallTestsWithSourceEpoch)
ERROR: test_magic_number (test.test_compileall.CompileallTestsWithSourceEpoch)
ERROR: test_mtime (test.test_compileall.CompileallTestsWithSourceEpoch)
ERROR: test_multiple_optimization_levels (test.test_compileall.CompileallTestsWithSourceEpoch)
ERROR: test_no_pycache_in_non_package (test.test_compileall.CompileallTestsWithSourceEpoch)
ERROR: test_optimize (test.test_compileall.CompileallTestsWithSourceEpoch)
ERROR: test_prepend_only (test.test_compileall.CompileallTestsWithSourceEpoch)
ERROR: test_strip_and_prepend (test.test_compileall.CompileallTestsWithSourceEpoch)
ERROR: test_strip_only (test.test_compileall.CompileallTestsWithSourceEpoch)
ERROR: test_strip_prepend_and_ddir (test.test_compileall.CompileallTestsWithSourceEpoch)
ERROR: test_compile_dir_pathlike (test.test_compileall.CompileallTestsWithoutSourceEpoch)
ERROR: test_compile_file_pathlike (test.test_compileall.CompileallTestsWithoutSourceEpoch)
ERROR: test_compile_file_pathlike_ddir (test.test_compileall.CompileallTestsWithoutSourceEpoch)
ERROR: test_compile_files (test.test_compileall.CompileallTestsWithoutSourceEpoch)
ERROR: test_compile_missing_multiprocessing (test.test_compileall.CompileallTestsWithoutSourceEpoch)
ERROR: test_compile_one_worker (test.test_compileall.CompileallTestsWithoutSourceEpoch)
ERROR: test_compile_path (test.test_compileall.CompileallTestsWithoutSourceEpoch)
ERROR: test_compile_pool_called (test.test_compileall.CompileallTestsWithoutSourceEpoch)
ERROR: test_compile_workers_cpu_count (test.test_compileall.CompileallTestsWithoutSourceEpoch)
ERROR: test_compile_workers_non_positive (test.test_compileall.CompileallTestsWithoutSourceEpoch)
ERROR: test_ignore_symlink_destination (test.test_compileall.CompileallTestsWithoutSourceEpoch)
ERROR: test_magic_number (test.test_compileall.CompileallTestsWithoutSourceEpoch)
ERROR: test_mtime (test.test_compileall.CompileallTestsWithoutSourceEpoch)
ERROR: test_multiple_optimization_levels (test.test_compileall.CompileallTestsWithoutSourceEpoch)
ERROR: test_no_pycache_in_non_package (test.test_compileall.CompileallTestsWithoutSourceEpoch)
ERROR: test_optimize (test.test_compileall.CompileallTestsWithoutSourceEpoch)
ERROR: test_prepend_only (test.test_compileall.CompileallTestsWithoutSourceEpoch)
ERROR: test_strip_and_prepend (test.test_compileall.CompileallTestsWithoutSourceEpoch)
ERROR: test_strip_only (test.test_compileall.CompileallTestsWithoutSourceEpoch)
ERROR: test_strip_prepend_and_ddir (test.test_compileall.CompileallTestsWithoutSourceEpoch)

pythoninfo says:

sys.windowsversion: sys.getwindowsversion(major=6, minor=3, build=9600, platform=2, service_pack='')
windows.RtlAreLongPathsEnabled: <function not available>

vstinner · 2019-09-26T07:16:23Z

Similar errors on AMD64 Windows10 3.x:
https://buildbot.python.org/all/#/builders/3/builds/3567

pythoninfo:

sys.windowsversion: sys.getwindowsversion(major=10, minor=0, build=17134, platform=2, service_pack='')
windows.RtlAreLongPathsEnabled: False

--

And AMD64 Windows7 SP1 3.x:
https://buildbot.python.org/all/#/builders/40/builds/3180

pythoninfo:

sys.windowsversion: sys.getwindowsversion(major=6, minor=1, build=7601, platform=2, service_pack='Service Pack 1')
windows.RtlAreLongPathsEnabled: <function not available>

encukou · 2019-09-26T07:24:44Z

Thanks!
The buildbot complained in the pull request a while ago; I commented there that I'm looking into it.

I'd appreciate a review for the fix; I don't have a good way to test it other than offer it to the buildbots.

encukou · 2019-09-26T09:54:06Z

New changeset 4267c98 by Petr Viktorin in branch 'master':
bpo-38112: compileall: Skip long path path on Windows if the path can't be created (GH-16414)
4267c98

vstinner · 2019-09-26T15:16:00Z

FYI test_compileall.test_compile_dir_maxlevels() also fails on AMD64 FreeBSD CURRENT Shared 3.x:
https://buildbot.python.org/all/#builders/168/builds/1529

pablogsal · 2019-10-14T00:46:41Z

The AMD64 Windows7 SP1 3.x buildbot has been failing for a while since commit 4267c98, please take a look at https://bugs.python.org/issue38470.

vstinner · 2019-10-15T13:40:03Z

Should be fixed by:

New changeset eb1dda2 by Petr Viktorin (Victor Stinner) in branch 'master':
bpo-38470: Fix test_compileall.test_compile_dir_maxlevels() (GH-16789)
eb1dda2

gpshead · 2020-02-27T07:33:08Z

https://bugs.python.org/issue39769 may be of interest. compileall's ddir= does not behave properly in recursive subdirectories, it did the right thing in 2.7 which is similar to a combination of compile_dir(dir, prependdir=ddir, stripdir=dir) after this issue's feature.

vstinner · 2020-05-18T13:23:44Z

New changeset adc72bb by Lumír 'Frenzy' Balhar in branch 'master':
bpo-38112: Document that compileall.compile_[dir,file] also accept multiple opt levels (GH-20174)
adc72bb

encukou added the topic-installation label Sep 11, 2019

vstinner closed this as completed Oct 15, 2019

ezio-melotti transferred this issue from another repository Apr 10, 2022

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Compileall improvements #82293

Compileall improvements #82293

encukou commented Sep 11, 2019

encukou commented Sep 11, 2019

encukou commented Sep 26, 2019

vstinner commented Sep 26, 2019

vstinner commented Sep 26, 2019

encukou commented Sep 26, 2019

encukou commented Sep 26, 2019

vstinner commented Sep 26, 2019

pablogsal commented Oct 14, 2019

vstinner commented Oct 15, 2019

gpshead commented Feb 27, 2020

vstinner commented May 18, 2020

Compileall improvements #82293

Compileall improvements #82293

Comments

encukou commented Sep 11, 2019

encukou commented Sep 11, 2019

encukou commented Sep 26, 2019

vstinner commented Sep 26, 2019

vstinner commented Sep 26, 2019

encukou commented Sep 26, 2019

encukou commented Sep 26, 2019

vstinner commented Sep 26, 2019

pablogsal commented Oct 14, 2019

vstinner commented Oct 15, 2019

gpshead commented Feb 27, 2020

vstinner commented May 18, 2020