CPython build options for out-of-the box performance

[alecsandrupatrascu]

BPO	26359
Nosy	@malemburg, @brettcannon, @gpshead, @pitrou, @scoder, @vstinner, @ned-deily, @bitdancer, @skrah, @zware, @zooba
PRs	[Do Not Merge] Convert Misc/NEWS so that it is managed by towncrier #552
Dependencies	bpo-21122: CPython fails to build modules with LLVM LTO on Mac OS X
Files	cpython2_pgo_default-v01.patch cpython3_pgo_default-v02.patch cpython3_with_optimizations_v01.patch cpython2_with_optimizations_v01.patch cpython3_with_optimizations_v02.patch cpython2_with_optimizations_v02.patch cpython3_with_optimizations_v02-gps01.patch issue26359.diff

^{Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.}

Show more details

GitHub fields:

assignee = 'https://github.com/gpshead'
closed_at = <Date 2016-09-08.07:10:04.089>
created_at = <Date 2016-02-14.10:28:12.931>
labels = ['build', 'performance']
title = 'CPython build options for out-of-the box performance'
updated_at = <Date 2017-03-31.16:36:31.892>
user = 'https://bugs.python.org/alecsandrupatrascu'

bugs.python.org fields:

activity = <Date 2017-03-31.16:36:31.892>
actor = 'dstufft'
assignee = 'gregory.p.smith'
closed = True
closed_date = <Date 2016-09-08.07:10:04.089>
closer = 'gregory.p.smith'
components = ['Build']
creation = <Date 2016-02-14.10:28:12.931>
creator = 'alecsandru.patrascu'
dependencies = ['21122']
files = ['41918', '41919', '42576', '42871', '42872', '43142', '44406', '44412']
hgrepos = []
issue_num = 26359
keywords = ['patch']
message_count = 32.0
messages = ['260271', '263353', '263372', '263374', '263384', '263408', '263415', '263452', '263466', '264058', '265432', '265692', '265693', '265699', '266735', '267074', '267076', '273936', '274277', '274647', '274678', '274679', '274683', '274687', '274689', '274981', '275055', '275104', '275105', '275172', '281301', '288528']
nosy_count = 13.0
nosy_names = ['lemburg', 'brett.cannon', 'gregory.p.smith', 'pitrou', 'scoder', 'vstinner', 'ned.deily', 'r.david.murray', 'skrah', 'python-dev', 'zach.ware', 'steve.dower', 'alecsandru.patrascu']
pr_nums = ['552']
priority = 'normal'
resolution = 'fixed'
stage = 'resolved'
status = 'closed'
superseder = None
type = 'performance'
url = 'https://bugs.python.org/issue26359'
versions = ['Python 2.7', 'Python 3.5', 'Python 3.6']

[alecsandrupatrascu]

Hello,

This is Alecsandru from the Dynamic Scripting Languages Optimization Team at Intel Corporation. I would like to submit a patch as a response to the discussion thread opened in Python-Dev (https://mail.python.org/pipermail/python-dev/2016-February/143215.html), regarding the way CPython is built, mainly the options that are available to the programmers. Analyzing the CPython ecosystem we can see that there are a lot of users that just download the sources and hit the commands "./configure", "make" and "make install" once and then continue using it with their Python scripts. One of the problems with this workflow it that the users do not benefit from the entire optimization features that are existing in the build system, such as PGO and LTO.

Therefore, I propose a workflow, like the following. Assume some work has to be done into the CPython interpreter, a developer can do the following steps:
A. Implementation and debugging phase.
1. The command "./configure PYDIST=debug" is ran once. It will enable the Py_DEBUG, -O0 and -g flags
2. The command "make" is ran once or multiple times

B. Testing the implementation from step A, in a pre-release environment
1. The command "./configure PYDIST=devel" is ran once. It will disable the Py_DEBUG flags and will enable the -O3 and -g flags, and it is just like the current implementation in CPython
2. The command "make" is ran once or multiple times

C. For any other CPython usage, for example distributing the interpreter, installing it inside an operating system, or just the majority of users who are not CPython developers and only want to compile it once and use it as-is:
1. The command "./configure" is ran once. Alternatively, the command "./configure PYDIST=release" can be used. It will disable all debugging functionality, enable the -O3 flag and will enable PGO and LTO.
2. The command "make" is ran once

Thank you,
Alecsandru

[vstinner]

Sorry, I don't understand this issue. I don't think that we need 3 flavors: debug, devel and release. We already have debug (--with-pydebug) and release, and IMHO debug is what developers should use.

I like the idea of using the best optimizers options *by default* for the release mode. In practice, I suggest to enable PGO *by default*.

[alecsandrupatrascu]

Hello Victor,

Indeed, the best outcome is to have PGO, LTO, etc enabled by default when running ./configure && make, for production level. I also feel that they should be on by default, if only developers would use the debug version.

Do you think that modifying the patches attached to this issue and enabling all optimizations by default could be considered acceptable by the Python community?

Thank you,
Alecsandru

[skrah]

Not acceptable, I'm afraid. See bpo-25702. I'm not sure why it is
considered so bothersome to type --with-pgo and --with-lto for
the rare case of a real production build.

[alecsandrupatrascu]

Maybe that end users are lazy and want to type just make and let things happen magically behind the scenes :-)

[zooba]

FWIW, I'm also against enabling it by default due to the extended build time. There are plenty of reasons to build Python with the non-debug ABI that don't require extended optimizations. Production builds are the exception I believe, not the rule.

[brettcannon]

What if we added a --with-optimizations flag to build --with-pgo, --with-lto, and avoid having to run make twice thanks to make/make profileopt? That way the default build is still quick and reasonable but gain a simple way to use the right flags and avoid having to remember to run make twice?

[gpshead]

--with-optimizations seems fine.

As does having the final thing the Makefile prints out when run from a configuration that did not specify any of --with-pgo, --with-lto, --with-pydebug, or --with-optimizations be to echo message reminding people to configure --with-optimizations when making a release build.

The default "./configure && make" target needs to remain optimized for CPython developer speed. Not release builds.

[alecsandrupatrascu]

That sounds like a good idea and plan, to have everything enabled with just one dedicated configure flag.

I can work on this and post the patches as soon as they are done.

Thank you

[alecsandrupatrascu]

I've added the --with-optimizations option to configure, for CPython3 only at this point. If it looks good I'll create a version also for CPython2.

[brettcannon]

The patch LGTM but I'm not a configure/make expert. Anyone else want to have a look?

Alecsandru, if no one takes a look within the next week then just assign the issue to me.

[alecsandrupatrascu]

I've attached a cleaner version for CPython 3 and one for CPython 2. The problem with CPython 2 is that Gregory did not yet merged the LTO on this branch (bpo-25702), but as soon as he does it, I'll post updated patches.

Regards,
Alecsandru

[alecsandrupatrascu]

Brett, I tried to add you in the "assigned to", but I do not have any rights to update those :(

[brettcannon]

Just leave a comment next week in the issue if no one else beats me to the issue (I'm at OSCON so I don't have time to commit until May 24 at the earliest).

[alecsandrupatrascu]

Hello Brett, I saw that there was no other activity on this issue so far. Can you please update this, as I do not have rights?

[alecsandrupatrascu]

I've updated the patch for 2.7 branch to contain the LTO part merged by Gregory.

[alecsandrupatrascu]

The patches in discussions are the v02 ones.

[brettcannon]

Just FYI, Alecsandru, I plan on applying your patches this week or next, so if you can just double-check they still apply cleanly that would be great.

[brettcannon]

So if I enable this manually, I get various test failures from modules on OS X El Capitan with the same type of failure:

[ 95/398] test_bytes
Traceback (most recent call last):
  File "/Users/brettcannon/Repositories/python/cpython/3.5/Lib/runpy.py", line 193, in _run_module_as_main
    "__main__", mod_spec)
  File "/Users/brettcannon/Repositories/python/cpython/3.5/Lib/runpy.py", line 85, in _run_code
    exec(code, run_globals)
  File "/Users/brettcannon/Repositories/python/cpython/3.5/Lib/test/__main__.py", line 3, in <module>
    regrtest.main_in_temp_cwd()
  File "/Users/brettcannon/Repositories/python/cpython/3.5/Lib/test/regrtest.py", line 1593, in main_in_temp_cwd
    main()
  File "/Users/brettcannon/Repositories/python/cpython/3.5/Lib/test/regrtest.py", line 756, in main
    raise Exception("Child error on {}: {}".format(test, result[1]))
Exception: Child error on test_bytes: Exit code -6

Since I can't get the test suite to pass with PGO+LTO I'm not comfortable committing this patch and promoting people turn them on.

[gpshead]

Minor edit to avoid printing the message about --with-optimizations when doing a --with-pydebug build. otherwise this patch works well on Ubuntu xenial 16.04 amd64.

From discussions in the CPython Core Sprint room today - we've agreed that we should just modify the configure.ac check to not enable --with-lto as part of this when on MacOS until an xcode toolchain for that to work properly on can be figured out.

[brettcannon]

Here is modification of Greg's patch to turn off LTO under OS X since LTO+PGO do not get along under El Capitan.

[brettcannon]

Patch with the missing Makefile.pre.in part :P

[python-dev]

New changeset b4b73473ecc6 by Brett Cannon in branch 'default':
Issue bpo-26359: Add the --with-optimizations configure flag.
https://hg.python.org/cpython/rev/b4b73473ecc6

[python-dev]

New changeset 1e61cc86df03 by Brett Cannon in branch '3.5':
Issue bpo-26359: Add the --with-optimizations flag.
https://hg.python.org/cpython/rev/1e61cc86df03

New changeset f9b52b31f1b4 by Brett Cannon in branch 'default':
Merge for bpo-26359
https://hg.python.org/cpython/rev/f9b52b31f1b4

[brettcannon]

Leaving this open in case someone wants to backport to 2.7.

[python-dev]

New changeset bc28cbd49070 by Gregory P. Smith in branch '2.7':
Issue bpo-26359: Add the --with-optimizations flag.
https://hg.python.org/cpython/rev/bc28cbd49070

[ned-deily]

Thanks for the new option. Perhaps the top-level README should be updated to mention --with-optimizations?

[python-dev]

New changeset e1987bf14148 by Gregory P. Smith in branch '3.5':
document --with-optimizations in the README for bpo-26359.
https://hg.python.org/cpython/rev/e1987bf14148

New changeset 4afb12077e08 by Gregory P. Smith in branch 'default':
document --with-optimizations in the README for bpo-26359.
https://hg.python.org/cpython/rev/4afb12077e08

[python-dev]

New changeset 8567bc2876af by Gregory P. Smith in branch '2.7':
document --with-optimizations in the README for bpo-26359.
https://hg.python.org/cpython/rev/8567bc2876af

[gpshead]

see https://bugs.python.org/issue28032 for flaws with --with-optimizations builds.

[gpshead]

per comments in bpo-28032 the new configure flag has been renamed from --with-optimizations to --enable-optimizations in all branches it was added to:

remote: notified python-checkins@python.org of incoming changeset c0ea81315fb6
remote: notified python-checkins@python.org of incoming changeset 58c1a49a10b4
remote: notified python-checkins@python.org of incoming changeset 11cacf9f9a33
remote: notified python-checkins@python.org of incoming changeset 0d2b42344ae5
remote: notified python-checkins@python.org of incoming changeset 2d1d70b53376
remote: notified python-checkins@python.org of incoming changeset d5ff5a2f33fd

[vstinner]

It seems like it's not possible to install Python configured with --enable-optimizations: see my issue bpo-29641, "make install failure when using ./configure --enable-optimizations.