Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

CPython build options for out-of-the box performance #70547

Closed
alecsandrupatrascu mannequin opened this issue Feb 14, 2016 · 32 comments
Closed

CPython build options for out-of-the box performance #70547

alecsandrupatrascu mannequin opened this issue Feb 14, 2016 · 32 comments
Assignees
Labels
build The build process and cross-build performance Performance or resource usage

Comments

@alecsandrupatrascu
Copy link
Mannequin

alecsandrupatrascu mannequin commented Feb 14, 2016

BPO 26359
Nosy @malemburg, @brettcannon, @gpshead, @pitrou, @scoder, @vstinner, @ned-deily, @bitdancer, @skrah, @zware, @zooba
PRs
  • [Do Not Merge] Convert Misc/NEWS so that it is managed by towncrier #552
  • Dependencies
  • bpo-21122: CPython fails to build modules with LLVM LTO on Mac OS X
  • Files
  • cpython2_pgo_default-v01.patch
  • cpython3_pgo_default-v02.patch
  • cpython3_with_optimizations_v01.patch
  • cpython2_with_optimizations_v01.patch
  • cpython3_with_optimizations_v02.patch
  • cpython2_with_optimizations_v02.patch
  • cpython3_with_optimizations_v02-gps01.patch
  • issue26359.diff
  • Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.

    Show more details

    GitHub fields:

    assignee = 'https://github.com/gpshead'
    closed_at = <Date 2016-09-08.07:10:04.089>
    created_at = <Date 2016-02-14.10:28:12.931>
    labels = ['build', 'performance']
    title = 'CPython build options for out-of-the box performance'
    updated_at = <Date 2017-03-31.16:36:31.892>
    user = 'https://bugs.python.org/alecsandrupatrascu'

    bugs.python.org fields:

    activity = <Date 2017-03-31.16:36:31.892>
    actor = 'dstufft'
    assignee = 'gregory.p.smith'
    closed = True
    closed_date = <Date 2016-09-08.07:10:04.089>
    closer = 'gregory.p.smith'
    components = ['Build']
    creation = <Date 2016-02-14.10:28:12.931>
    creator = 'alecsandru.patrascu'
    dependencies = ['21122']
    files = ['41918', '41919', '42576', '42871', '42872', '43142', '44406', '44412']
    hgrepos = []
    issue_num = 26359
    keywords = ['patch']
    message_count = 32.0
    messages = ['260271', '263353', '263372', '263374', '263384', '263408', '263415', '263452', '263466', '264058', '265432', '265692', '265693', '265699', '266735', '267074', '267076', '273936', '274277', '274647', '274678', '274679', '274683', '274687', '274689', '274981', '275055', '275104', '275105', '275172', '281301', '288528']
    nosy_count = 13.0
    nosy_names = ['lemburg', 'brett.cannon', 'gregory.p.smith', 'pitrou', 'scoder', 'vstinner', 'ned.deily', 'r.david.murray', 'skrah', 'python-dev', 'zach.ware', 'steve.dower', 'alecsandru.patrascu']
    pr_nums = ['552']
    priority = 'normal'
    resolution = 'fixed'
    stage = 'resolved'
    status = 'closed'
    superseder = None
    type = 'performance'
    url = 'https://bugs.python.org/issue26359'
    versions = ['Python 2.7', 'Python 3.5', 'Python 3.6']

    @alecsandrupatrascu
    Copy link
    Mannequin Author

    alecsandrupatrascu mannequin commented Feb 14, 2016

    Hello,

    This is Alecsandru from the Dynamic Scripting Languages Optimization Team at Intel Corporation. I would like to submit a patch as a response to the discussion thread opened in Python-Dev (https://mail.python.org/pipermail/python-dev/2016-February/143215.html), regarding the way CPython is built, mainly the options that are available to the programmers. Analyzing the CPython ecosystem we can see that there are a lot of users that just download the sources and hit the commands "./configure", "make" and "make install" once and then continue using it with their Python scripts. One of the problems with this workflow it that the users do not benefit from the entire optimization features that are existing in the build system, such as PGO and LTO.

    Therefore, I propose a workflow, like the following. Assume some work has to be done into the CPython interpreter, a developer can do the following steps:
    A. Implementation and debugging phase.
    1. The command "./configure PYDIST=debug" is ran once. It will enable the Py_DEBUG, -O0 and -g flags
    2. The command "make" is ran once or multiple times

    B. Testing the implementation from step A, in a pre-release environment
    1. The command "./configure PYDIST=devel" is ran once. It will disable the Py_DEBUG flags and will enable the -O3 and -g flags, and it is just like the current implementation in CPython
    2. The command "make" is ran once or multiple times

    C. For any other CPython usage, for example distributing the interpreter, installing it inside an operating system, or just the majority of users who are not CPython developers and only want to compile it once and use it as-is:
    1. The command "./configure" is ran once. Alternatively, the command "./configure PYDIST=release" can be used. It will disable all debugging functionality, enable the -O3 flag and will enable PGO and LTO.
    2. The command "make" is ran once

    Thank you,
    Alecsandru

    @alecsandrupatrascu alecsandrupatrascu mannequin added build The build process and cross-build performance Performance or resource usage labels Feb 14, 2016
    @vstinner
    Copy link
    Member

    vstinner commented Apr 13, 2016

    Sorry, I don't understand this issue. I don't think that we need 3 flavors: debug, devel and release. We already have debug (--with-pydebug) and release, and IMHO debug is what developers should use.

    I like the idea of using the best optimizers options *by default* for the release mode. In practice, I suggest to enable PGO *by default*.

    @alecsandrupatrascu
    Copy link
    Mannequin Author

    alecsandrupatrascu mannequin commented Apr 14, 2016

    Hello Victor,

    Indeed, the best outcome is to have PGO, LTO, etc enabled by default when running ./configure && make, for production level. I also feel that they should be on by default, if only developers would use the debug version.

    Do you think that modifying the patches attached to this issue and enabling all optimizations by default could be considered acceptable by the Python community?

    Thank you,
    Alecsandru

    @skrah
    Copy link
    Mannequin

    skrah mannequin commented Apr 14, 2016

    Not acceptable, I'm afraid. See bpo-25702. I'm not sure why it is
    considered so bothersome to type --with-pgo and --with-lto for
    the rare case of a real production build.

    @alecsandrupatrascu
    Copy link
    Mannequin Author

    alecsandrupatrascu mannequin commented Apr 14, 2016

    Maybe that end users are lazy and want to type just make and let things happen magically behind the scenes :-)

    @zooba
    Copy link
    Member

    zooba commented Apr 14, 2016

    FWIW, I'm also against enabling it by default due to the extended build time. There are plenty of reasons to build Python with the non-debug ABI that don't require extended optimizations. Production builds are the exception I believe, not the rule.

    @brettcannon
    Copy link
    Member

    brettcannon commented Apr 14, 2016

    What if we added a --with-optimizations flag to build --with-pgo, --with-lto, and avoid having to run make twice thanks to make/make profileopt? That way the default build is still quick and reasonable but gain a simple way to use the right flags and avoid having to remember to run make twice?

    @gpshead
    Copy link
    Member

    gpshead commented Apr 15, 2016

    --with-optimizations seems fine.

    As does having the final thing the Makefile prints out when run from a configuration that did not specify any of --with-pgo, --with-lto, --with-pydebug, or --with-optimizations be to echo message reminding people to configure --with-optimizations when making a release build.

    The default "./configure && make" target needs to remain optimized for CPython developer speed. Not release builds.

    @alecsandrupatrascu
    Copy link
    Mannequin Author

    alecsandrupatrascu mannequin commented Apr 15, 2016

    That sounds like a good idea and plan, to have everything enabled with just one dedicated configure flag.

    I can work on this and post the patches as soon as they are done.

    Thank you

    @alecsandrupatrascu
    Copy link
    Mannequin Author

    alecsandrupatrascu mannequin commented Apr 23, 2016

    I've added the --with-optimizations option to configure, for CPython3 only at this point. If it looks good I'll create a version also for CPython2.

    @brettcannon
    Copy link
    Member

    brettcannon commented May 12, 2016

    The patch LGTM but I'm not a configure/make expert. Anyone else want to have a look?

    Alecsandru, if no one takes a look within the next week then just assign the issue to me.

    @alecsandrupatrascu
    Copy link
    Mannequin Author

    alecsandrupatrascu mannequin commented May 16, 2016

    I've attached a cleaner version for CPython 3 and one for CPython 2. The problem with CPython 2 is that Gregory did not yet merged the LTO on this branch (bpo-25702), but as soon as he does it, I'll post updated patches.

    Regards,
    Alecsandru

    @alecsandrupatrascu
    Copy link
    Mannequin Author

    alecsandrupatrascu mannequin commented May 16, 2016

    Brett, I tried to add you in the "assigned to", but I do not have any rights to update those :(

    @brettcannon
    Copy link
    Member

    brettcannon commented May 16, 2016

    Just leave a comment next week in the issue if no one else beats me to the issue (I'm at OSCON so I don't have time to commit until May 24 at the earliest).

    @alecsandrupatrascu
    Copy link
    Mannequin Author

    alecsandrupatrascu mannequin commented May 31, 2016

    Hello Brett, I saw that there was no other activity on this issue so far. Can you please update this, as I do not have rights?

    @brettcannon brettcannon self-assigned this Jun 2, 2016
    @alecsandrupatrascu
    Copy link
    Mannequin Author

    alecsandrupatrascu mannequin commented Jun 3, 2016

    I've updated the patch for 2.7 branch to contain the LTO part merged by Gregory.

    @alecsandrupatrascu
    Copy link
    Mannequin Author

    alecsandrupatrascu mannequin commented Jun 3, 2016

    The patches in discussions are the v02 ones.

    @brettcannon
    Copy link
    Member

    brettcannon commented Aug 30, 2016

    Just FYI, Alecsandru, I plan on applying your patches this week or next, so if you can just double-check they still apply cleanly that would be great.

    @brettcannon brettcannon removed their assignment Sep 2, 2016
    @brettcannon
    Copy link
    Member

    brettcannon commented Sep 2, 2016

    So if I enable this manually, I get various test failures from modules on OS X El Capitan with the same type of failure:

    [ 95/398] test_bytes
    Traceback (most recent call last):
      File "/Users/brettcannon/Repositories/python/cpython/3.5/Lib/runpy.py", line 193, in _run_module_as_main
        "__main__", mod_spec)
      File "/Users/brettcannon/Repositories/python/cpython/3.5/Lib/runpy.py", line 85, in _run_code
        exec(code, run_globals)
      File "/Users/brettcannon/Repositories/python/cpython/3.5/Lib/test/__main__.py", line 3, in <module>
        regrtest.main_in_temp_cwd()
      File "/Users/brettcannon/Repositories/python/cpython/3.5/Lib/test/regrtest.py", line 1593, in main_in_temp_cwd
        main()
      File "/Users/brettcannon/Repositories/python/cpython/3.5/Lib/test/regrtest.py", line 756, in main
        raise Exception("Child error on {}: {}".format(test, result[1]))
    Exception: Child error on test_bytes: Exit code -6

    Since I can't get the test suite to pass with PGO+LTO I'm not comfortable committing this patch and promoting people turn them on.

    @gpshead
    Copy link
    Member

    gpshead commented Sep 6, 2016

    Minor edit to avoid printing the message about --with-optimizations when doing a --with-pydebug build. otherwise this patch works well on Ubuntu xenial 16.04 amd64.

    From discussions in the CPython Core Sprint room today - we've agreed that we should just modify the configure.ac check to not enable --with-lto as part of this when on MacOS until an xcode toolchain for that to work properly on can be figured out.

    @brettcannon
    Copy link
    Member

    brettcannon commented Sep 7, 2016

    Here is modification of Greg's patch to turn off LTO under OS X since LTO+PGO do not get along under El Capitan.

    @brettcannon
    Copy link
    Member

    brettcannon commented Sep 7, 2016

    Patch with the missing Makefile.pre.in part :P

    @python-dev
    Copy link
    Mannequin

    python-dev mannequin commented Sep 7, 2016

    New changeset b4b73473ecc6 by Brett Cannon in branch 'default':
    Issue bpo-26359: Add the --with-optimizations configure flag.
    https://hg.python.org/cpython/rev/b4b73473ecc6

    @python-dev
    Copy link
    Mannequin

    python-dev mannequin commented Sep 7, 2016

    New changeset 1e61cc86df03 by Brett Cannon in branch '3.5':
    Issue bpo-26359: Add the --with-optimizations flag.
    https://hg.python.org/cpython/rev/1e61cc86df03

    New changeset f9b52b31f1b4 by Brett Cannon in branch 'default':
    Merge for bpo-26359
    https://hg.python.org/cpython/rev/f9b52b31f1b4

    @brettcannon
    Copy link
    Member

    brettcannon commented Sep 7, 2016

    Leaving this open in case someone wants to backport to 2.7.

    @gpshead gpshead self-assigned this Sep 8, 2016
    @python-dev
    Copy link
    Mannequin

    python-dev mannequin commented Sep 8, 2016

    New changeset bc28cbd49070 by Gregory P. Smith in branch '2.7':
    Issue bpo-26359: Add the --with-optimizations flag.
    https://hg.python.org/cpython/rev/bc28cbd49070

    @gpshead gpshead closed this as completed Sep 8, 2016
    @ned-deily
    Copy link
    Member

    ned-deily commented Sep 8, 2016

    Thanks for the new option. Perhaps the top-level README should be updated to mention --with-optimizations?

    @python-dev
    Copy link
    Mannequin

    python-dev mannequin commented Sep 8, 2016

    New changeset e1987bf14148 by Gregory P. Smith in branch '3.5':
    document --with-optimizations in the README for bpo-26359.
    https://hg.python.org/cpython/rev/e1987bf14148

    New changeset 4afb12077e08 by Gregory P. Smith in branch 'default':
    document --with-optimizations in the README for bpo-26359.
    https://hg.python.org/cpython/rev/4afb12077e08

    @python-dev
    Copy link
    Mannequin

    python-dev mannequin commented Sep 8, 2016

    New changeset 8567bc2876af by Gregory P. Smith in branch '2.7':
    document --with-optimizations in the README for bpo-26359.
    https://hg.python.org/cpython/rev/8567bc2876af

    @gpshead
    Copy link
    Member

    gpshead commented Sep 8, 2016

    see https://bugs.python.org/issue28032 for flaws with --with-optimizations builds.

    @gpshead
    Copy link
    Member

    gpshead commented Nov 20, 2016

    per comments in bpo-28032 the new configure flag has been renamed from --with-optimizations to --enable-optimizations in all branches it was added to:

    remote: notified python-checkins@python.org of incoming changeset c0ea81315fb6
    remote: notified python-checkins@python.org of incoming changeset 58c1a49a10b4
    remote: notified python-checkins@python.org of incoming changeset 11cacf9f9a33
    remote: notified python-checkins@python.org of incoming changeset 0d2b42344ae5
    remote: notified python-checkins@python.org of incoming changeset 2d1d70b53376
    remote: notified python-checkins@python.org of incoming changeset d5ff5a2f33fd

    @vstinner
    Copy link
    Member

    vstinner commented Feb 24, 2017

    It seems like it's not possible to install Python configured with --enable-optimizations: see my issue bpo-29641, "make install failure when using ./configure --enable-optimizations.

    @ezio-melotti ezio-melotti transferred this issue from another repository Apr 10, 2022
    Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
    Labels
    build The build process and cross-build performance Performance or resource usage
    Projects
    None yet
    Development

    No branches or pull requests

    5 participants