Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Enhance the BOLT build process #101282

Closed
4 tasks done
Tracked by #101525
corona10 opened this issue Jan 24, 2023 · 5 comments
Closed
4 tasks done
Tracked by #101525

Enhance the BOLT build process #101282

corona10 opened this issue Jan 24, 2023 · 5 comments
Assignees
@corona10
Labels
release-blocker type-feature A feature request or enhancement

Comments

@corona10
Copy link
Member

corona10 commented Jan 24, 2023 •
edited by bedevere-bot

From @indygreg 's #101093

TODO

  • Optional -fno-reorder-blocks-and-partition
  • Drop -gdwarf-4
  • Run BOLT instrumented binary with `$(RUNSHARED)``
  • Enable BOLT optimization of libpython

Linked PRs
@corona10 corona10 added the type-feature A feature request or enhancement label Jan 24, 2023
@corona10 corona10 mentioned this issue Jan 24, 2023
Draft
@corona10
Copy link
Member Author

cc @aaupov

@corona10 corona10 assigned corona10 and unassigned corona10 Jan 24, 2023
corona10 added a commit to corona10/cpython that referenced this issue Feb 4, 2023
@corona10
pythongh-101282: Update BOLT --split-functions flag not to use deprec…
9756331 
…ated usage
@bedevere-bot bedevere-bot mentioned this issue Feb 4, 2023
Merged
corona10 added a commit that referenced this issue Feb 4, 2023
@corona10
gh-101282: Update BOLT --split-functions flag not to use deprecated u… (
144aaa7 
gh-101557)

gh-101282: Update BOLT --split-functions flag not to use deprecated usage
@corona10 corona10 mentioned this issue Apr 2, 2023
Open
6 tasks
@corona10 corona10 mentioned this issue May 15, 2023
Closed
@bedevere-bot bedevere-bot mentioned this issue May 15, 2023
Merged
corona10 pushed a commit that referenced this issue May 15, 2023
@indygreg
gh-101282: move BOLT config after PGO (gh-104493)
27d8ecd
carljm added a commit to carljm/cpython that referenced this issue May 15, 2023
@carljm
Merge branch 'main' into supermon
343867a 
* main: (29 commits)
  pythongh-101819: Fix _io clinic input for unused base class method stubs (python#104418)
  pythongh-101819: Isolate `_io` (python#101948)
  Bump mypy from 1.2.0 to 1.3.0 in /Tools/clinic (python#104501)
  pythongh-104494: Update certain Tkinter pack/place tests for Tk 8.7 errors (python#104495)
  pythongh-104050: Run mypy on `clinic.py` in CI (python#104421)
  pythongh-104490: Consistently define phony make targets (python#104491)
  pythongh-67056: document that registering/unregistering an atexit func from within an atexit func is undefined (python#104473)
  pythongh-104487: PYTHON_FOR_REGEN must be minimum Python 3.10 (python#104488)
  pythongh-101282: move BOLT config after PGO (pythongh-104493)
  pythongh-104469 Convert _testcapi/float.c to use AC (pythongh-104470)
  pythongh-104456: Fix ref leak in _ctypes.COMError (python#104457)
  pythongh-98539: Make _SSLTransportProtocol.abort() safe to call when closed (python#104474)
  pythongh-104337: Clarify random.gammavariate doc entry  (python#104410)
  Minor improvements to typing docs (python#104465)
  pythongh-87092: avoid gcc warning on uninitialized struct field in assemble.c (python#104460)
  pythonGH-71383: IDLE - Document testing subsets of modules (python#104463)
  pythongh-104454: Fix refleak in AttributeError_reduce (python#104455)
  pythongh-75710: IDLE - add docstrings and comments to editor module (python#104446)
  pythongh-91896: Revert some very noisy DeprecationWarnings for `ByteString` (python#104424)
  Add a mention of PYTHONBREAKPOINT to breakpoint() docs (python#104430)
  ...
carljm added a commit to carljm/cpython that referenced this issue May 15, 2023
@carljm
Merge branch 'main' into cached_property_docs
452d9f5 
* main: (204 commits)
  pythongh-101819: Fix _io clinic input for unused base class method stubs (python#104418)
  pythongh-101819: Isolate `_io` (python#101948)
  Bump mypy from 1.2.0 to 1.3.0 in /Tools/clinic (python#104501)
  pythongh-104494: Update certain Tkinter pack/place tests for Tk 8.7 errors (python#104495)
  pythongh-104050: Run mypy on `clinic.py` in CI (python#104421)
  pythongh-104490: Consistently define phony make targets (python#104491)
  pythongh-67056: document that registering/unregistering an atexit func from within an atexit func is undefined (python#104473)
  pythongh-104487: PYTHON_FOR_REGEN must be minimum Python 3.10 (python#104488)
  pythongh-101282: move BOLT config after PGO (pythongh-104493)
  pythongh-104469 Convert _testcapi/float.c to use AC (pythongh-104470)
  pythongh-104456: Fix ref leak in _ctypes.COMError (python#104457)
  pythongh-98539: Make _SSLTransportProtocol.abort() safe to call when closed (python#104474)
  pythongh-104337: Clarify random.gammavariate doc entry  (python#104410)
  Minor improvements to typing docs (python#104465)
  pythongh-87092: avoid gcc warning on uninitialized struct field in assemble.c (python#104460)
  pythonGH-71383: IDLE - Document testing subsets of modules (python#104463)
  pythongh-104454: Fix refleak in AttributeError_reduce (python#104455)
  pythongh-75710: IDLE - add docstrings and comments to editor module (python#104446)
  pythongh-91896: Revert some very noisy DeprecationWarnings for `ByteString` (python#104424)
  Add a mention of PYTHONBREAKPOINT to breakpoint() docs (python#104430)
  ...
indygreg added a commit to indygreg/cpython that referenced this issue May 20, 2023
@indygreg
pythongh-101282: rework the BOLT build process
f7851cd 
(This change is a quick and dirty way to merge some of the build system
improvements I'm proposing in pythongh-101093 before the 3.12 feature freeze.
I wanted to scope bloat myself to fix some longstanding deficiencies in
the build system around profile-guided builds. But I'm getting soft
resistance to the reviews so close to the freeze deadline and it is
obvious that we need a simpler solution to hit the 3.12 deadline. While
this change is quick and dirty, it attempts to not make things worse.)

Before this change, we only applied bolt to the main python binary.
After this change, we apply bolt to libpython if it is configured. In
shared library builds, most of the C code is in libpython so it is
critical to apply bolt to libpython to realize bolt benefits.

This change also reworks how bolt instrumentation is applied. It
effectively removes the readelf based logic added in pythongh-101525 and
replaces it with a mechanism that saves a copy of the pre-bolt binary
and restores that copy when necessary. This allows us to perform
bolt optimizations without having to manually delete the output binary
to force a new bolt run.

We also add a new make target for purging bolt files and hook it up
to `clean` so bolt state is purged when appropriate.

`.gitignore` rules have been added to ignore files related to bolt.

Before and after this refactor, `make` will no-op after a previous run.
Both versions should also share common make DAG deficiencies where
targets fail to trigger as often as they need to or can trigger
prematurely in certain scenarios. e.g. after this change you may need
to `rm profile-bolt-stamp` to force a bolt run because there aren't
appropriate non-phony targets for bolt's make target to depend on.
Fixing this is a non-trivial amount of work that will likely have to
wait until the 3.13 window.

To make it easier to iterate on custom BOLT settings, the flags to
pass to instrumentation and application are now defined in configure
and can be overridden by passing `BOLT_INSTRUMENT_FLAGS` and
`BOLT_APPLY_FLAGS`.
@bedevere-bot bedevere-bot mentioned this issue May 20, 2023
Merged
@indygreg
Copy link
Contributor

The remaining 2 tasks in this issue are implemented by gh-104709. Unsure if we want to call this issue done after that or not.

I still have a ton of patches to overhaul the profile-related make rules (gh-104523). But those will have to wait until 3.13.

@corona10
Copy link
Member Author

The remaining 2 tasks in this issue are implemented by #104709.

Thanks for the work, let's ship the feature at CPython 3.12 as the experimental feature.

indygreg added a commit to indygreg/cpython that referenced this issue May 21, 2023
@indygreg
pythongh-101282: rework the BOLT build process
555686d 
(This change is a quick and dirty way to merge some of the build system
improvements I'm proposing in pythongh-101093 before the 3.12 feature freeze.
I wanted to scope bloat myself to fix some longstanding deficiencies in
the build system around profile-guided builds. But I'm getting soft
resistance to the reviews so close to the freeze deadline and it is
obvious that we need a simpler solution to hit the 3.12 deadline. While
this change is quick and dirty, it attempts to not make things worse.)

Before this change, we only applied bolt to the main python binary.
After this change, we apply bolt to libpython if it is configured. In
shared library builds, most of the C code is in libpython so it is
critical to apply bolt to libpython to realize bolt benefits.

This change also reworks how bolt instrumentation is applied. It
effectively removes the readelf based logic added in pythongh-101525 and
replaces it with a mechanism that saves a copy of the pre-bolt binary
and restores that copy when necessary. This allows us to perform
bolt optimizations without having to manually delete the output binary
to force a new bolt run.

We also add a new make target for purging bolt files and hook it up
to `clean` so bolt state is purged when appropriate.

`.gitignore` rules have been added to ignore files related to bolt.

Before and after this refactor, `make` will no-op after a previous run.
Both versions should also share common make DAG deficiencies where
targets fail to trigger as often as they need to or can trigger
prematurely in certain scenarios. e.g. after this change you may need
to `rm profile-bolt-stamp` to force a bolt run because there aren't
appropriate non-phony targets for bolt's make target to depend on.
Fixing this is a non-trivial amount of work that will likely have to
wait until the 3.13 window.

To make it easier to iterate on custom BOLT settings, the flags to
pass to instrumentation and application are now defined in configure
and can be overridden by passing `BOLT_INSTRUMENT_FLAGS` and
`BOLT_APPLY_FLAGS`.
erlend-aasland pushed a commit that referenced this issue May 22, 2023
@indygreg
gh-101282: Apply BOLT optimizations to libpython for shared builds (#…
5360cb3 
…104709)

Apply BOLT optimizations to libpython for shared builds. Most of the C
code is in libpython so it is critical to apply BOLT there fully realize
BOLT benefits.

This change also reworks how BOLT instrumentation is applied. It
effectively removes the readelf based logic added in gh-101525 and
replaces it with a mechanism that saves a copy of the pre-bolt binary
and restores that copy when necessary. This allows us to perform BOLT
optimizations without having to manually delete the output binary to
force a new bolt run.

Also:
- add a clean-bolt target for purging BOLT files and hook that up to the
  clean target
- .gitignore BOLT related files

Before and after this refactor, `make` will no-op after a previous run.
Both versions should also share common make DAG deficiencies where
targets fail to trigger as often as they need to or can trigger
prematurely in certain scenarios. e.g. after this change you may need to
`rm profile-bolt-stamp` to force a BOLT run because there aren't
appropriate non-phony targets for BOLT's make target to depend on.

To make it easier to iterate on custom BOLT settings, the flags to pass
to instrumentation and application are now defined in configure and can
be overridden by passing BOLT_INSTRUMENT_FLAGS and BOLT_APPLY_FLAGS.
@erlend-aasland
Copy link
Contributor

@indygreg, looks like this part can be marked completed.

corona10 added a commit to corona10/cpython that referenced this issue May 22, 2023
@corona10
pythongh-101282: Revert some M4 style for BOLT configuration
baa1318
This was referenced May 22, 2023
Closed
Merged
corona10 added a commit to corona10/cpython that referenced this issue May 22, 2023
@corona10
pythongh-101282: Fix M4 style for BOLT configuration
e9597e5
@bedevere-bot bedevere-bot mentioned this issue May 23, 2023
Closed
erlend-aasland added a commit to erlend-aasland/cpython that referenced this issue May 23, 2023
@erlend-aasland
pythongh-101282: Group and quote BOLT CLI flags
c14ec85
@corona10 corona10 reopened this May 24, 2023
@corona10
Copy link
Member Author

corona10 commented May 24, 2023 •
edited

Release blocker for CPython 3.12 beta2

candidate PRs

corona10 added a commit that referenced this issue May 24, 2023
@corona10
gh-101282: Enclose BOLT_APPLY_FLAGS value in double quotes (gh-104752)
c437851
corona10 added a commit to corona10/cpython that referenced this issue May 24, 2023
@corona10
[3.12] pythongh-101282: Enclose BOLT_APPLY_FLAGS value in double quot…
36f5894 
…es (pythongh-104752)

(cherry picked from commit c437851)

Co-authored-by: Dong-hee Na <donghee.na@python.org>
@bedevere-bot bedevere-bot mentioned this issue May 24, 2023
Merged
corona10 added a commit that referenced this issue May 24, 2023
@corona10
[3.12] gh-101282: Enclose BOLT_APPLY_FLAGS value in double quotes (gh… (
b151660 
gh-104853)

[3.12] gh-101282: Enclose BOLT_APPLY_FLAGS value in double quotes (gh-104752)
(cherry picked from commit c437851)
@erlend-aasland erlend-aasland mentioned this issue Jun 6, 2023
Closed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@corona10 corona10

Labels
release-blocker type-feature A feature request or enhancement
Projects
Release and Deferred blockers 🚫
Archived in project
Milestone
No milestone
Development

No branches or pull requests
3 participants
@indygreg @corona10 @erlend-aasland