Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

bpo-28015: Support LTO build with clang #9908

Merged
merged 4 commits into from Oct 24, 2018

Conversation

Projects
None yet
6 participants
@serge-sans-paille
Copy link
Contributor

commented Oct 16, 2018

.o generated by clang in LTO mode actually are LLVM bitcode files, which
leads to a few errors during configure/build step:

  • add lto flags to the BASECFLAGS instead of CFLAGS, as CFLAGS are used
    to build autoconf test case, and some are not compatible with clang LTO
    (they assume binary in the .o, not bitcode)
  • force llvm-ar instead of ar, as ar is not aware of .o files generated
    by clang -flto

https://bugs.python.org/issue28015

Support LTO build with clang
.o generated by clang in LTO mode actually are LLVM bitcode files, which
leads to a few errors during configure/build step:

- add lto flags to the BASECFLAGS instead of CFLAGS, as CFLAGS are used
  to build autoconf test case, and some are not compatible with clang LTO
  (they assume binary in the .o, not bitcode)
- force llvm-ar instead of ar, as ar is not aware of .o files generated
  by clang -flto
@the-knights-who-say-ni

This comment has been minimized.

Copy link

commented Oct 16, 2018

Hello, and thanks for your contribution!

I'm a bot set up to make sure that the project can legally accept your contribution by verifying you have signed the PSF contributor agreement (CLA).

Unfortunately we couldn't find an account corresponding to your GitHub username on bugs.python.org (b.p.o) to verify you have signed the CLA (this might be simply due to a missing "GitHub Name" entry in your b.p.o account settings). This is necessary for legal reasons before we can look at your contribution. Please follow the steps outlined in the CPython devguide to rectify this issue.

You can check yourself to see if the CLA has been received.

Thanks again for your contribution, we look forward to reviewing it!

@serge-sans-paille

This comment has been minimized.

Copy link
Contributor Author

commented Oct 16, 2018

CLA just signed, that should be ok now.

I did not commit the autoreconf-generated file, don't know what's the procedure there.

@vstinner
Copy link
Member

left a comment

Python source code includes configure for practical reasons, you must run autoconf and include the modified configure in your PR.

@bedevere-bot

This comment has been minimized.

Copy link

commented Oct 16, 2018

A Python core developer has requested some changes be made to your pull request before we can consider merging it. If you could please address their requests along with any other requests in other reviews from core developers that would be appreciated.

Once you have made the requested changes, please leave a comment on this pull request containing the phrase I have made the requested changes; please review again. I will then notify any core developers who have left a review that you're ready for them to take another look at this pull request.

@vstinner
Copy link
Member

left a comment

It would be convenient if configure fails with an error and an explanation how to fix it if llvm-ar is missing:

AC_PATH_TARGET_TOOL(LLVM_AR, llvm-ar, '', ${llvm_path})
(...)
AR="${LLVM_AR}"
@vstinner

This comment has been minimized.

Copy link
Member

commented Oct 16, 2018

I tested the 3rd version. It seems like -flto is passed during the compilation of object files and to the final linker step:

$ grep lto Makefile
BASECFLAGS=	 -flto -g -Wno-unused-result -Wsign-compare
CONFIGURE_LDFLAGS=	 -flto -g
CONFIG_ARGS=	 '--with-pydebug' 'CC=clang' '--with-lto'

$ make
...
clang -pthread -fPIC -flto (...)
...
/usr/bin/llvm-ar rcs libpython3.8dm.a Modules/getbuildinfo.o (...) Python/frozen.o
clang -pthread -flto -g  -Xlinker -export-dynamic -o python Programs/python.o libpython3.8dm.a -lpthread -ldl  -lutil -lm   -lm 
@vstinner
Copy link
Member

left a comment

LGTM. It works as expected (see my comment above) and it fix an obvious bug https://bugs.python.org/issue28015#msg327817

@vstinner

This comment has been minimized.

Copy link
Member

commented Oct 16, 2018

You may add a NEWS entry using the tool "blurb".

@serge-sans-paille signed the CLA, but I have to wait until someone review his signature.

@vstinner

This comment has been minimized.

Copy link
Member

commented Oct 16, 2018

I will remove the "CLA not signed" label once "Contrib Form Received" will become Yes on https://bugs.python.org/user22883

@pablogsal pablogsal requested a review from gpshead Oct 17, 2018

@vstinner vstinner merged commit 5ad36f9 into python:master Oct 24, 2018

5 checks passed

Azure Pipelines PR #20181016.11 succeeded
Details
bedevere/issue-number Issue number 28015 found
Details
bedevere/news News entry found in Misc/NEWS.d
continuous-integration/appveyor/pr AppVeyor build succeeded
Details
continuous-integration/travis-ci/pr The Travis CI build passed
Details
@miss-islington

This comment has been minimized.

Copy link

commented Oct 25, 2018

Thanks @serge-sans-paille for the PR, and @vstinner for merging it 🌮🎉.. I'm working now to backport this PR to: 3.7.
🐍🍒🤖 I'm not a witch! I'm not a witch!

@bedevere-bot

This comment has been minimized.

Copy link

commented Oct 25, 2018

GH-10089 is a backport of this pull request to the 3.7 branch.

miss-islington added a commit to miss-islington/cpython that referenced this pull request Oct 25, 2018

bpo-28015: Support LTO build with clang (pythonGH-9908)
.o generated by clang in LTO mode actually are LLVM bitcode files, which
leads to a few errors during configure/build step:

- add lto flags to the BASECFLAGS instead of CFLAGS, as CFLAGS are used
  to build autoconf test case, and some are not compatible with clang LTO
  (they assume binary in the .o, not bitcode)
- force llvm-ar instead of ar, as ar is not aware of .o files generated
  by clang -flto
(cherry picked from commit 5ad36f9)

Co-authored-by: serge-sans-paille <serge.guelton@telecom-bretagne.eu>
@vstinner

This comment has been minimized.

Copy link
Member

commented Oct 25, 2018

@serge-sans-paille: The backport to 3.7 was straighfoward and could be automated, but backport to 3.6 creates a conflict on configure.ac. Do you want to try the backport? I'm not sure if it's worth it to backport the change up to 3.6. It's up to you.

miss-islington added a commit that referenced this pull request Oct 25, 2018

bpo-28015: Support LTO build with clang (GH-9908)
.o generated by clang in LTO mode actually are LLVM bitcode files, which
leads to a few errors during configure/build step:

- add lto flags to the BASECFLAGS instead of CFLAGS, as CFLAGS are used
  to build autoconf test case, and some are not compatible with clang LTO
  (they assume binary in the .o, not bitcode)
- force llvm-ar instead of ar, as ar is not aware of .o files generated
  by clang -flto
(cherry picked from commit 5ad36f9)

Co-authored-by: serge-sans-paille <serge.guelton@telecom-bretagne.eu>
@serge-sans-paille

This comment has been minimized.

Copy link
Contributor Author

commented Oct 29, 2018

@vstinner I think I have the 3.6 patch, should I create a new PR on the 3.6 branch?

@serge-sans-paille

This comment has been minimized.

Copy link
Contributor Author

commented Oct 29, 2018

Finally, there a re many complex interactions with RANLIB and AR, dropping the backport.

@vstinner

This comment has been minimized.

Copy link
Member

commented Oct 29, 2018

Finally, there a re many complex interactions with RANLIB and AR, dropping the backport.

The Python 3.6 backport? It's too complex? It's ok to only fix Python 3.7 and newer. (I just need your confirmation to close the issue.)

@serge-sans-paille

This comment has been minimized.

Copy link
Contributor Author

commented Oct 29, 2018

stratakis added a commit to stratakis/cpython that referenced this pull request Oct 31, 2018

[3.6] bpo-28015: Support LTO build with clang (pythonGH-9908)
.o generated by clang in LTO mode actually are LLVM bitcode files, which
leads to a few errors during configure/build step:

- add lto flags to the BASECFLAGS instead of CFLAGS, as CFLAGS are used
  to build autoconf test case, and some are not compatible with clang LTO
  (they assume binary in the .o, not bitcode)
- force llvm-ar instead of ar, as ar is not aware of .o files generated
  by clang -flto
(cherry picked from commit 5ad36f9)

Co-authored-by: serge-sans-paille <serge.guelton@telecom-bretagne.eu>

stratakis added a commit to stratakis/cpython that referenced this pull request Oct 31, 2018

[3.6] bpo-28015: Support LTO build with clang (pythonGH-9908)
.o generated by clang in LTO mode actually are LLVM bitcode files, which
leads to a few errors during configure/build step:

- add lto flags to the BASECFLAGS instead of CFLAGS, as CFLAGS are used
  to build autoconf test case, and some are not compatible with clang LTO
  (they assume binary in the .o, not bitcode)
- force llvm-ar instead of ar, as ar is not aware of .o files generated
  by clang -flto
@stratakis

This comment has been minimized.

Copy link
Contributor

commented Oct 31, 2018

The 3.6 backport works fine if https://bugs.python.org/issue31625 is applied for 3.6 as well.

yahya-abou-imran added a commit to yahya-abou-imran/cpython that referenced this pull request Nov 2, 2018

bpo-28015: Support LTO build with clang (pythonGH-9908)
.o generated by clang in LTO mode actually are LLVM bitcode files, which
leads to a few errors during configure/build step:

- add lto flags to the BASECFLAGS instead of CFLAGS, as CFLAGS are used
  to build autoconf test case, and some are not compatible with clang LTO
  (they assume binary in the .o, not bitcode)
- force llvm-ar instead of ar, as ar is not aware of .o files generated
  by clang -flto

stratakis added a commit to stratakis/cpython that referenced this pull request Dec 5, 2018

bpo-28015: Support LTO build with clang (pythonGH-9908)
.o generated by clang in LTO mode actually are LLVM bitcode files, which
leads to a few errors during configure/build step:

- add lto flags to the BASECFLAGS instead of CFLAGS, as CFLAGS are used
  to build autoconf test case, and some are not compatible with clang LTO
  (they assume binary in the .o, not bitcode)
- force llvm-ar instead of ar, as ar is not aware of .o files generated
  by clang -flto
(cherry picked from commit 5ad36f9)

Co-authored-by: serge-sans-paille <serge.guelton@telecom-bretagne.eu>

ned-deily added a commit that referenced this pull request Dec 9, 2018

bpo-28015: Support LTO build with clang (GH-9908) (GH-10922)
.o generated by clang in LTO mode actually are LLVM bitcode files, which
leads to a few errors during configure/build step:

- add lto flags to the BASECFLAGS instead of CFLAGS, as CFLAGS are used
  to build autoconf test case, and some are not compatible with clang LTO
  (they assume binary in the .o, not bitcode)
- force llvm-ar instead of ar, as ar is not aware of .o files generated
  by clang -flto
(cherry picked from commit 5ad36f9)

Co-authored-by: serge-sans-paille <serge.guelton@telecom-bretagne.eu>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
You can’t perform that action at this time.