Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

fix: Use CMake 3.12+ FindPython to get Python components #1906

Merged
merged 2 commits into from
Feb 15, 2023

Conversation

matthewfeickert
Copy link
Contributor

@matthewfeickert matthewfeickert commented Feb 14, 2023

Resolves #1474

Demonstration of fixing Issue 1474:
$ docker run --rm -ti -u root matthewfeickert/pyenv-virtualenv-conda:pyenv-v2.3.12       
root@8f5f7b99e0cf:~/data# pyenv virtualenv 3.11.1 issue-1474 && pyenv activate issue-1474
(issue-1474) root@8f5f7b99e0cf:~/data# apt update && apt install -y cmake python3-dev
(issue-1474) root@8f5f7b99e0cf:~/data# python -m pip --quiet install --upgrade pip setuptools wheel
(issue-1474) root@8f5f7b99e0cf:~/data# git clone --recursive https://github.com/matthewfeickert/xrootd.git && cd xrootd
(issue-1474) root@8f5f7b99e0cf:~/data/xrootd# git checkout origin/fix/use-modern-find-package -b fix/use-modern-find-package
(issue-1474) root@8f5f7b99e0cf:~/data/xrootd# cp packaging/wheel/* .
(issue-1474) root@8f5f7b99e0cf:~/data/xrootd# ./publish.sh
(issue-1474) root@8f5f7b99e0cf:~/data/xrootd# python -m pip install ./dist/xrootd-*.tar.gz
(issue-1474) root@8f5f7b99e0cf:~/data/xrootd# python -m pip list
Package         Version
--------------- -------------
build           0.10.0
packaging       23.0
pip             23.0
pyproject_hooks 1.0.0
setuptools      67.3.1
wheel           0.38.4
xrootd          5.5.2.post169
(issue-1474) root@8f5f7b99e0cf:~/data/xrootd# python -m pip show xrootd
Name: xrootd
Version: 5.5.2.post169
Summary: XRootD Python bindings
Home-page: http://xrootd.org
Author: XRootD Developers
Author-email: xrootd-dev@slac.stanford.edu
License: LGPLv3+
Location: /home/docker/.pyenv/versions/issue-1474/lib/python3.11/site-packages
Requires: 
Required-by: 
(issue-1474) root@8f5f7b99e0cf:~/data/xrootd#

@matthewfeickert
Copy link
Contributor Author

This PR is currently failing on the rpm-fedora build but passing all others. If @amadio and @henryiii can offer advice later this week (I have to write for the rest of Tuesday 2023-02-14) then I think we can knock out Issue #1474 for good and speed up some things.

@@ -12,7 +12,7 @@ component_status( TESTS BUILD_TESTS CPPUNIT_FOUND )
component_status( HTTP BUILD_HTTP OPENSSL_FOUND )
component_status( TPC BUILD_TPC CURL_FOUND )
component_status( MACAROONS BUILD_MACAROONS MACAROONS_FOUND )
component_status( PYTHON BUILD_PYTHON PYTHON_FOUND )
component_status( PYTHON BUILD_PYTHON Python_FOUND )
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This should probably match the other check, i.e.

Python_Interpreter_FOUND AND Python_Development_FOUND

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@amadio I updated this from Python_FOUND to Python_Interpreter_FOUND AND Python_Development_FOUND as you suggested though note from https://cmake.org/cmake/help/v3.12/module/FindPython.html#result-variables that Python_FOUND is a boolean of if

System has the Python requested components

and so would be more robust to changes in the associated find_package command.

@matthewfeickert
Copy link
Contributor Author

matthewfeickert commented Feb 14, 2023

I have very limited experience doing anything with rpmbuild, so @amadio and @simonmichal do you have any advice on debugging these build errors that are only happening for fedora?

errors from rpm-fedora CI logs:
g++ -shared -Wl,-z,relro -Wl,--as-needed -Wl,-z,now -Wl,--build-id=sha1 -g -Wl,-z,relro -Wl,--as-needed -Wl,-z,now -Wl,--build-id=sha1 -g -Wl,-rpath,$ORIGIN/../../.. -Wl,-z,relro -Wl,--as-needed -Wl,-z,now -specs=/usr/lib/rpm/redhat/redhat-hardened-ld -specs=/usr/lib/rpm/redhat/redhat-annobin-cc1 -Wl,--build-id=sha1 -specs=/usr/lib/rpm/redhat/redhat-package-notes -O2 -flto=auto -ffat-lto-objects -fexceptions -g -grecord-gcc-switches -pipe -Wall -Werror=format-security -Wp,-D_FORTIFY_SOURCE=2 -Wp,-D_GLIBCXX_ASSERTIONS -specs=/usr/lib/rpm/redhat/redhat-hardened-cc1 -fstack-protector-strong -specs=/usr/lib/rpm/redhat/redhat-annobin-cc1 -m64 -mtune=generic -fasynchronous-unwind-tables -fstack-clash-protection -fcf-protection build/temp.linux-x86_64-cpython-311/github/home/rpmbuild/BUILD/xrootd/xrootd/bindings/python/src/PyXRootDCopyProcess.o build/temp.linux-x86_64-cpython-311/github/home/rpmbuild/BUILD/xrootd/xrootd/bindings/python/src/PyXRootDCopyProgressHandler.o build/temp.linux-x86_64-cpython-311/github/home/rpmbuild/BUILD/xrootd/xrootd/bindings/python/src/PyXRootDFile.o build/temp.linux-x86_64-cpython-311/github/home/rpmbuild/BUILD/xrootd/xrootd/bindings/python/src/PyXRootDFileSystem.o build/temp.linux-x86_64-cpython-311/github/home/rpmbuild/BUILD/xrootd/xrootd/bindings/python/src/PyXRootDModule.o build/temp.linux-x86_64-cpython-311/github/home/rpmbuild/BUILD/xrootd/xrootd/bindings/python/src/PyXRootDURL.o build/temp.linux-x86_64-cpython-311/github/home/rpmbuild/BUILD/xrootd/xrootd/bindings/python/src/Utils.o -L/github/home/rpmbuild/BUILD/xrootd/xrootd/build/src -L/github/home/rpmbuild/BUILD/xrootd/xrootd/build/src/XrdCl -L/usr/lib64 -lXrdCl -lXrdUtils -ldl -o build/lib.linux-x86_64-cpython-311/pyxrootd/client.cpython-311-x86_64-linux-gnu.so
+ popd
~/rpmbuild/BUILD/xrootd/xrootd
+ RPM_EC=0
++ jobs -p
+ exit 0
Executing(%install): /bin/sh -e /var/tmp/rpm-tmp.ITHyv2
+ umask 022
+ cd /github/home/rpmbuild/BUILD
+ '[' /github/home/rpmbuild/BUILDROOT/xrootd-5.5.2.post167-1.fc37.x86_64 '!=' / ']'
+ rm -rf /github/home/rpmbuild/BUILDROOT/xrootd-5.5.2.post167-1.fc37.x86_64
++ dirname /github/home/rpmbuild/BUILDROOT/xrootd-5.5.2.post167-1.fc37.x86_64
+ mkdir -p /github/home/rpmbuild/BUILDROOT
+ mkdir /github/home/rpmbuild/BUILDROOT/xrootd-5.5.2.post167-1.fc37.x86_64
+ CFLAGS='-O2 -flto=auto -ffat-lto-objects -fexceptions -g -grecord-gcc-switches -pipe -Wall -Werror=format-security -Wp,-D_FORTIFY_SOURCE=2 -Wp,-D_GLIBCXX_ASSERTIONS -specs=/usr/lib/rpm/redhat/redhat-hardened-cc1 -fstack-protector-strong -specs=/usr/lib/rpm/redhat/redhat-annobin-cc1  -m64  -mtune=generic -fasynchronous-unwind-tables -fstack-clash-protection -fcf-protection'
+ export CFLAGS
+ CXXFLAGS='-O2 -flto=auto -ffat-lto-objects -fexceptions -g -grecord-gcc-switches -pipe -Wall -Werror=format-security -Wp,-D_FORTIFY_SOURCE=2 -Wp,-D_GLIBCXX_ASSERTIONS -specs=/usr/lib/rpm/redhat/redhat-hardened-cc1 -fstack-protector-strong -specs=/usr/lib/rpm/redhat/redhat-annobin-cc1  -m64  -mtune=generic -fasynchronous-unwind-tables -fstack-clash-protection -fcf-protection'
+ export CXXFLAGS
+ FFLAGS='-O2 -flto=auto -ffat-lto-objects -fexceptions -g -grecord-gcc-switches -pipe -Wall -Werror=format-security -Wp,-D_FORTIFY_SOURCE=2 -Wp,-D_GLIBCXX_ASSERTIONS -specs=/usr/lib/rpm/redhat/redhat-hardened-cc1 -fstack-protector-strong -specs=/usr/lib/rpm/redhat/redhat-annobin-cc1  -m64  -mtune=generic -fasynchronous-unwind-tables -fstack-clash-protection -fcf-protection -I/usr/lib64/gfortran/modules'
+ export FFLAGS
+ FCFLAGS='-O2 -flto=auto -ffat-lto-objects -fexceptions -g -grecord-gcc-switches -pipe -Wall -Werror=format-security -Wp,-D_FORTIFY_SOURCE=2 -Wp,-D_GLIBCXX_ASSERTIONS -specs=/usr/lib/rpm/redhat/redhat-hardened-cc1 -fstack-protector-strong -specs=/usr/lib/rpm/redhat/redhat-annobin-cc1  -m64  -mtune=generic -fasynchronous-unwind-tables -fstack-clash-protection -fcf-protection -I/usr/lib64/gfortran/modules'
+ export FCFLAGS
+ LDFLAGS='-Wl,-z,relro -Wl,--as-needed  -Wl,-z,now -specs=/usr/lib/rpm/redhat/redhat-hardened-ld -specs=/usr/lib/rpm/redhat/redhat-annobin-cc1  -Wl,--build-id=sha1 -specs=/usr/lib/rpm/redhat/redhat-package-notes'
+ export LDFLAGS
+ LT_SYS_LIBRARY_PATH=/usr/lib64:
+ export LT_SYS_LIBRARY_PATH
~/rpmbuild/BUILD/xrootd/xrootd ~/rpmbuild/BUILD/xrootd
~/rpmbuild/BUILD/xrootd/xrootd/build ~/rpmbuild/BUILD/xrootd/xrootd ~/rpmbuild/BUILD/xrootd
+ CC=gcc
+ export CC
+ CXX=g++
+ export CXX
+ cd xrootd
+ rm -rf /github/home/rpmbuild/BUILDROOT/xrootd-5.5.2.post167-1.fc37.x86_64
+ pushd xrootd
+ pushd build
+ make install DESTDIR=/github/home/rpmbuild/BUILDROOT/xrootd-5.5.2.post167-1.fc37.x86_64
[I] No git repository info found. Trying to interpret VERSION_INFO
[I] The XrdVersion.hh file already exists
[  0%] Built target XrdVersion.hh
[ 23%] Built target XrdUtils
[ 24%] Built target XrdXml
[ 32%] Built target XrdCl
[ 35%] Built target XrdPosix
[ 35%] Built target xrdadler32
[ 36%] Built target xrdcks
[ 36%] Built target xrdcrc32c
[ 36%] Built target cconfig
[ 36%] Built target XrdAppUtils
[ 36%] Built target mpxstats
[ 36%] Built target xrdprep
[ 37%] Built target wait41
[ 54%] Built target XrdServer
[ 54%] Built target xrdacctest
[ 55%] Built target xrdmapc
[ 56%] Built target xrdpinls
[ 57%] Built target xrdqstats
[ 58%] Built target XrdClProxyPlugin-5
[ 58%] Built target XrdClRecorder-5
[ 58%] Built target xrdreplay
[ 60%] Built target XrdCrypto
[ 61%] Built target XrdCryptoLite
[ 62%] Built target XrdCryptossl-5
[ 62%] Built target XrdPosixPreload
[ 63%] Built target XrdSec-5
[ 63%] Built target XrdSecProt-5
[ 63%] Built target XrdSecpwd-5
[ 63%] Built target xrdpwdadmin
[ 63%] Built target XrdSecsss-5
[ 63%] Built target xrdsssadmin
[ 64%] Built target XrdSecunix-5
[ 65%] Built target XrdSecgsi-5
[ 65%] Built target XrdSecgsiAUTHZVO-5
[ 65%] Built target XrdSecgsiGMAPDN-5
[ 65%] Built target xrdgsiproxy
[ 65%] Built target xrdgsitest
[ 66%] Built target XrdSeckrb5-5
[ 67%] Built target XrdXrootd-5
[ 68%] Built target xrootd
[ 68%] Linking CXX executable cmsd
/github/home/rpmbuild/BUILD/xrootd/xrootd/src/XrdCms/XrdCmsRRQ.cc: In function 'sendRedResp.constprop':
/github/home/rpmbuild/BUILD/xrootd/xrootd/src/XrdCms/XrdCmsRRQ.cc:370:23: error: 'port' may be used uninitialized [-Werror=maybe-uninitialized]
  370 |       {redrResp.Val = htonl(port);
      |                       ^
/github/home/rpmbuild/BUILD/xrootd/xrootd/src/XrdCms/XrdCmsRRQ.cc:363:17: note: 'port' was declared here
  363 |    int doredir, port, hlen = 0;
      |                 ^
lto1: all warnings being treated as errors
make[3]: *** [/tmp/ccb9s0vQ.mk:5: /tmp/ccwnhEpT.ltrans1.ltrans.o] Error 1
make[3]: *** Waiting for unfinished jobs....
lto-wrapper: fatal error: make returned 2 exit status
compilation terminated.
/usr/bin/ld: error: lto-wrapper failed
collect2: error: ld returned 1 exit status
make[2]: *** [src/CMakeFiles/cmsd.dir/build.make:499: src/cmsd] Error 1
make[1]: *** [CMakeFiles/Makefile2:1454: src/CMakeFiles/cmsd.dir/all] Error 2
make: *** [Makefile:136: all] Error 2

error: Bad exit status from /var/tmp/rpm-tmp.ITHyv2 (%install)
RPM build warnings:

RPM build errors:
    line 256: It's not recommended to have unversioned Obsoletes: Obsoletes:  xrootd-macaroons
    line 257: It's not recommended to have unversioned Obsoletes: Obsoletes:  xrootd-tpc
    line 428: It's not recommended to have unversioned Obsoletes: Obsoletes: xrootd-voms-plugin
    Bad exit status from /var/tmp/rpm-tmp.ITHyv2 (%install)

I guess the real issue is

/github/home/rpmbuild/BUILD/xrootd/xrootd/src/XrdCms/XrdCmsRRQ.cc:370:23: error: 'port' may be used uninitialized [-Werror=maybe-uninitialized]
  370 |       {redrResp.Val = htonl(port);
      |   

{redrResp.Val = htonl(port);

though I'm not sure if there's a way to change these build defaults to turn off -Werror=maybe-uninitialized as that's not a setting that you're currently using with the very old

cmake_minimum_required( VERSION 3.1 )

and so I assume you don't care.

@adriansev
Copy link
Contributor

the problem comes from usage of -flto that converts warnings to errors.. @simonmichal i heard bad things from usage of LTO (errors,breakages with fixes from gcc-11 and up), do you think that is really needed? is the usage of lto given by some benchmark results?
IMHO, unless is really needed, i think that the whole LTO thing can be dropped..

@matthewfeickert
Copy link
Contributor Author

matthewfeickert commented Feb 14, 2023

the problem comes from usage of -flto that converts warnings to errors..

Yeah, that was clear. Sorry I should be more specific. As git grep "flto" yields no results in the codebase but it is clearly showing up in the logs (-flto=auto) I'm not sure what controls this and what is the preferred method of dealing with it in the dev team's mind.

@adriansev
Copy link
Contributor

adriansev commented Feb 14, 2023

Sorry @matthewfeickert i spoke to soon without proper digging ..
It would seem that CMAKE_INTERPROCEDURAL_OPTIMIZATION is a cmake property that is taken as default on and this enable lto .. so far did not find a cmake argument level way to disable this, seems that only a line in CMakeLists.txt on the form of set(CMAKE_INTERPROCEDURAL_OPTIMIZATION FALSE) would disable -flto but i still do not why exactly is this the default

@adriansev
Copy link
Contributor

So, i can reproduce this with the exact commands from workflow, but the really weird thing is that both the error location and what i think is the actual flag that enable errors on warnings (DFORCE_WERROR=TRUE in packaging/rhel/xrootd.spec.in line 512 and 552) are old code
the warning is anyway legit as line https://github.com/xrootd/xrootd/blame/master/src/XrdCms/XrdCmsRRQ.cc#L363
instead of int doredir, port, hlen = 0;
should be int doredir = 0, port = 0, hlen = 0;
let's see what @simonmichal or @amadio have to say ..

@matthewfeickert
Copy link
Contributor Author

matthewfeickert commented Feb 14, 2023

Nice! Thanks for catching all this already @adriansev.

With permalinks for posterity:

So, i can reproduce this with the exact commands from workflow, but the really weird thing is that both the error location and what i think is the actual flag that enable errors on warnings (DFORCE_WERROR=TRUE in

-DCMAKE_INSTALL_PREFIX=/usr -DCMAKE_BUILD_TYPE=RelWithDebInfo \
-DFORCE_WERROR=TRUE \

-DCMAKE_INSTALL_PREFIX=/usr -DCMAKE_BUILD_TYPE=RelWithDebInfo \
-DFORCE_WERROR=TRUE \

) are old code

instead of

int doredir, port, hlen = 0;

should be int doredir = 0, port = 0, hlen = 0;

@matthewfeickert
Copy link
Contributor Author

If this PR is rebased on top of PR #1909 it will pass. As that's easy I would advocate for accepting a solution of PR #1909 and this PR and then doing revisions to what error checking is done in discussion in a later Issue.

@matthewfeickert matthewfeickert marked this pull request as ready for review February 14, 2023 22:22
@matthewfeickert
Copy link
Contributor Author

@adriansev Can you also give this PR a test run through your workflows?

@adriansev
Copy link
Contributor

@matthewfeickert yes, i can confirm the successful execution of rpm-fedora workflow. Thanks a lot!

@amadio
Copy link
Member

amadio commented Feb 15, 2023

This is all good, especially if it fixes #1474. However, if we're going to move CMake minimum requirements up, I'd rather push it to 3.16 instead of 3.12, because there are important fixes in the new FindPython modules and several other new features that we can make use of when updating the build system later.

BTW, please add "Fixes: #1474" somewhere in your last commit message so the issue gets automatically closed when this is merged. Thanks.

@matthewfeickert matthewfeickert changed the title fix: Use CMake 3.12 FindPython to get Python components fix: Use CMake 3.12+ FindPython to get Python components Feb 15, 2023
@matthewfeickert
Copy link
Contributor Author

matthewfeickert commented Feb 15, 2023

However, if we're going to move CMake minimum requirements up, I'd rather push it to 3.16 instead of 3.12, because there are important fixes in the new FindPython modules and several other new features that we can make use of when updating the build system later.

@amadio Sounds great and done! 👍 I've rebased and pushed these changes.

BTW, please add "Fixes: #1474" somewhere in your last commit message so the issue gets automatically closed when this is merged. Thanks.

I've been doing this already in the PR body so that things are cleanly linked back rather than from some commit that might never actually make it into a deploy branch. "Resolves" is in the word list of keywords that can close Issues (c.f. https://docs.github.com/en/issues/tracking-your-work-with-issues/linking-a-pull-request-to-an-issue) so in the PR body when I start it with

Resolves #1474

it sets up the close on merge just as "fixes" would do:

Screenshot from 2023-02-15 03-02-57

@amadio
Copy link
Member

amadio commented Feb 15, 2023

I didn't notice the "Resolves #1474" in the description of the PR before, that will do just fine, no problem.

@@ -76,7 +76,7 @@ jobs:
-DCMAKE_CXX_STANDARD=17 \
-DCMAKE_BUILD_TYPE=RelWithDebInfo \
-DCMAKE_INSTALL_PREFIX=/usr \
-DPYTHON_EXECUTABLE=$(command -v python3) \
-DPython_EXECUTABLE=$(command -v python3) \
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think these could actually be removed, as CMake will find /usr/bin/python3 just fine, but I can do this later.

* Require CMake 3.16 at minimum to allow use of modern find_package
  from CMake 3.12+ as well as fixes in FindPython from CMake 3.16+
  for future planned work.
   - c.f. https://cmake.org/cmake/help/v3.16/command/find_package.html
* Set range of valid CMake versions to allow for more modern CMakes
  to be used if available.
   - c.f. https://cliutils.gitlab.io/modern-cmake/chapters/basics.html
* Use FindPython from CMake 3.12+ which relies on find_package
  advancements to select the components that are required. Select the
  'Interpreter' and 'Development' components to find the required
  executables and files.
   - c.f. https://cmake.org/cmake/help/v3.16/module/FindPython.html
* Modern FindPython is capitalization sensitive and will set the
  following variables that are used throughout the project.
  * Python_FOUND
  * Python_Interpreter_FOUND
  * Python_EXECUTABLE
  * Python_Development_FOUND
   - c.f. https://cmake.org/cmake/help/v3.12/module/FindPython.html#result-variables
@matthewfeickert
Copy link
Contributor Author

Sorry, final force-with-lease push was just to match the commit naming scheme with prefixing them with [CMake] — forgot to do that. This is done now though, so I won't keep messing with it anymore.

@amadio
Copy link
Member

amadio commented Feb 15, 2023

Not a big deal, the CI can handle it, I'm doing many such force pushes myself on the musl PR. In any case, thanks for following the conventions.

@amadio amadio merged commit 565243b into xrootd:master Feb 15, 2023
@matthewfeickert matthewfeickert deleted the fix/use-modern-find-package branch February 15, 2023 19:43
@matthewfeickert
Copy link
Contributor Author

matthewfeickert commented Feb 17, 2023

@raymondEhlers Quick note that, as this didn't make it into the v5.5.3 release, installs from PyPI will continue to fail under Issue #1474 (so probably need to wait until v5.5.4 or v5.6.0).

@amadio
Copy link
Member

amadio commented Feb 17, 2023

This is a new feature, so it will go into 5.6 (there is a change in behavior relative to the previous module).

@matthewfeickert
Copy link
Contributor Author

matthewfeickert commented Feb 17, 2023

This is a new feature, so it will go into 5.6 (there is a change in behavior relative to the previous module).

There is no change in functionality beyond a bug fix. There is a change in the minimum required CMake version, but that doesn't change the functionality from a user's perspective and realistically isn't affecting any distributions as the CI is passing with no additional changes. I'm not sure what you think the feature is.

@raymondEhlers
Copy link

@raymondEhlers Quick note that, as this didn't make it into the v5.5.3 release, installs from PyPI will continue to fail under Issue #1474 (so probably need to wait until v5.5.4 or v5.6.0).

Thanks for the fix and for the heads up!

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

Installation via pip fails (pyenv virtualenv)
4 participants