Python 3.8.2: updates (git, pcre) and cleanups (qt, pyqt, thriftpy) #5648

smuzaffar · 2020-03-15T19:29:25Z

Updated python3 version to 3.8.2. As changing python means rebuilding nearly half of externals, so we propose to integrate some cleanup from CC8 branch which includes

Delete unused specs/files
Cleanup for geting config.gues and config.sub files
Use build in PRM macros instead of custom one
Remove explicit generation of dependencies-setup.*sh files
Fixing autotools to use included xml instead of looking at system headers
Make sure to fail is pip can not find a installed package name
Backports:
- cmsos changes
- git version 2.23
- pcre version 8.43
Drop Qt and PyQt ( as part of Qt cleanup Is FWCore/GuiBrowser really needed? cmssw#28436 )
Unify Zlib build for x86_64 and non-x86_64 archs
Drop thriftpy it has been deprecated and does not build with new python 3.8. Although there is thriftpy2 but as non of our externals require it so we propose to remove it.

cmsbuild · 2020-03-15T19:29:49Z

A new Pull Request was created by @smuzaffar (Malik Shahzad Muzaffar) for branch IB/CMSSW_11_1_X/master.

@cmsbuild, @smuzaffar, @mrodozov, @tulamor can you please review it and eventually sign? Thanks.
cms-bot commands are listed here

smuzaffar · 2020-03-15T19:30:09Z

please test

cmsbuild · 2020-03-15T19:30:13Z

The tests are being triggered in jenkins.
https://cmssdt.cern.ch/jenkins/job/ib-run-pr-tests/5185/console Started: 2020/03/15 20:38
https://cmssdt.cern.ch/jenkins/job/ib-run-pr-tests/5191/console Started: 2020/03/16 09:04

cmsbuild · 2020-03-16T00:28:04Z

-1

Tested at: 3366d40

Build:

I found compilation error when building:

File "/data/cmsbld/jenkins/workspace/ib-run-pr-tests/testBuildDir/slc7_amd64_gcc820/external/py2-pip/9.0.3-b0eba1/lib/python3.8/site-packages/pip/req/req_install.py", line 435, in run_egg_info
call_subprocess(
File "/data/cmsbld/jenkins/workspace/ib-run-pr-tests/testBuildDir/slc7_amd64_gcc820/external/py2-pip/9.0.3-b0eba1/lib/python3.8/site-packages/pip/utils/__init__.py", line 705, in call_subprocess
raise InstallationError(
pip.exceptions.InstallationError: Command "python setup.py egg_info" failed with error code 1 in /data/cmsbld/jenkins/workspace/ib-run-pr-tests/testBuildDir/tmp/pip-tjz2g4ut-build/
error: Bad exit status from /data/cmsbld/jenkins/workspace/ib-run-pr-tests/testBuildDir/tmp/rpm-tmp.CPvRKi (%build)


RPM build errors:
Bad exit status from /data/cmsbld/jenkins/workspace/ib-run-pr-tests/testBuildDir/tmp/rpm-tmp.CPvRKi (%build)

You can see the results of the tests here:
https://cmssdt.cern.ch/SDT/jenkins-artifacts/pull-request-integration/PR-b0eba1/5185/summary.html

cmsbuild · 2020-03-16T12:45:22Z

-1

Tested at: 3366d40

Build:

I found compilation error when building:

File "/data/cmsbld/jenkins/workspace/ib-run-pr-tests/testBuildDir/slc7_amd64_gcc820/external/py2-pip/9.0.3-b0eba1/lib/python3.8/site-packages/pip/req/req_install.py", line 435, in run_egg_info
call_subprocess(
File "/data/cmsbld/jenkins/workspace/ib-run-pr-tests/testBuildDir/slc7_amd64_gcc820/external/py2-pip/9.0.3-b0eba1/lib/python3.8/site-packages/pip/utils/__init__.py", line 705, in call_subprocess
raise InstallationError(
pip.exceptions.InstallationError: Command "python setup.py egg_info" failed with error code 1 in /data/cmsbld/jenkins/workspace/ib-run-pr-tests/testBuildDir/tmp/pip-oxxwfcyg-build/
error: Bad exit status from /data/cmsbld/jenkins/workspace/ib-run-pr-tests/testBuildDir/tmp/rpm-tmp.sp7tjl (%build)


RPM build errors:
Bad exit status from /data/cmsbld/jenkins/workspace/ib-run-pr-tests/testBuildDir/tmp/rpm-tmp.sp7tjl (%build)

You can see the results of the tests here:
https://cmssdt.cern.ch/SDT/jenkins-artifacts/pull-request-integration/PR-b0eba1/5191/summary.html

smuzaffar · 2020-03-19T08:11:06Z

+externals

cmsbuild · 2020-03-19T08:11:29Z

This pull request is fully signed and it will be integrated in one of the next IB/CMSSW_11_1_X/master IBs (tests are also fine). This pull request will now be reviewed by the release team before it's merged. @davidlange6, @silviodonato, @fabiocos (and backports should be raised in the release meeting by the corresponding L2)

smuzaffar · 2020-03-19T08:15:35Z

@silviodonato , any objection on merging this.

smuzaffar · 2020-03-19T08:27:42Z

please test for slc7_aarch64_gcc820

cmsbuild · 2020-03-19T08:27:59Z

The tests are being triggered in jenkins.
Test Parameters:

ARCHITECTURE_FILTER = slc7_aarch64_gcc820
https://cmssdt.cern.ch/jenkins/job/ib-run-pr-tests/5267/console Started: 2020/03/19 09:29
https://cmssdt.cern.ch/jenkins/job/ib-run-pr-tests/5268/console Started: 2020/03/19 09:29

fabiocos · 2020-03-19T08:39:31Z

cmsswdata.spec

@@ -66,15 +63,12 @@ Requires: data-RecoHGCal-TICL
 Requires: data-SimG4CMS-HGCalTestBeam
 Requires: data-SimPPS-PPSPixelDigiProducer

-%if %isnotonline


@smuzaffar out of curiosity, what is the reason of this change? For sure it makes life simpler, as there is no more distinction between what is "online" and the rest, but I wonder why this was in place there earlier at this point

in past we had architectures like slc5onl_amd64_gcc* for special online builds. We are not building any more for those archs, so its time to clean up.

@smuzaffar ok, so no reason to make it more complex than it needs to be now

cmsbuild · 2020-03-19T14:45:23Z

-1

Tested at: 2132266

CMSSW: CMSSW_11_1_X_2020-03-18-2300
SCRAM_ARCH: slc7_ppc64le_gcc820
You can see the results of the tests here:
https://cmssdt.cern.ch/SDT/jenkins-artifacts/pull-request-integration/PR-b0eba1/5268/summary.html

I found follow errors while testing this PR

Failed tests: Build

Build:

I found compilation error when building:

>> Package PhysicsTools/MXNet built
>> Entering Package PhysicsTools/ONNXRuntime
Entering library rule at PhysicsTools/ONNXRuntime
>> Compiling  /build/cmsbld/jenkins_b/workspace/ib-run-pr-tests/CMSSW_11_1_X_2020-03-18-2300/src/PhysicsTools/ONNXRuntime/src/ONNXRuntime.cc 
In file included from /build/cmsbld/jenkins_b/workspace/ib-run-pr-tests/CMSSW_11_1_X_2020-03-18-2300/src/PhysicsTools/ONNXRuntime/src/ONNXRuntime.cc:8:
/build/cmsbld/jenkins_b/workspace/ib-run-pr-tests/CMSSW_11_1_X_2020-03-18-2300/src/PhysicsTools/ONNXRuntime/interface/ONNXRuntime.h:19:10: fatal error: onnxruntime/core/session/onnxruntime_cxx_api.h: No such file or directory
 #include "onnxruntime/core/session/onnxruntime_cxx_api.h"
          ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
compilation terminated.
In file included from /build/cmsbld/jenkins_b/workspace/ib-run-pr-tests/CMSSW_11_1_X_2020-03-18-2300/src/PhysicsTools/ONNXRuntime/src/ONNXRuntime.cc:8:
/build/cmsbld/jenkins_b/workspace/ib-run-pr-tests/CMSSW_11_1_X_2020-03-18-2300/src/PhysicsTools/ONNXRuntime/interface/ONNXRuntime.h:19:10: fatal error: onnxruntime/core/session/onnxruntime_cxx_api.h: No such file or directory

cmsbuild · 2020-03-19T14:45:24Z

Comparison not run due to Build errors (RelVals and Igprof tests were also skipped)

smuzaffar · 2020-03-19T15:04:55Z

#5648 (comment) is known build error for ppc64le . externals build fine which is good news.

silviodonato · 2020-03-20T16:53:13Z

+1

smuzaffar and others added 16 commits March 15, 2020 09:11

Deleted unused specs/files

3551da0

backport clanup changes from cc8 branch

4c916be

autoconf:build with included xml

4fee888

build-pip: fail is package name not found

4ea542b

cmsos: sync back changes from cc8 branch

4f9d7ae

doxygen: updated to version 1.8.16

da54599

git: updated to version 2.23.

42145ba

pcre: cleanup and update to version 8.43

653db6f

qt: remove qt and pyqt

6ba82a8

qt: remove pyqt

600482a

zlib: cleanup: unify build recipe, backported from cc8 branch

966b522

python3: update to version 3.8.2

de493db

remove unused zlib-x86_64.spec

217e46b

cleanup for arch/os checks

cc42d99

cleanup

0d615e3

remove unused/deprecated thriftpy

3366d40

cmsbuild added comparison-pending externals-pending orp-pending pending-signatures tests-pending labels Mar 15, 2020

cmsbuild added tests-started and removed tests-pending labels Mar 15, 2020

cmsbuild added tests-rejected and removed tests-started labels Mar 16, 2020

cmsbuild added the comparison-available label Mar 18, 2020

cmsbuild added externals-approved fully-signed and removed externals-pending pending-signatures labels Mar 19, 2020

cmsbuild added comparison-pending tests-pending tests-started and removed comparison-available tests-approved tests-pending labels Mar 19, 2020

fabiocos reviewed Mar 19, 2020

View reviewed changes

cmsbuild added comparison-notrun tests-rejected and removed comparison-pending tests-started labels Mar 19, 2020

cmsbuild added orp-approved and removed orp-pending labels Mar 20, 2020

smuzaffar merged commit 72b6504 into IB/CMSSW_11_1_X/master Mar 20, 2020

smuzaffar deleted the python3.8 branch March 21, 2020 21:40

iarspider mentioned this pull request Dec 6, 2021

Drop qtconsole, qtpy and py3-jupyter #7491

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Python 3.8.2: updates (git, pcre) and cleanups (qt, pyqt, thriftpy) #5648

Python 3.8.2: updates (git, pcre) and cleanups (qt, pyqt, thriftpy) #5648

smuzaffar commented Mar 15, 2020

cmsbuild commented Mar 15, 2020

smuzaffar commented Mar 15, 2020

cmsbuild commented Mar 15, 2020 •

edited

cmsbuild commented Mar 16, 2020

cmsbuild commented Mar 16, 2020

smuzaffar commented Mar 19, 2020

cmsbuild commented Mar 19, 2020

smuzaffar commented Mar 19, 2020

smuzaffar commented Mar 19, 2020

cmsbuild commented Mar 19, 2020 •

edited

fabiocos Mar 19, 2020

smuzaffar Mar 19, 2020

fabiocos Mar 20, 2020

cmsbuild commented Mar 19, 2020

cmsbuild commented Mar 19, 2020

smuzaffar commented Mar 19, 2020 •

edited

silviodonato commented Mar 20, 2020

Python 3.8.2: updates (git, pcre) and cleanups (qt, pyqt, thriftpy) #5648

Python 3.8.2: updates (git, pcre) and cleanups (qt, pyqt, thriftpy) #5648

Conversation

smuzaffar commented Mar 15, 2020

cmsbuild commented Mar 15, 2020

smuzaffar commented Mar 15, 2020

cmsbuild commented Mar 15, 2020 • edited

cmsbuild commented Mar 16, 2020

cmsbuild commented Mar 16, 2020

smuzaffar commented Mar 19, 2020

cmsbuild commented Mar 19, 2020

smuzaffar commented Mar 19, 2020

smuzaffar commented Mar 19, 2020

cmsbuild commented Mar 19, 2020 • edited

fabiocos Mar 19, 2020

Choose a reason for hiding this comment

smuzaffar Mar 19, 2020

Choose a reason for hiding this comment

fabiocos Mar 20, 2020

Choose a reason for hiding this comment

cmsbuild commented Mar 19, 2020

cmsbuild commented Mar 19, 2020

smuzaffar commented Mar 19, 2020 • edited

silviodonato commented Mar 20, 2020

cmsbuild commented Mar 15, 2020 •

edited

cmsbuild commented Mar 19, 2020 •

edited

smuzaffar commented Mar 19, 2020 •

edited