Move boost to python3

[davidlange6]

likely this will break unit tests. Trying to find out which ones.

[cmsbuild]

A new Pull Request was created by @davidlange6 (David Lange) for branch IB/CMSSW_12_0_X/master.

@cmsbuild, @smuzaffar, @mrodozov, @iarspider can you please review it and eventually sign? Thanks.
@silviodonato, @dpiparo, @qliphy you are the release manager for this.
cms-bot commands are listed here

[davidlange6]

test parameters:

• full_cmssw = true

[davidlange6]

please test

[cmsbuild]

-1

Failed Tests: Build
Summary: https://cmssdt.cern.ch/SDT/jenkins-artifacts/pull-request-integration/PR-0bd24c/16358/summary.html
COMMIT: 9667815
CMSSW: CMSSW_12_0_X_2021-06-29-2300/slc7_amd64_gcc900
User test area: For local testing, you can use /cvmfs/cms-ci.cern.ch/week1/cms-sw/cmsdist/7085/16358/install.sh to create a dev area with all the needed externals and cmssw changes.

The following merge commits were also included on top of IB + this PR after doing git cms-merge-topic:

• build sherpa using python3 #7078 @cms-sw: build sherpa using python3
• added rootntuple tool definition #7075 @cms-sw: added rootntuple tool definition
• [SCRAM] Extra checks of tags and attributes, deploy docs #7077 @cms-sw: [SCRAM] Extra checks of tags and attributes, deploy docs
• Moving externals towards python3 #7070 @davidlange6: Moving externals towards python3

You can see more details here:
https://cmssdt.cern.ch/SDT/jenkins-artifacts/pull-request-integration/PR-0bd24c/16358/git-recent-commits.json
https://cmssdt.cern.ch/SDT/jenkins-artifacts/pull-request-integration/PR-0bd24c/16358/git-merge-result

Build

I found compilation error when building:

/cvmfs/cms-ib.cern.ch/nweek-02687/slc7_amd64_gcc900/external/gcc/9.3.0/bin/../lib/gcc/x86_64-unknown-linux-gnu/9.3.0/../../../../x86_64-unknown-linux-gnu/bin/ld: tmp/slc7_amd64_gcc900/src/CondCore/CondDB/plugins/CondDBV2PyInterface/CondDBPyWrappers.cc.o:(.data.rel.ro._ZTVN5boost6python7objects23caller_py_function_implINS0_6detail6callerIPFN4cond5Iov_tEyRKNSt7__cxx1112basic_stringIcSt11char_traitsIcESaIcEEEENS0_21default_call_policiesENS_3mpl7vector3IS6_ySE_EEEEEE[_ZTVN5boost6python7objects23caller_py_function_implINS0_6detail6callerIPFN4cond5Iov_tEyRKNSt7__cxx1112basic_stringIcSt11char_traitsIcESaIcEEEENS0_21default_call_policiesENS_3mpl7vector3IS6_ySE_EEEEEE]+0x30): undefined reference to `boost::python::objects::py_function_impl_base::max_arity() const'
/cvmfs/cms-ib.cern.ch/nweek-02687/slc7_amd64_gcc900/external/gcc/9.3.0/bin/../lib/gcc/x86_64-unknown-linux-gnu/9.3.0/../../../../x86_64-unknown-linux-gnu/bin/ld: tmp/slc7_amd64_gcc900/src/CondCore/CondDB/plugins/CondDBV2PyInterface/CondDBPyWrappers.cc.o:(.data.rel.ro._ZTVN5boost6python7objects23caller_py_function_implINS0_6detail6callerINS3_6memberIyN4cond5Iov_tEEENS0_21default_call_policiesENS_3mpl7vector3IvRS7_RKyEEEEEE[_ZTVN5boost6python7objects23caller_py_function_implINS0_6detail6callerINS3_6memberIyN4cond5Iov_tEEENS0_21default_call_policiesENS_3mpl7vector3IvRS7_RKyEEEEEE]+0x30): undefined reference to `boost::python::objects::py_function_impl_base::max_arity() const'
/cvmfs/cms-ib.cern.ch/nweek-02687/slc7_amd64_gcc900/external/gcc/9.3.0/bin/../lib/gcc/x86_64-unknown-linux-gnu/9.3.0/../../../../x86_64-unknown-linux-gnu/bin/ld: tmp/slc7_amd64_gcc900/src/CondCore/CondDB/plugins/CondDBV2PyInterface/CondDBPyWrappers.cc.o:(.data.rel.ro._ZTVN5boost6python7objects23caller_py_function_implINS0_6detail6callerINS3_6memberINSt7__cxx1112basic_stringIcSt11char_traitsIcESaIcEEEN4cond5Iov_tEEENS0_21default_call_policiesENS_3mpl7vector3IvRSD_RKSB_EEEEEE[_ZTVN5boost6python7objects23caller_py_function_implINS0_6detail6callerINS3_6memberINSt7__cxx1112basic_stringIcSt11char_traitsIcESaIcEEEN4cond5Iov_tEEENS0_21default_call_policiesENS_3mpl7vector3IvRSD_RKSB_EEEEEE]+0x30): undefined reference to `boost::python::objects::py_function_impl_base::max_arity() const'
/cvmfs/cms-ib.cern.ch/nweek-02687/slc7_amd64_gcc900/external/gcc/9.3.0/bin/../lib/gcc/x86_64-unknown-linux-gnu/9.3.0/../../../../x86_64-unknown-linux-gnu/bin/ld: tmp/slc7_amd64_gcc900/src/CondCore/CondDB/plugins/CondDBV2PyInterface/CondDBPyWrappers.cc.o:(.data.rel.ro._ZTVN5boost6python7objects23caller_py_function_implINS0_6detail6callerIPFvP7_objectENS0_21default_call_policiesENS_3mpl7vector2IvS6_EEEEEE[_ZTVN5boost6python7objects23caller_py_function_implINS0_6detail6callerIPFvP7_objectENS0_21default_call_policiesENS_3mpl7vector2IvS6_EEEEEE]+0x30): undefined reference to `boost::python::objects::py_function_impl_base::max_arity() const'
/cvmfs/cms-ib.cern.ch/nweek-02687/slc7_amd64_gcc900/external/gcc/9.3.0/bin/../lib/gcc/x86_64-unknown-linux-gnu/9.3.0/../../../../x86_64-unknown-linux-gnu/bin/ld: tmp/slc7_amd64_gcc900/src/CondCore/CondDB/plugins/CondDBV2PyInterface/CondDBPyWrappers.cc.o:(.data.rel.ro._ZTVN5boost6python7objects23caller_py_function_implINS0_6detail6callerINS3_6memberIyN4cond5Iov_tEEENS0_19return_value_policyINS0_15return_by_valueENS0_21default_call_policiesEEENS_3mpl7vector2IRyRS7_EEEEEE[_ZTVN5boost6python7objects23caller_py_function_implINS0_6detail6callerINS3_6memberIyN4cond5Iov_tEEENS0_19return_value_policyINS0_15return_by_valueENS0_21default_call_policiesEEENS_3mpl7vector2IRyRS7_EEEEEE]+0x30): more undefined references to `boost::python::objects::py_function_impl_base::max_arity() const' follow
collect2: error: ld returned 1 exit status
gmake: *** [tmp/slc7_amd64_gcc900/src/CondCore/CondDB/plugins/CondDBV2PyInterface/libCondDBV2PyInterface.so] Error 1
Leaving library rule at src/CondCore/CondDB/plugins
>> Compiling  /build/cmsbld/jenkins/workspace/ib-run-pr-tests/CMSSW_12_0_X_2021-06-29-2300/src/CondCore/CondDB/test/testConditionDatabase_0.cpp
>> Building binary testConditionDatabase_0
Copying tmp/slc7_amd64_gcc900/src/CondCore/CondDB/test/testConditionDatabase_0/testConditionDatabase_0 to productstore area:


[davidlange6]

please test

[cmsbuild]

Pull request #7085 was updated.

[davidlange6]

fyi @smuzaffar - only one unit test fails - which is fixed by making getPayloadData.py use /usr/bin/env python3. So I can make a PR to change that

[davidlange6]

please abort

[davidlange6]

test parameters:

• full_cmssw = true
• pull_request = update getPayloadData.py to python3 in sync with changing boost_python to python3 cmssw#34298

[davidlange6]

please test

[cmsbuild]

-1

Failed Tests: UnitTests
Summary: https://cmssdt.cern.ch/SDT/jenkins-artifacts/pull-request-integration/PR-0bd24c/16378/summary.html
COMMIT: b4dce06
CMSSW: CMSSW_12_0_X_2021-06-30-2300/slc7_amd64_gcc900
User test area: For local testing, you can use /cvmfs/cms-ci.cern.ch/week1/cms-sw/cmsdist/7085/16378/install.sh to create a dev area with all the needed externals and cmssw changes.

The following merge commits were also included on top of IB + this PR after doing git cms-merge-topic:

• remove pyminuit and its dependencies #7081 @davidlange6: remove pyminuit and its dependencies
• use python to build ninja #7083 @cms-sw: use python to build ninja
• updated xrootd and blackhat to use py3 only; drop distcc/ccache #7082 @cms-sw: updated xrootd and blackhat to use py3 only; drop distcc/ccache

You can see more details here:
https://cmssdt.cern.ch/SDT/jenkins-artifacts/pull-request-integration/PR-0bd24c/16378/git-recent-commits.json
https://cmssdt.cern.ch/SDT/jenkins-artifacts/pull-request-integration/PR-0bd24c/16378/git-merge-result

Unit Tests

I found errors in the following unit tests:

---> test test_PixelBaryCentreTool had ERRORS
---> test testGetPayloadData had ERRORS

Comparison Summary

Summary:

• No significant changes to the logs found
• Reco comparison results: 0 differences found in the comparisons
• DQMHistoTests: Total files compared: 38
• DQMHistoTests: Total histograms compared: 2785711
• DQMHistoTests: Total failures: 1
• DQMHistoTests: Total nulls: 0
• DQMHistoTests: Total successes: 2785688
• DQMHistoTests: Total skipped: 22
• DQMHistoTests: Total Missing objects: 0
• DQMHistoSizes: Histogram memory added: 0.0 KiB( 37 files compared)
• Checked 160 log files, 37 edm output root files, 38 DQM output files
• TriggerResults: no differences found

[davidlange6]

@smuzaffar - did this not run the tests with cmssw#34298?

[davidlange6]

hum, to answer my question - yes, the cmssw PR was used.

[davidlange6]

  File "/data/cmsbld/jenkins/workspace/ib-run-pr-tests/CMSSW_12_0_X_2021-06-30-2300/bin/slc7_amd64_gcc900/getPayloadData.py", line 292
    except ValueError, e:```
    
   ah - this error is in the PY3 IB:(

[smuzaffar]

yes it did. There are few py3 related changes needed for getPayloadData script itself

[davidlange6]

please test

[cmsbuild]

-1

Failed Tests: UnitTests
Summary: https://cmssdt.cern.ch/SDT/jenkins-artifacts/pull-request-integration/PR-0bd24c/16404/summary.html
COMMIT: b4dce06
CMSSW: CMSSW_12_0_X_2021-07-01-2300/slc7_amd64_gcc900
User test area: For local testing, you can use /cvmfs/cms-ci.cern.ch/week1/cms-sw/cmsdist/7085/16404/install.sh to create a dev area with all the needed externals and cmssw changes.

The following merge commits were also included on top of IB + this PR after doing git cms-merge-topic:

• [GOSAM] updated gosam to 2.1.0 #7089 @cms-sw: [GOSAM] updated gosam to 2.1.0
• python3 migration: drop py2-backports #7090 @cms-sw: python3 migration: drop py2-backports
• py2 cleanup: fastjet and mxnet-predict build with py3 #7087 @cms-sw: py2 cleanup: fastjet and mxnet-predict build with py3
• py2 cleanup: doxygen lhapdf and libxslt #7086 @cms-sw: py2 cleanup: doxygen lhapdf and libxslt

You can see more details here:
https://cmssdt.cern.ch/SDT/jenkins-artifacts/pull-request-integration/PR-0bd24c/16404/git-recent-commits.json
https://cmssdt.cern.ch/SDT/jenkins-artifacts/pull-request-integration/PR-0bd24c/16404/git-merge-result

Unit Tests

I found errors in the following unit tests:

---> test test_PixelBaryCentreTool had ERRORS

Comparison Summary

Summary:

• No significant changes to the logs found
• Reco comparison results: 6 differences found in the comparisons
• DQMHistoTests: Total files compared: 38
• DQMHistoTests: Total histograms compared: 2785711
• DQMHistoTests: Total failures: 12
• DQMHistoTests: Total nulls: 1
• DQMHistoTests: Total successes: 2785676
• DQMHistoTests: Total skipped: 22
• DQMHistoTests: Total Missing objects: 0
• DQMHistoSizes: Histogram memory added: -0.004 KiB( 37 files compared)
• DQMHistoSizes: changed ( 312.0 ): -0.004 KiB MessageLogger/Warnings
• Checked 160 log files, 37 edm output root files, 38 DQM output files
• TriggerResults: no differences found

[smuzaffar]

+externals

[cmsbuild]

This pull request is fully signed and it will be integrated in one of the next IB/CMSSW_12_0_X/master IBs (but tests are reportedly failing). This pull request will now be reviewed by the release team before it's merged. @silviodonato, @dpiparo, @qliphy (and backports should be raised in the release meeting by the corresponding L2)

[smuzaffar]

this should be merged along with cms-sw/cmssw#34298

[qliphy]

merge