Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Updates python packages to their latest version #7263

Merged
merged 12 commits into from Sep 6, 2021

Conversation

smuzaffar
Copy link
Contributor

This PR updates all the python packages to use their latest version.

@smuzaffar
Copy link
Contributor Author

please test

@cmsbuild
Copy link
Contributor

cmsbuild commented Sep 1, 2021

A new Pull Request was created by @smuzaffar (Malik Shahzad Muzaffar) for branch IB/CMSSW_12_1_X/master.

@smuzaffar, @mrodozov, @iarspider can you please review it and eventually sign? Thanks.
@perrotta, @dpiparo, @qliphy you are the release manager for this.
cms-bot commands are listed here

@smuzaffar
Copy link
Contributor Author

please test for slc7_ppc64le_gcc9

@smuzaffar
Copy link
Contributor Author

please test for slc7_aarch64_gcc9

@smuzaffar
Copy link
Contributor Author

please test for cc8_amd64_gcc9

@smuzaffar
Copy link
Contributor Author

please test for slc7_amd64_gcc11

@cmsbuild
Copy link
Contributor

cmsbuild commented Sep 1, 2021

-1

Summary: https://cmssdt.cern.ch/SDT/jenkins-artifacts/pull-request-integration/PR-632f12/18213/summary.html
COMMIT: c6ec2c3
CMSSW: CMSSW_12_1_X_2021-08-31-2300/slc7_ppc64le_gcc9
User test area: For local testing, you can use /cvmfs/cms-ci.cern.ch/week0/cms-sw/cmsdist/7263/18213/install.sh to create a dev area with all the needed externals and cmssw changes.

External Build

I found compilation error when building:

+ pip_package_dirname=isort-5.9.3
+ pushd isort-5.9.3
/scratch/cmsbuild/jenkins_b/workspace/ib-run-pr-tests/testBuildDir/BUILD/slc7_ppc64le_gcc9/external/py3-isort/5.9.3-eee86ad62641b9c53ac715c41cdef8c0/cmsdist-patch-py3-isort/isort-5.9.3 /scratch/cmsbuild/jenkins_b/workspace/ib-run-pr-tests/testBuildDir/BUILD/slc7_ppc64le_gcc9/external/py3-isort/5.9.3-eee86ad62641b9c53ac715c41cdef8c0/cmsdist-patch-py3-isort /scratch/cmsbuild/jenkins_b/workspace/ib-run-pr-tests/testBuildDir/BUILD/slc7_ppc64le_gcc9/external/py3-isort/5.9.3-eee86ad62641b9c53ac715c41cdef8c0
+ sed -i -e '/.*futures.*/d' isort.egg-info/requires.txt setup.py
sed: can't read isort.egg-info/requires.txt: No such file or directory
error: Bad exit status from /scratch/cmsbuild/jenkins_b/workspace/ib-run-pr-tests/testBuildDir/tmp/rpm-tmp.6Tt8ou (%build)


RPM build errors:
Macro %rpmbuild_libdir defined but not used within scope
Bad exit status from /scratch/cmsbuild/jenkins_b/workspace/ib-run-pr-tests/testBuildDir/tmp/rpm-tmp.6Tt8ou (%build)


@cmsbuild
Copy link
Contributor

cmsbuild commented Sep 1, 2021

-1

Summary: https://cmssdt.cern.ch/SDT/jenkins-artifacts/pull-request-integration/PR-632f12/18212/summary.html
COMMIT: c6ec2c3
CMSSW: CMSSW_12_1_X_2021-08-31-2300/slc7_amd64_gcc900
User test area: For local testing, you can use /cvmfs/cms-ci.cern.ch/week0/cms-sw/cmsdist/7263/18212/install.sh to create a dev area with all the needed externals and cmssw changes.

External Build

I found compilation error when building:

File "/data/cmsbld/jenkins/workspace/ib-run-pr-tests/testBuildDir/slc7_amd64_gcc900/external/py3-pip/20.3.3-llifpc/lib/python3.9/site-packages/pip/_internal/utils/subprocess.py", line 240, in call_subprocess
raise InstallationError(exc_msg)
pip._internal.exceptions.InstallationError: Command errored out with exit status 1: /data/cmsbld/jenkins/workspace/ib-run-pr-tests/testBuildDir/slc7_amd64_gcc900/external/python3/3.9.6/bin/python3 /data/cmsbld/jenkins/workspace/ib-run-pr-tests/testBuildDir/slc7_amd64_gcc900/external/py3-pip/20.3.3-llifpc/lib/python3.9/site-packages/pip/_vendor/pep517/_in_process.py prepare_metadata_for_build_wheel /data/cmsbld/jenkins/workspace/ib-run-pr-tests/testBuildDir/BUILD/slc7_amd64_gcc900/external/py3-defusedxml/0.7.1-563437b75603347164bc31f03d798e90/cmsdist-tmp/tmphwa63h98 Check the logs for full command output.
Removed file:///data/cmsbld/jenkins/workspace/ib-run-pr-tests/testBuildDir/BUILD/slc7_amd64_gcc900/external/py3-defusedxml/0.7.1-563437b75603347164bc31f03d798e90/src.tar.gz from build tracker '/data/cmsbld/jenkins/workspace/ib-run-pr-tests/testBuildDir/BUILD/slc7_amd64_gcc900/external/py3-defusedxml/0.7.1-563437b75603347164bc31f03d798e90/cmsdist-tmp/pip-req-tracker-e438ek9j'
Removed build tracker: '/data/cmsbld/jenkins/workspace/ib-run-pr-tests/testBuildDir/BUILD/slc7_amd64_gcc900/external/py3-defusedxml/0.7.1-563437b75603347164bc31f03d798e90/cmsdist-tmp/pip-req-tracker-e438ek9j'
error: Bad exit status from /data/cmsbld/jenkins/workspace/ib-run-pr-tests/testBuildDir/tmp/rpm-tmp.gSoXUv (%build)


RPM build errors:
Macro %rpmbuild_libdir defined but not used within scope
Bad exit status from /data/cmsbld/jenkins/workspace/ib-run-pr-tests/testBuildDir/tmp/rpm-tmp.gSoXUv (%build)


@cmsbuild
Copy link
Contributor

cmsbuild commented Sep 1, 2021

-1

Summary: https://cmssdt.cern.ch/SDT/jenkins-artifacts/pull-request-integration/PR-632f12/18215/summary.html
COMMIT: c6ec2c3
CMSSW: CMSSW_12_1_X_2021-08-31-1100/slc7_amd64_gcc11
User test area: For local testing, you can use /cvmfs/cms-ci.cern.ch/week0/cms-sw/cmsdist/7263/18215/install.sh to create a dev area with all the needed externals and cmssw changes.

External Build

I found compilation error when building:

patching file lib/sqlalchemy/connectors/frontier.py
patching file lib/sqlalchemy/dialects/oracle/__init__.py
Hunk #1 FAILED at 8.
1 out of 1 hunk FAILED -- saving rejects to file lib/sqlalchemy/dialects/oracle/__init__.py.rej
patching file lib/sqlalchemy/dialects/oracle/frontier.py
error: Bad exit status from /data/cmsbld/jenkins/workspace/ib-run-pr-tests/testBuildDir/tmp/rpm-tmp.IsFX9h (%build)


RPM build errors:
line 36: It's not recommended to have unversioned Obsoletes: Obsoletes: external+py3-sqlalchemy+1.4.23-a695c86837c44977996e6b88b7789a65
Macro expanded in comment on line 341: %{pkginstroot}/bin/*


@cmsbuild
Copy link
Contributor

cmsbuild commented Sep 1, 2021

-1

Summary: https://cmssdt.cern.ch/SDT/jenkins-artifacts/pull-request-integration/PR-632f12/18214/summary.html
COMMIT: c6ec2c3
CMSSW: CMSSW_12_1_X_2021-08-31-2300/cc8_amd64_gcc9
User test area: For local testing, you can use /cvmfs/cms-ci.cern.ch/week0/cms-sw/cmsdist/7263/18214/install.sh to create a dev area with all the needed externals and cmssw changes.

External Build

I found compilation error when building:

File "/data/cmsbld/jenkins/workspace/ib-run-pr-tests/testBuildDir/cc8_amd64_gcc9/external/py3-pip/20.3.3-67f8042e1ea9b2137c8d4cc0e2d1175f/lib/python3.9/site-packages/pip/_internal/utils/subprocess.py", line 240, in call_subprocess
raise InstallationError(exc_msg)
pip._internal.exceptions.InstallationError: Command errored out with exit status 1: /data/cmsbld/jenkins/workspace/ib-run-pr-tests/testBuildDir/cc8_amd64_gcc9/external/python3/3.9.6-08d3bbf663e5ec71dd2cc6e7be45d0b0/bin/python3 /data/cmsbld/jenkins/workspace/ib-run-pr-tests/testBuildDir/cc8_amd64_gcc9/external/py3-pip/20.3.3-67f8042e1ea9b2137c8d4cc0e2d1175f/lib/python3.9/site-packages/pip/_vendor/pep517/_in_process.py prepare_metadata_for_build_wheel /data/cmsbld/jenkins/workspace/ib-run-pr-tests/testBuildDir/BUILD/cc8_amd64_gcc9/external/py3-immutables/0.16-2668801d1ea7c02c6cae98cf8d5eee4f/cmsdist-tmp/tmp2smyg2f4 Check the logs for full command output.
Removed file:///data/cmsbld/jenkins/workspace/ib-run-pr-tests/testBuildDir/BUILD/cc8_amd64_gcc9/external/py3-immutables/0.16-2668801d1ea7c02c6cae98cf8d5eee4f/src.tar.gz from build tracker '/data/cmsbld/jenkins/workspace/ib-run-pr-tests/testBuildDir/BUILD/cc8_amd64_gcc9/external/py3-immutables/0.16-2668801d1ea7c02c6cae98cf8d5eee4f/cmsdist-tmp/pip-req-tracker-brriqwx9'
Removed build tracker: '/data/cmsbld/jenkins/workspace/ib-run-pr-tests/testBuildDir/BUILD/cc8_amd64_gcc9/external/py3-immutables/0.16-2668801d1ea7c02c6cae98cf8d5eee4f/cmsdist-tmp/pip-req-tracker-brriqwx9'
error: Bad exit status from /data/cmsbld/jenkins/workspace/ib-run-pr-tests/testBuildDir/tmp/rpm-tmp.aJ3esU (%build)


RPM build errors:
line 36: It's not recommended to have unversioned Obsoletes: Obsoletes: external+py3-immutables+0.16-2668801d1ea7c02c6cae98cf8d5eee4f
Macro expanded in comment on line 337: %{pkginstroot}/bin/*


@cmsbuild
Copy link
Contributor

cmsbuild commented Sep 2, 2021

-1

Summary: https://cmssdt.cern.ch/SDT/jenkins-artifacts/pull-request-integration/PR-632f12/18220/summary.html
COMMIT: c6ec2c3
CMSSW: CMSSW_12_1_X_2021-08-31-2300/slc7_aarch64_gcc9
User test area: For local testing, you can use /cvmfs/cms-ci.cern.ch/week0/cms-sw/cmsdist/7263/18220/install.sh to create a dev area with all the needed externals and cmssw changes.

External Build

I found compilation error when building:

+ pip_package_dirname=isort-5.9.3
+ pushd isort-5.9.3
/home/cmsbuild/jenkins_c/workspace/ib-run-pr-tests/testBuildDir/BUILD/slc7_aarch64_gcc9/external/py3-isort/5.9.3-eee86ad62641b9c53ac715c41cdef8c0/cmsdist-patch-py3-isort/isort-5.9.3 /home/cmsbuild/jenkins_c/workspace/ib-run-pr-tests/testBuildDir/BUILD/slc7_aarch64_gcc9/external/py3-isort/5.9.3-eee86ad62641b9c53ac715c41cdef8c0/cmsdist-patch-py3-isort /home/cmsbuild/jenkins_c/workspace/ib-run-pr-tests/testBuildDir/BUILD/slc7_aarch64_gcc9/external/py3-isort/5.9.3-eee86ad62641b9c53ac715c41cdef8c0
+ sed -i -e '/.*futures.*/d' isort.egg-info/requires.txt setup.py
sed: can't read isort.egg-info/requires.txt: No such file or directory
error: Bad exit status from /home/cmsbuild/jenkins_c/workspace/ib-run-pr-tests/testBuildDir/tmp/rpm-tmp.WXkdkT (%build)


RPM build errors:
Macro %rpmbuild_libdir defined but not used within scope
Bad exit status from /home/cmsbuild/jenkins_c/workspace/ib-run-pr-tests/testBuildDir/tmp/rpm-tmp.WXkdkT (%build)


@cmsbuild
Copy link
Contributor

cmsbuild commented Sep 2, 2021

Pull request #7263 was updated.

@smuzaffar
Copy link
Contributor Author

test parameters:

  • full_cmssw = true

@smuzaffar
Copy link
Contributor Author

please test

@cmsbuild
Copy link
Contributor

cmsbuild commented Sep 2, 2021

-1

Summary: https://cmssdt.cern.ch/SDT/jenkins-artifacts/pull-request-integration/PR-632f12/18259/summary.html
COMMIT: 11821ed
CMSSW: CMSSW_12_1_X_2021-09-02-1100/slc7_amd64_gcc900
User test area: For local testing, you can use /cvmfs/cms-ci.cern.ch/week0/cms-sw/cmsdist/7263/18259/install.sh to create a dev area with all the needed externals and cmssw changes.

External Build

I found compilation error when building:

ERROR: no such package '@tf_runtime//': java.io.IOException: Error downloading [http://mirror.tensorflow.org/github.com/tensorflow/runtime/archive/b570a1921c9e55ac53c8972bd2bfd37cd0eb510d.tar.gz, https://github.com/tensorflow/runtime/archive/b570a1921c9e55ac53c8972bd2bfd37cd0eb510d.tar.gz] to /data/cmsbld/jenkins/workspace/ib-run-pr-tests/testBuildDir/BUILD/slc7_amd64_gcc900/external/tensorflow-sources/2.6.0-625c0a85de668f6357505f9c5d750b61/build/4d16e41fbcf077a0cee8cdd41d7160c1/external/tf_runtime/temp213067392128748662/b570a1921c9e55ac53c8972bd2bfd37cd0eb510d.tar.gz: connect timed out
INFO: Elapsed time: 57.594s
INFO: 0 processes.
FAILED: Build did NOT complete successfully (0 packages loaded)
FAILED: Build did NOT complete successfully (0 packages loaded)
error: Bad exit status from /data/cmsbld/jenkins/workspace/ib-run-pr-tests/testBuildDir/tmp/rpm-tmp.ik8D04 (%build)


RPM build errors:
Macro %rpmbuild_libdir defined but not used within scope
Bad exit status from /data/cmsbld/jenkins/workspace/ib-run-pr-tests/testBuildDir/tmp/rpm-tmp.ik8D04 (%build)


@smuzaffar
Copy link
Contributor Author

please test

@smuzaffar
Copy link
Contributor Author

please test for cc8_amd64_gcc9

@cmsbuild
Copy link
Contributor

cmsbuild commented Sep 5, 2021

-1

Summary: https://cmssdt.cern.ch/SDT/jenkins-artifacts/pull-request-integration/PR-632f12/18323/summary.html
COMMIT: 0e0d104
CMSSW: CMSSW_12_1_X_2021-09-05-0000/cc8_amd64_gcc9
User test area: For local testing, you can use /cvmfs/cms-ci.cern.ch/week1/cms-sw/cmsdist/7263/18323/install.sh to create a dev area with all the needed externals and cmssw changes.

External Build

I found compilation error when building:

Target //tensorflow/tools/pip_package:build_pip_package failed to build
INFO: Elapsed time: 56.712s, Critical Path: 6.44s
INFO: 697 processes: 484 internal, 213 local.
FAILED: Build did NOT complete successfully
FAILED: Build did NOT complete successfully
error: Bad exit status from /data/cmsbld/jenkins/workspace/ib-run-pr-tests/testBuildDir/tmp/rpm-tmp.2VdGOQ (%build)


RPM build errors:
line 37: It's not recommended to have unversioned Obsoletes: Obsoletes: external+tensorflow-sources+2.6.0-3260aab615e98f6ad9f0340a87f9c13f
Bad exit status from /data/cmsbld/jenkins/workspace/ib-run-pr-tests/testBuildDir/tmp/rpm-tmp.2VdGOQ (%build)


@cmsbuild
Copy link
Contributor

cmsbuild commented Sep 5, 2021

Pull request #7263 was updated.

@smuzaffar
Copy link
Contributor Author

please test for cc8_amd64_gcc9

@smuzaffar
Copy link
Contributor Author

please test

@cmsbuild
Copy link
Contributor

cmsbuild commented Sep 6, 2021

+1

Summary: https://cmssdt.cern.ch/SDT/jenkins-artifacts/pull-request-integration/PR-632f12/18327/summary.html
COMMIT: aff6168
CMSSW: CMSSW_12_1_X_2021-09-05-0000/slc7_amd64_gcc900
User test area: For local testing, you can use /cvmfs/cms-ci.cern.ch/week1/cms-sw/cmsdist/7263/18327/install.sh to create a dev area with all the needed externals and cmssw changes.

The following merge commits were also included on top of IB + this PR after doing git cms-merge-topic:

You can see more details here:
https://cmssdt.cern.ch/SDT/jenkins-artifacts/pull-request-integration/PR-632f12/18327/git-recent-commits.json
https://cmssdt.cern.ch/SDT/jenkins-artifacts/pull-request-integration/PR-632f12/18327/git-merge-result

Comparison Summary

Summary:

  • No significant changes to the logs found
  • Reco comparison results: 0 differences found in the comparisons
  • DQMHistoTests: Total files compared: 39
  • DQMHistoTests: Total histograms compared: 3000491
  • DQMHistoTests: Total failures: 0
  • DQMHistoTests: Total nulls: 0
  • DQMHistoTests: Total successes: 3000469
  • DQMHistoTests: Total skipped: 22
  • DQMHistoTests: Total Missing objects: 0
  • DQMHistoSizes: Histogram memory added: 235.416 KiB( 38 files compared)
  • DQMHistoSizes: changed ( 10024.0,... ): 13.848 KiB MuonCSCDigisV/CSCDigiTask
  • Checked 165 log files, 37 edm output root files, 39 DQM output files
  • TriggerResults: no differences found

@smuzaffar
Copy link
Contributor Author

looks good

@smuzaffar
Copy link
Contributor Author

+externals

@cmsbuild
Copy link
Contributor

cmsbuild commented Sep 6, 2021

This pull request is fully signed and it will be integrated in one of the next IB/CMSSW_12_1_X/master IBs (tests are also fine). This pull request will now be reviewed by the release team before it's merged. @perrotta, @dpiparo, @qliphy (and backports should be raised in the release meeting by the corresponding L2)

@cmsbuild
Copy link
Contributor

cmsbuild commented Sep 6, 2021

+1

Summary: https://cmssdt.cern.ch/SDT/jenkins-artifacts/pull-request-integration/PR-632f12/18326/summary.html
COMMIT: aff6168
CMSSW: CMSSW_12_1_X_2021-09-05-0000/cc8_amd64_gcc9
User test area: For local testing, you can use /cvmfs/cms-ci.cern.ch/week1/cms-sw/cmsdist/7263/18326/install.sh to create a dev area with all the needed externals and cmssw changes.

The following merge commits were also included on top of IB + this PR after doing git cms-merge-topic:

You can see more details here:
https://cmssdt.cern.ch/SDT/jenkins-artifacts/pull-request-integration/PR-632f12/18326/git-recent-commits.json
https://cmssdt.cern.ch/SDT/jenkins-artifacts/pull-request-integration/PR-632f12/18326/git-merge-result

Comparison Summary

Summary:

  • No significant changes to the logs found
  • Reco comparison results: 57312 differences found in the comparisons
  • DQMHistoTests: Total files compared: 39
  • DQMHistoTests: Total histograms compared: 3000491
  • DQMHistoTests: Total failures: 298291
  • DQMHistoTests: Total nulls: 242
  • DQMHistoTests: Total successes: 2701936
  • DQMHistoTests: Total skipped: 22
  • DQMHistoTests: Total Missing objects: 0
  • DQMHistoSizes: Histogram memory added: 228.58 KiB( 38 files compared)
  • DQMHistoSizes: changed ( 10024.0,... ): 13.848 KiB MuonCSCDigisV/CSCDigiTask
  • DQMHistoSizes: changed ( 10224.0 ): -0.117 KiB SiStrip/MechanicalView
  • DQMHistoSizes: changed ( 136.874 ): -0.023 KiB JetMET/SUSYDQM
  • DQMHistoSizes: changed ( 250202.181 ): -0.176 KiB SiStrip/MechanicalView
  • DQMHistoSizes: changed ( 25202.0 ): 0.010 KiB SiStrip/MechanicalView
  • DQMHistoSizes: changed ( 4.53 ): 0.004 KiB JetMET/SUSYDQM
  • DQMHistoSizes: changed ( 7.3 ): -6.532 KiB SiStrip/MechanicalView
  • Checked 165 log files, 37 edm output root files, 39 DQM output files
  • TriggerResults: found differences in 14 / 38 workflows

@Dr15Jones
Copy link

Are all the failed comparisons because the histogram entries for this PR all missing?

@mrodozov mrodozov deleted the py3-update branch September 6, 2021 13:10
@mrodozov mrodozov restored the py3-update branch September 6, 2021 13:10
@smuzaffar
Copy link
Contributor Author

@Dr15Jones , for production arch (slc7_amd64_gcc900), there are no comparison diffs ( #7263 (comment) ).
For CC8 ( #7263 (comment) ) , yes there are a lot of diffs but I do not know why (may be due to slc7 baseline vs cc8 PR)

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging this pull request may close these issues.

None yet

4 participants