Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[Spack] Update py-packages to the latest version #7767

Merged

Conversation

aandvalenzuela
Copy link
Contributor

No description provided.

@cmsbuild
Copy link
Contributor

cmsbuild commented Apr 7, 2022

A new Pull Request was created by @aandvalenzuela (Andrea Valenzuela) for branch IB/CMSSW_12_4_X/master.

@cmsbuild, @smuzaffar, @aandvalenzuela, @iarspider can you please review it and eventually sign? Thanks.
@perrotta, @dpiparo, @qliphy you are the release manager for this.
cms-bot commands are listed here

@@ -281,7 +281,7 @@ tenacity==8.0.1
#NO_AUTO_UPDATE:3: Force to use tensorflow 2.6.0; this should match the version in tensorflow-sources.spec
tensorboard==2.6.0
tensorflow==2.6.0
tensorflow-estimator==2.6.0
tensorflow-estimator==2.8.0
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think we should keep this one at 2.6.0 as well, to match tensorflow. But could be wrong. @smuzaffar ?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Let's try the test for 2.6.0 (tensorflow latest version is also 2.8.0)

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

right, we need to keep it to 2.6.0 (to match the version in tensorflow-sources)

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thank you!

@cmsbuild
Copy link
Contributor

cmsbuild commented Apr 7, 2022

Pull request #7767 was updated.

@aandvalenzuela
Copy link
Contributor Author

please test

@cmsbuild
Copy link
Contributor

cmsbuild commented Apr 7, 2022

-1

Summary: https://cmssdt.cern.ch/SDT/jenkins-artifacts/pull-request-integration/PR-647cd0/23725/summary.html
COMMIT: c5fadae
CMSSW: CMSSW_12_4_X_2022-04-07-1100/slc7_amd64_gcc10
User test area: For local testing, you can use /cvmfs/cms-ci.cern.ch/week1/cms-sw/cmsdist/7767/23725/install.sh to create a dev area with all the needed externals and cmssw changes.

External Build

I found compilation error when building:

warning: no files found matching 'README.rst'
adding license file 'LICENSE'
writing manifest file '/data/cmsbld/jenkins/workspace/ib-run-pr-tests/testBuildDir/BUILD/slc7_amd64_gcc10/external/py3-typing/3.10.0.0-d6543bedcddaf7b6b167af71c279f4bc/cmsdist-tmp/pip-pip-egg-info-ojkryc0o/typing.egg-info/SOURCES.txt'
Preparing metadata (setup.py): finished with status 'done'
ERROR: Package 'typing' requires a different Python: 3.9.6 not in '!=3.0.*,!=3.1.*,!=3.2.*,!=3.3.*,<3.5,>=2.7'
error: Bad exit status from /data/cmsbld/jenkins/workspace/ib-run-pr-tests/testBuildDir/tmp/rpm-tmp.z0hcA2 (%build)


RPM build errors:
Bad exit status from /data/cmsbld/jenkins/workspace/ib-run-pr-tests/testBuildDir/tmp/rpm-tmp.z0hcA2 (%build)



@@ -298,7 +298,7 @@ tqdm==4.62.2
traitlets==5.1.0
typed-ast==1.4.3
typing-extensions==3.10.0.2
typing==3.7.4.3
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Did we not drop py3-typing some time ago? If not, we should:

NOTE: in Python 3.5 and later, the typing module lives in the stdlib, and installing this package has NO EFFECT

(from https://pypi.org/project/typing/)

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I do not remember such a change. Yes, we should drop it as it is picked up from python3 installations

[muzaffar@lxplus796 ~]$ source /cvmfs/cms.cern.ch/slc7_amd64_gcc10/external/py3-typing/3.7.4.3/etc/profile.d/init.sh
[muzaffar@lxplus796 ~]$ python3 -c 'import typing;print(typing.__file__)'
/cvmfs/cms.cern.ch/slc7_amd64_gcc10/external/python3/3.8.2-llifpc/lib/python3.8/typing.py

@aandvalenzuela , can you please remove it from requirements.txt and also remove all of its usages e.g. https://github.com/cms-sw/cmsdist/search?q=py3-typing

@cmsbuild
Copy link
Contributor

cmsbuild commented Apr 8, 2022

Pull request #7767 was updated.

@aandvalenzuela
Copy link
Contributor Author

please test

@cmsbuild
Copy link
Contributor

cmsbuild commented Apr 8, 2022

-1

Summary: https://cmssdt.cern.ch/SDT/jenkins-artifacts/pull-request-integration/PR-647cd0/23768/summary.html
COMMIT: 52751d5
CMSSW: CMSSW_12_4_X_2022-04-08-1100/slc7_amd64_gcc10
User test area: For local testing, you can use /cvmfs/cms-ci.cern.ch/week1/cms-sw/cmsdist/7767/23768/install.sh to create a dev area with all the needed externals and cmssw changes.

External Build

I found compilation error when building:

+ DEPS='py3-dill '
+ '[' 'py3-dill ' '!=' '' ']'
+ echo 'ERROR: Missing dependencies for py3-pylint (python3) found: py3-dill '
ERROR: Missing dependencies for py3-pylint (python3) found: py3-dill 
+ exit 1
error: Bad exit status from /data/cmsbld/jenkins/workspace/ib-run-pr-tests/testBuildDir/tmp/rpm-tmp.eOcZYq (%build)


RPM build errors:
Bad exit status from /data/cmsbld/jenkins/workspace/ib-run-pr-tests/testBuildDir/tmp/rpm-tmp.eOcZYq (%build)



@smuzaffar
Copy link
Contributor

smuzaffar commented Apr 8, 2022

@aandvalenzuela , I would suggest to try to build it locally first. These types of updates take too many iteration. In order to build locally you need to run following on cmsdev machines

git clone git@github.com:aandvalenzuela/cmsdist -b update-packages
git clone git@github.com:cms-sw/pkgtools -b V00-34-XX
./pkgtools/cmsBuild --weekly -a slc7_amd64_gcc10 -i work -j 10 build cmssw-tool-conf

you might get errors like the above ERROR: Missing dependencies for py3-pylint (python3) found: py3-dill . To fix this type of errors you need to add Requires: py3-dill in cmsdist/pip/pylint.file and rerun the ./pkgtools/cmsBuild .... command again. If dill is missing in pip/requirements.txt then please add its entry with the proper version.

@cmsbuild
Copy link
Contributor

cmsbuild commented Apr 9, 2022

Pull request #7767 was updated.

@smuzaffar
Copy link
Contributor

please test

@cmsbuild
Copy link
Contributor

cmsbuild commented Apr 9, 2022

Pull request #7767 was updated.

@cmsbuild
Copy link
Contributor

cmsbuild commented Apr 9, 2022

Pull request #7767 was updated.

@smuzaffar
Copy link
Contributor

please test
looks lile pylint needs newer astroid

@cmsbuild
Copy link
Contributor

+1

Summary: https://cmssdt.cern.ch/SDT/jenkins-artifacts/pull-request-integration/PR-647cd0/23799/summary.html
COMMIT: 335b07e
CMSSW: CMSSW_12_4_X_2022-04-09-1100/slc7_amd64_gcc10
User test area: For local testing, you can use /cvmfs/cms-ci.cern.ch/week1/cms-sw/cmsdist/7767/23799/install.sh to create a dev area with all the needed externals and cmssw changes.

The following merge commits were also included on top of IB + this PR after doing git cms-merge-topic:

You can see more details here:
https://cmssdt.cern.ch/SDT/jenkins-artifacts/pull-request-integration/PR-647cd0/23799/git-recent-commits.json
https://cmssdt.cern.ch/SDT/jenkins-artifacts/pull-request-integration/PR-647cd0/23799/git-merge-result

Comparison Summary

Summary:

  • No significant changes to the logs found
  • Reco comparison results: 4 differences found in the comparisons
  • DQMHistoTests: Total files compared: 48
  • DQMHistoTests: Total histograms compared: 3593039
  • DQMHistoTests: Total failures: 7
  • DQMHistoTests: Total nulls: 1
  • DQMHistoTests: Total successes: 3593009
  • DQMHistoTests: Total skipped: 22
  • DQMHistoTests: Total Missing objects: 0
  • DQMHistoSizes: Histogram memory added: -0.004 KiB( 47 files compared)
  • DQMHistoSizes: changed ( 312.0 ): -0.004 KiB MessageLogger/Warnings
  • Checked 200 log files, 45 edm output root files, 48 DQM output files
  • TriggerResults: no differences found

@smuzaffar
Copy link
Contributor

please test for slc7_aarch64_gcc11

@smuzaffar
Copy link
Contributor

please test for slc7_ppc64le_gcc11

@cmsbuild
Copy link
Contributor

-1

Failed Tests: UnitTests RelVals
Summary: https://cmssdt.cern.ch/SDT/jenkins-artifacts/pull-request-integration/PR-647cd0/23802/summary.html
COMMIT: 335b07e
CMSSW: CMSSW_12_4_X_2022-04-07-2300/slc7_aarch64_gcc11
User test area: For local testing, you can use /cvmfs/cms-ci.cern.ch/week1/cms-sw/cmsdist/7767/23802/install.sh to create a dev area with all the needed externals and cmssw changes.

The following merge commits were also included on top of IB + this PR after doing git cms-merge-topic:

You can see more details here:
https://cmssdt.cern.ch/SDT/jenkins-artifacts/pull-request-integration/PR-647cd0/23802/git-recent-commits.json
https://cmssdt.cern.ch/SDT/jenkins-artifacts/pull-request-integration/PR-647cd0/23802/git-merge-result

Unit Tests

I found errors in the following unit tests:

---> test DRNTest had ERRORS
---> test testFWCoreUtilities had ERRORS
---> test TestFWCoreServicesDriver had ERRORS
---> test import-tables had ERRORS
and more ...

RelVals

----- Begin Fatal Exception 10-Apr-2022 11:09:01 CEST-----------------------
An exception of category 'Vertex' occurred while
   [0] Processing  Event run: 194533 lumi: 329 event: 462355458 stream: 0
   [1] Running path 'dqmofflineOnPAT_1_step'
   [2] Prefetching for module SingleTopTChannelLeptonDQM_miniAOD/'singleTopElectronMediumDQM_miniAOD'
   [3] Prefetching for module PATMuonSlimmer/'slimmedMuons'
   [4] Prefetching for module PATMuonSelector/'selectedPatMuons'
   [5] Prefetching for module PATMuonProducer/'patMuons'
   [6] Prefetching for module MuonProducer/'muons'
   [7] Prefetching for module PFProducer/'particleFlowTmp'
   [8] Prefetching for module PFBlockProducer/'particleFlowBlock'
   [9] Prefetching for module PFElecTkProducer/'pfTrackElec'
   [10] Prefetching for module PFConversionProducer/'pfConversions'
   [11] Calling method for module ConversionProducer/'allConversions'
Exception Message:
Refitted track not found in list
----- End Fatal Exception -------------------------------------------------
----- Begin Fatal Exception 10-Apr-2022 11:19:02 CEST-----------------------
An exception of category 'Vertex' occurred while
   [0] Processing  Event run: 326479 lumi: 7 event: 1579493 stream: 0
   [1] Running path 'dqmoffline_8_step'
   [2] Prefetching for module SMPDQM/'SMPDQM'
   [3] Prefetching for module MuonProducer/'muons'
   [4] Prefetching for module PFProducer/'particleFlowTmp'
   [5] Prefetching for module PFBlockProducer/'particleFlowBlock'
   [6] Prefetching for module PFElecTkProducer/'pfTrackElec'
   [7] Prefetching for module PFConversionProducer/'pfConversions'
   [8] Calling method for module ConversionProducer/'allConversions'
Exception Message:
Refitted track not found in list
----- End Fatal Exception -------------------------------------------------
----- Begin Fatal Exception 10-Apr-2022 11:27:21 CEST-----------------------
An exception of category 'Vertex' occurred while
   [0] Processing  Event run: 319450 lumi: 76 event: 106007323 stream: 0
   [1] Running path 'dqmoffline_10_step'
   [2] Prefetching for module SMPDQM/'SMPDQM'
   [3] Prefetching for module MuonProducer/'muons'
   [4] Prefetching for module PFProducer/'particleFlowTmp'
   [5] Prefetching for module PFBlockProducer/'particleFlowBlock'
   [6] Prefetching for module PFElecTkProducer/'pfTrackElec'
   [7] Prefetching for module PFConversionProducer/'pfConversions'
   [8] Calling method for module ConversionProducer/'allConversions'
Exception Message:
Refitted track not found in list
----- End Fatal Exception -------------------------------------------------

@cmsbuild
Copy link
Contributor

-1

Failed Tests: UnitTests
Summary: https://cmssdt.cern.ch/SDT/jenkins-artifacts/pull-request-integration/PR-647cd0/23803/summary.html
COMMIT: 335b07e
CMSSW: CMSSW_12_4_X_2022-04-07-2300/slc7_ppc64le_gcc11
User test area: For local testing, you can use /cvmfs/cms-ci.cern.ch/week1/cms-sw/cmsdist/7767/23803/install.sh to create a dev area with all the needed externals and cmssw changes.

The following merge commits were also included on top of IB + this PR after doing git cms-merge-topic:

You can see more details here:
https://cmssdt.cern.ch/SDT/jenkins-artifacts/pull-request-integration/PR-647cd0/23803/git-recent-commits.json
https://cmssdt.cern.ch/SDT/jenkins-artifacts/pull-request-integration/PR-647cd0/23803/git-merge-result

Unit Tests

I found errors in the following unit tests:

---> test DRNTest had ERRORS
---> test testFWCoreUtilities had ERRORS
---> test import-tables had ERRORS
---> test testTables had ERRORS
and more ...

Copy link
Contributor

@smuzaffar smuzaffar left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

looks good to go in

@smuzaffar
Copy link
Contributor

@aandvalenzuela , this looks good to go in, do you plan to update it ?

@aandvalenzuela aandvalenzuela marked this pull request as ready for review April 11, 2022 09:47
@smuzaffar smuzaffar merged commit 11b0aab into cms-sw:IB/CMSSW_12_4_X/master Apr 11, 2022
@aandvalenzuela aandvalenzuela deleted the update-packages branch April 27, 2022 13:25
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging this pull request may close these issues.

None yet

4 participants