Fix check for requirements (#7534), add missing dependencies #7537

iarspider · 2022-01-11T08:45:21Z

@smuzaffar I can move c823a9b (proposed fix for #7534) to a separate PR to avoid rebuilding unchanged python packages if necessary.

cmsbuild · 2022-01-11T08:45:45Z

A new Pull Request was created by @iarspider for branch IB/CMSSW_12_3_X/master.

@cmsbuild, @smuzaffar, @iarspider can you please review it and eventually sign? Thanks.
@perrotta, @dpiparo, @qliphy you are the release manager for this.
cms-bot commands are listed here

iarspider · 2022-01-11T09:34:33Z

please test

iarspider · 2022-01-11T09:39:03Z

There are some dependencies (build ones) that are not caught by pip check. This is a side-effect of using --no-build-isolation. For example, py3-correctionlib actually requires py3-make and py3-cmake (although it seems that it can be installed even if these packages are missing and only make and cmake commands are available).

cmsbuild · 2022-01-11T10:34:13Z

Pull request #7537 was updated.

cmsbuild · 2022-01-11T10:36:28Z

Pull request #7537 was updated.

smuzaffar · 2022-01-11T10:43:39Z

@iarspider , yes please remove the change for build-with-pip-file. I think all you need is to add the following in

cmsdist/pip/cachecontrol.file

%define patchsrc0 sed -i -e 's|name="CacheControl"|name="cachecontrol"|' setup.py

cmsdist/pip/keyring.file

%define patchsrc0 sed -i -e 's|\\[metadata\\]|[metadata]\\nversion = %{realversion}|' setup.cfg

note that keyring version is set to 0.0.0.0 and check at https://github.com/cms-sw/cmsdist/blob/IB/CMSSW_12_3_X/master/build-with-pip.file#L107 is still going to fail as we also search for %realversion.

I think we can improve build-with-with.file but I would suggest to change it only when we have updated gcc, python3, pip3 or setuptools

smuzaffar · 2022-01-11T10:45:35Z

I have a rust.spec which allows to use or llvm, I will open a PR which you can use for this PR test

iarspider · 2022-01-11T10:47:42Z

@smuzaffar py3-keyring should be fixed now (let me check once more) - the version is set with using py3-setuptools-scm, which was one of the missing deps. For py3-cachecontrol, this won't be necessary once we fix build-with-pip.file. For rust - this would be useful, thanks!

smuzaffar · 2022-01-11T10:51:14Z

see #7538 for rust.
I would suggest to fix cachecontrol.file and undo the build-with-pip.file changes. I have a better fix for build-with-pip.file which we will only include when we will updateany base poackage. For now, it is better to include these changes in 12.3.X without massive rebuilt

iarspider · 2022-01-11T10:52:52Z

I would suggest to fix cachecontrol.file and undo the build-with-pip.file changes. I have a better fix for build-with-pip.file which we will only include when we will updateany base poackage. For now, it is better to include these changes in 12.3.X without massive rebuilt

Sure. I have a few more fixes (for build-time dependencies), will push them soon.

build-with-pip.file

cmsbuild · 2022-01-12T21:30:13Z

Pull request #7537 was updated.

cmsbuild · 2022-01-13T13:46:49Z

Pull request #7537 was updated.

iarspider · 2022-01-13T13:47:33Z

please test

smuzaffar · 2022-01-13T13:49:07Z

pip/correctionlib.file

@@ -1,3 +1,3 @@
 Requires: py3-pybind11 py3-numpy py3-pydantic py3-python-rapidjson py3-rich
-BuildRequires: py3-setuptools-scm py3-scikit-build py3-cmake py3-make
+BuildRequires: py3-setuptools-scm py3-scikit-build cmake py3-make


do we need py3-make ?

Technically, yes. Actually - not sure.

cmsbuild · 2022-01-13T14:26:31Z

Pull request #7537 was updated.

cmsbuild · 2022-01-13T20:48:38Z

+1

Summary: https://cmssdt.cern.ch/SDT/jenkins-artifacts/pull-request-integration/PR-676bf4/21696/summary.html
COMMIT: 722f517
CMSSW: CMSSW_12_3_X_2022-01-12-2300/slc7_amd64_gcc10
User test area: For local testing, you can use /cvmfs/cms-ci.cern.ch/week1/cms-sw/cmsdist/7537/21696/install.sh to create a dev area with all the needed externals and cmssw changes.

The following merge commits were also included on top of IB + this PR after doing git cms-merge-topic:

@cms-sw Update UTM to version 0.10.0 #7543

You can see more details here:
https://cmssdt.cern.ch/SDT/jenkins-artifacts/pull-request-integration/PR-676bf4/21696/git-recent-commits.json
https://cmssdt.cern.ch/SDT/jenkins-artifacts/pull-request-integration/PR-676bf4/21696/git-merge-result

Comparison Summary

Summary:

No significant changes to the logs found
Reco comparison results: 5 differences found in the comparisons
DQMHistoTests: Total files compared: 43
DQMHistoTests: Total histograms compared: 3461659
DQMHistoTests: Total failures: 5
DQMHistoTests: Total nulls: 1
DQMHistoTests: Total successes: 3461631
DQMHistoTests: Total skipped: 22
DQMHistoTests: Total Missing objects: 0
DQMHistoSizes: Histogram memory added: -0.004 KiB( 42 files compared)
DQMHistoSizes: changed ( 312.0 ): -0.004 KiB MessageLogger/Warnings
Checked 181 log files, 42 edm output root files, 43 DQM output files
TriggerResults: no differences found

iarspider · 2022-01-14T09:56:01Z

please test for slc7_ppc64le_gcc11

smuzaffar · 2022-01-14T10:20:09Z

please test for slc7_aarch64_gcc11

smuzaffar · 2022-01-14T10:20:48Z

please test for cs8_amd64_gcc11

cmsbuild · 2022-01-14T16:28:47Z

+1

Summary: https://cmssdt.cern.ch/SDT/jenkins-artifacts/pull-request-integration/PR-676bf4/21717/summary.html
COMMIT: 722f517
CMSSW: CMSSW_12_3_X_2022-01-13-2300/slc7_ppc64le_gcc11
User test area: For local testing, you can use /cvmfs/cms-ci.cern.ch/week1/cms-sw/cmsdist/7537/21717/install.sh to create a dev area with all the needed externals and cmssw changes.

cmsbuild · 2022-01-14T19:22:49Z

-1

Failed Tests: RelVals
Summary: https://cmssdt.cern.ch/SDT/jenkins-artifacts/pull-request-integration/PR-676bf4/21725/summary.html
COMMIT: 722f517
CMSSW: CMSSW_12_3_X_2022-01-13-2300/slc7_aarch64_gcc11
User test area: For local testing, you can use /cvmfs/cms-ci.cern.ch/week1/cms-sw/cmsdist/7537/21725/install.sh to create a dev area with all the needed externals and cmssw changes.

The following merge commits were also included on top of IB + this PR after doing git cms-merge-topic:

@cms-sw Update py-pillow to version 9.0.0 #7545

You can see more details here:
https://cmssdt.cern.ch/SDT/jenkins-artifacts/pull-request-integration/PR-676bf4/21725/git-recent-commits.json
https://cmssdt.cern.ch/SDT/jenkins-artifacts/pull-request-integration/PR-676bf4/21725/git-merge-result

RelVals

4.53

----- Begin Fatal Exception 14-Jan-2022 19:15:49 CET-----------------------
An exception of category 'Vertex' occurred while
   [0] Processing  Event run: 194533 lumi: 329 event: 462355458 stream: 0
   [1] Running path 'dqmofflineOnPAT_1_step'
   [2] Prefetching for module SingleTopTChannelLeptonDQM_miniAOD/'singleTopElectronMediumDQM_miniAOD'
   [3] Prefetching for module PATMuonSlimmer/'slimmedMuons'
   [4] Prefetching for module PATMuonSelector/'selectedPatMuons'
   [5] Prefetching for module PATMuonProducer/'patMuons'
   [6] Prefetching for module MuonProducer/'muons'
   [7] Prefetching for module PFProducer/'particleFlowTmp'
   [8] Prefetching for module PFBlockProducer/'particleFlowBlock'
   [9] Prefetching for module PFElecTkProducer/'pfTrackElec'
   [10] Prefetching for module PFConversionProducer/'pfConversions'
   [11] Calling method for module ConversionProducer/'allConversions'
Exception Message:
Refitted track not found in list
----- End Fatal Exception -------------------------------------------------

140.56

----- Begin Fatal Exception 14-Jan-2022 19:27:36 CET-----------------------
An exception of category 'Vertex' occurred while
   [0] Processing  Event run: 326479 lumi: 7 event: 1579493 stream: 0
   [1] Running path 'dqmoffline_8_step'
   [2] Prefetching for module SMPDQM/'SMPDQM'
   [3] Prefetching for module MuonProducer/'muons'
   [4] Prefetching for module PFProducer/'particleFlowTmp'
   [5] Prefetching for module PFBlockProducer/'particleFlowBlock'
   [6] Prefetching for module PFElecTkProducer/'pfTrackElec'
   [7] Prefetching for module PFConversionProducer/'pfConversions'
   [8] Calling method for module ConversionProducer/'allConversions'
Exception Message:
Refitted track not found in list
----- End Fatal Exception -------------------------------------------------

136.874

----- Begin Fatal Exception 14-Jan-2022 19:44:12 CET-----------------------
An exception of category 'Vertex' occurred while
   [0] Processing  Event run: 319450 lumi: 76 event: 106007323 stream: 0
   [1] Running path 'dqmoffline_10_step'
   [2] Prefetching for module SMPDQM/'SMPDQM'
   [3] Prefetching for module MuonProducer/'muons'
   [4] Prefetching for module PFProducer/'particleFlowTmp'
   [5] Prefetching for module PFBlockProducer/'particleFlowBlock'
   [6] Prefetching for module PFElecTkProducer/'pfTrackElec'
   [7] Prefetching for module PFConversionProducer/'pfConversions'
   [8] Calling method for module ConversionProducer/'allConversions'
Exception Message:
Refitted track not found in list
----- End Fatal Exception -------------------------------------------------

iarspider · 2022-01-14T20:36:29Z

@smuzaffar I think it's ok to merge. The error doesn't seem to be related to PR

smuzaffar · 2022-01-14T20:57:07Z

+externals

cmsbuild · 2022-01-14T20:57:31Z

This pull request is fully signed and it will be integrated in one of the next IB/CMSSW_12_3_X/master IBs (tests are also fine). This pull request will now be reviewed by the release team before it's merged. @perrotta, @dpiparo, @qliphy (and backports should be raised in the release meeting by the corresponding L2)

cmsbuild · 2022-01-14T22:25:15Z

+1

Summary: https://cmssdt.cern.ch/SDT/jenkins-artifacts/pull-request-integration/PR-676bf4/21719/summary.html
COMMIT: 722f517
CMSSW: CMSSW_12_3_X_2022-01-13-2300/cs8_amd64_gcc11
User test area: For local testing, you can use /cvmfs/cms-ci.cern.ch/week1/cms-sw/cmsdist/7537/21719/install.sh to create a dev area with all the needed externals and cmssw changes.

Comparison Summary

Summary:

No significant changes to the logs found
Reco comparison results: 65790 differences found in the comparisons
DQMHistoTests: Total files compared: 43
DQMHistoTests: Total histograms compared: 3461659
DQMHistoTests: Total failures: 477294
DQMHistoTests: Total nulls: 311
DQMHistoTests: Total successes: 2984032
DQMHistoTests: Total skipped: 22
DQMHistoTests: Total Missing objects: 0
DQMHistoSizes: Histogram memory added: -0.346 KiB( 42 files compared)
DQMHistoSizes: changed ( 10224.0 ): -0.288 KiB SiStrip/MechanicalView
DQMHistoSizes: changed ( 11834.0 ): 2.191 KiB SiStrip/MechanicalView
DQMHistoSizes: changed ( 250202.181 ): -0.111 KiB SiStrip/MechanicalView
DQMHistoSizes: changed ( 25202.0 ): -0.129 KiB SiStrip/MechanicalView
DQMHistoSizes: changed ( 7.3 ): -2.009 KiB SiStrip/MechanicalView
Checked 181 log files, 42 edm output root files, 43 DQM output files
TriggerResults: found differences in 14 / 42 workflows

iarspider added 7 commits January 10, 2022 11:02

Fix check for requirements (#7534), start adding missing dependencies

c823a9b

Fix EOLs

15b2517

Try fixing rust spec

07ae1c2

Try fixing rust spec

4eab3a9

Fixed rust.spec

5c5b724

Done py3-keyring and dependenices

344c491

Fix missing dependencies of python packages

18d62e0

iarspider requested a review from smuzaffar January 11, 2022 08:45

cmsbuild added externals-pending orp-pending pending-signatures tests-pending labels Jan 11, 2022

cmsbuild added tests-started and removed tests-pending labels Jan 11, 2022

Add missing built-time dependencies for py3-correctionlib

8583d1d

cmsbuild removed the tests-started label Jan 11, 2022

cmsbuild added the tests-pending label Jan 11, 2022

Fix case

cdd1594

added rust 1.57.0

33e8450

just to make sure that rust is build with PRs

9b78386

smuzaffar reviewed Jan 12, 2022

View reviewed changes

build-with-pip.file Outdated Show resolved Hide resolved

Fixes for py3-jinja2-tome and py3-opt-einsum

77378cd

Remove py3-cmake

bdaf12d

Revert changes to build-with-pip.file

722f517

cmsbuild added tests-started and removed tests-pending labels Jan 13, 2022

smuzaffar reviewed Jan 13, 2022

View reviewed changes

cmsbuild added tests-approved and removed tests-started labels Jan 13, 2022

cms-sw deleted a comment from iarspider Jan 14, 2022

cmsbuild added externals-approved fully-signed and removed externals-pending pending-signatures labels Jan 14, 2022

smuzaffar merged commit 1d38608 into IB/CMSSW_12_3_X/master Jan 14, 2022

iarspider deleted the fix-py-deps branch January 17, 2022 08:32

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Fix check for requirements (#7534), add missing dependencies #7537

Fix check for requirements (#7534), add missing dependencies #7537

iarspider commented Jan 11, 2022

cmsbuild commented Jan 11, 2022

iarspider commented Jan 11, 2022

iarspider commented Jan 11, 2022

cmsbuild commented Jan 11, 2022

cmsbuild commented Jan 11, 2022

smuzaffar commented Jan 11, 2022

smuzaffar commented Jan 11, 2022

iarspider commented Jan 11, 2022

smuzaffar commented Jan 11, 2022

iarspider commented Jan 11, 2022

cmsbuild commented Jan 12, 2022

cmsbuild commented Jan 13, 2022

iarspider commented Jan 13, 2022

smuzaffar Jan 13, 2022

iarspider Jan 13, 2022

cmsbuild commented Jan 13, 2022

cmsbuild commented Jan 13, 2022

iarspider commented Jan 14, 2022

smuzaffar commented Jan 14, 2022

smuzaffar commented Jan 14, 2022

cmsbuild commented Jan 14, 2022

cmsbuild commented Jan 14, 2022

iarspider commented Jan 14, 2022

smuzaffar commented Jan 14, 2022

cmsbuild commented Jan 14, 2022

cmsbuild commented Jan 14, 2022

Fix check for requirements (#7534), add missing dependencies #7537

Fix check for requirements (#7534), add missing dependencies #7537

Conversation

iarspider commented Jan 11, 2022

cmsbuild commented Jan 11, 2022

iarspider commented Jan 11, 2022

iarspider commented Jan 11, 2022

cmsbuild commented Jan 11, 2022

cmsbuild commented Jan 11, 2022

smuzaffar commented Jan 11, 2022

smuzaffar commented Jan 11, 2022

iarspider commented Jan 11, 2022

smuzaffar commented Jan 11, 2022

iarspider commented Jan 11, 2022

cmsbuild commented Jan 12, 2022

cmsbuild commented Jan 13, 2022

iarspider commented Jan 13, 2022

smuzaffar Jan 13, 2022

Choose a reason for hiding this comment

iarspider Jan 13, 2022

Choose a reason for hiding this comment

cmsbuild commented Jan 13, 2022

cmsbuild commented Jan 13, 2022

Comparison Summary

iarspider commented Jan 14, 2022

smuzaffar commented Jan 14, 2022

smuzaffar commented Jan 14, 2022

cmsbuild commented Jan 14, 2022

cmsbuild commented Jan 14, 2022

RelVals

iarspider commented Jan 14, 2022

smuzaffar commented Jan 14, 2022

cmsbuild commented Jan 14, 2022

cmsbuild commented Jan 14, 2022

Comparison Summary