Add ppRef modifier instead of a GT #35909

tvami · 2021-10-29T13:25:13Z

PR description:

A technical change so that we don't need to create ppRef GTs anymore. Remove phase1_2017_realistic_ppref from autoCond and include it in the autoCondModifier

PR validation:

runTheMatrix.py -l 149.0 runs fine

if this PR is a backport please specify the original PR and why you need to backport that PR:

N/A

tvami · 2021-10-29T13:25:48Z

test parameters:

workflows = 149.0
pull_request = introduce COPYPASTER3 and use it as step2 in wf 138.3 #35890

cmsbuild · 2021-10-29T13:30:44Z

+code-checks

Logs: https://cmssdt.cern.ch/SDT/code-checks/cms-sw-PR-35909/26305

This PR adds an extra 12KB to repository

cmsbuild · 2021-10-29T13:31:08Z

A new Pull Request was created by @tvami (Tamas Vami) for master.

It involves the following packages:

Configuration/AlCa (alca)

@cmsbuild, @malbouis, @tvami, @yuanchao, @francescobrivio can you please review it and eventually sign? Thanks.
@Martin-Grunewald, @missirol, @mmusich, @fabiocos, @tocheng this is something you requested to watch as well.
@perrotta, @dpiparo, @qliphy you are the release manager for this.

cms-bot commands are listed here

tvami · 2021-10-29T13:31:21Z

@cmsbuild , please test

cmsbuild · 2021-10-29T17:49:52Z

-1

Failed Tests: UnitTests
Summary: https://cmssdt.cern.ch/SDT/jenkins-artifacts/pull-request-integration/PR-47c0c9/20065/summary.html
COMMIT: 6d80013
CMSSW: CMSSW_12_2_X_2021-10-28-2300/slc7_amd64_gcc900
User test area: For local testing, you can use /cvmfs/cms-ci.cern.ch/week0/cms-sw/cmssw/35909/20065/install.sh to create a dev area with all the needed externals and cmssw changes.

Unit Tests

I found errors in the following unit tests:

---> test TestDQMServicesDemo had ERRORS

Comparison Summary

@slava77 comparisons for the following workflows were not done due to missing matrix map:

/data/cmsbld/jenkins/workspace/compare-root-files-short-matrix/data/PR-47c0c9/149.0_QCD_Pt_80_120_13_PPREF+QCD_Pt_80_120_13_PPREF+DIGIPPREF2017+RECOPPREF2017+HARVESTPPREF2017

Summary:

No significant changes to the logs found
Reco comparison results: 2 differences found in the comparisons
DQMHistoTests: Total files compared: 42
DQMHistoTests: Total histograms compared: 2901440
DQMHistoTests: Total failures: 5
DQMHistoTests: Total nulls: 1
DQMHistoTests: Total successes: 2901412
DQMHistoTests: Total skipped: 22
DQMHistoTests: Total Missing objects: 0
DQMHistoSizes: Histogram memory added: 0.004 KiB( 41 files compared)
DQMHistoSizes: changed ( 312.0 ): 0.004 KiB MessageLogger/Warnings
Checked 177 log files, 37 edm output root files, 42 DQM output files
TriggerResults: no differences found

malbouis · 2021-10-30T08:56:02Z

The 'phase1_2017_realistic_ppref' key is based on the 'phase1_2017_realistic' with the addition of the tags included in the difference of 120X_mc2017_realistic_v1/120X_mc2017_realistic_forppRef5TeV_v2.

conddb diff 120X_mc2017_realistic_v1 120X_mc2017_realistic_forppRef5TeV_v2

BeamSpotObjects_pp_2017G_MC_2021sample_for_UL
EcalLaserAPDPNRatios_mc_Run2017G_306580
EcalPedestals_Run2017G_306580
EcalTPGLinearizationConst_Run2017G_306580
EcalTPGPedestals_Run2017G_306580
L1Menu_pp502Collisions2017_v4_m6_xml

The Modifier autoCond2017ppRef5TeV introduced in this PR takes exactly the same records/tags as above.

the unit test failure from above is not related to this PR.

malbouis · 2021-10-30T08:59:37Z

+alca

cmsbuild · 2021-10-30T08:59:59Z

This pull request is fully signed and it will be integrated in one of the next master IBs (but tests are reportedly failing). This pull request will now be reviewed by the release team before it's merged. @perrotta, @dpiparo, @qliphy (and backports should be raised in the release meeting by the corresponding L2)

qliphy · 2021-10-30T13:10:23Z

@tvami There is conflict to be resolved after merging #35914

tvami · 2021-10-30T13:25:43Z

@cmsbuild , please test

cmsbuild · 2021-10-30T13:32:39Z

+code-checks

Logs: https://cmssdt.cern.ch/SDT/code-checks/cms-sw-PR-35909/26319

This PR adds an extra 16KB to repository

cmsbuild · 2021-10-30T13:33:01Z

Pull request #35909 was updated. @malbouis, @tvami, @yuanchao, @francescobrivio can you please check and sign again.

cmsbuild · 2021-10-30T15:15:06Z

-1

Failed Tests: UnitTests RelVals RelVals-INPUT AddOn
Summary: https://cmssdt.cern.ch/SDT/jenkins-artifacts/pull-request-integration/PR-47c0c9/20103/summary.html
COMMIT: 1c83752
CMSSW: CMSSW_12_2_X_2021-10-30-1100/slc7_amd64_gcc900
User test area: For local testing, you can use /cvmfs/cms-ci.cern.ch/week0/cms-sw/cmssw/35909/20103/install.sh to create a dev area with all the needed externals and cmssw changes.

The following merge commits were also included on top of IB + this PR after doing git cms-merge-topic:

You can see more details here:
https://cmssdt.cern.ch/SDT/jenkins-artifacts/pull-request-integration/PR-47c0c9/20103/git-recent-commits.json
https://cmssdt.cern.ch/SDT/jenkins-artifacts/pull-request-integration/PR-47c0c9/20103/git-merge-result

Unit Tests

I found errors in the following unit tests:

---> test TestDQMOnlineClient-ctpps_dqm_sourceclient had ERRORS
---> test TestDQMOnlineClient-ecal_dqm_sourceclient had ERRORS
---> test TestDQMOnlineClient-csc_dqm_sourceclient had ERRORS
---> test TestDQMServicesDemo had ERRORS
and more ...

RelVals

135.4

----- Begin Fatal Exception 30-Oct-2021 16:22:00 CEST-----------------------
An exception of category 'EventSetupConflict' occurred while
   [0] Calling beginJob
Exception Message:
two EventSetup Sources want to deliver type="PPSAssociationCuts" label=""
 from record PPSAssociationCutsRcd. The two providers are 
1) type="PPSAssociationCutsESSource" label="ppsAssociationCutsESSource"
2) type="PoolDBESSource" label="GlobalTag"
Please either
   remove one of these Sources
   or find a way of configuring one of them so it does not deliver this data
   or use an es_prefer statement in the configuration to choose one.
----- End Fatal Exception -------------------------------------------------

136.7611

----- Begin Fatal Exception 30-Oct-2021 16:22:31 CEST-----------------------
An exception of category 'EventSetupConflict' occurred while
   [0] Calling beginJob
Exception Message:
two EventSetup Sources want to deliver type="PPSAssociationCuts" label=""
 from record PPSAssociationCutsRcd. The two providers are 
1) type="PPSAssociationCutsESSource" label="ppsAssociationCutsESSource"
2) type="PoolDBESSource" label="GlobalTag"
Please either
   remove one of these Sources
   or find a way of configuring one of them so it does not deliver this data
   or use an es_prefer statement in the configuration to choose one.
----- End Fatal Exception -------------------------------------------------

136.8311

----- Begin Fatal Exception 30-Oct-2021 16:22:31 CEST-----------------------
An exception of category 'EventSetupConflict' occurred while
   [0] Calling beginJob
Exception Message:
two EventSetup Sources want to deliver type="PPSAssociationCuts" label=""
 from record PPSAssociationCutsRcd. The two providers are 
1) type="PPSAssociationCutsESSource" label="ppsAssociationCutsESSource"
2) type="PoolDBESSource" label="GlobalTag"
Please either
   remove one of these Sources
   or find a way of configuring one of them so it does not deliver this data
   or use an es_prefer statement in the configuration to choose one.
----- End Fatal Exception -------------------------------------------------

Expand to see more relval errors ...

RelVals-INPUT

4.64.6_MinimumBias2010A+MinimumBias2010A+RECOSKIMALCA+HARVESTDR1/step2_MinimumBias2010A+MinimumBias2010A+RECOSKIMALCA+HARVESTDR1.log
136.72411136.72411_RunJetHT2016B_reminiaodUL+RunJetHT2016B_reminiaodUL+REMINIAOD_data2016UL_HIPM+HARVESTDR2_REMINIAOD_data2016UL_HIPM/step2_RunJetHT2016B_reminiaodUL+RunJetHT2016B_reminiaodUL+REMINIAOD_data2016UL_HIPM+HARVESTDR2_REMINIAOD_data2016UL_HIPM.log
136.72412136.72412_RunJetHT2016B_reminiaodUL+RunJetHT2016B_reminiaodUL+REMININANO_data2016UL_HIPM+HARVESTDR2_REMININANO_data2016UL_HIPM/step2_RunJetHT2016B_reminiaodUL+RunJetHT2016B_reminiaodUL+REMININANO_data2016UL_HIPM+HARVESTDR2_REMININANO_data2016UL_HIPM.log

Expand to see more relval errors ...

AddOn Tests

fastsim1

----- Begin Fatal Exception 30-Oct-2021 16:16:04 CEST-----------------------
An exception of category 'EventSetupConflict' occurred while
   [0] Calling beginJob
Exception Message:
two EventSetup Sources want to deliver type="PPSAssociationCuts" label=""
 from record PPSAssociationCutsRcd. The two providers are 
1) type="PPSAssociationCutsESSource" label="ppsAssociationCutsESSource"
2) type="PoolDBESSource" label="GlobalTag"
Please either
   remove one of these Sources
   or find a way of configuring one of them so it does not deliver this data
   or use an es_prefer statement in the configuration to choose one.
----- End Fatal Exception -------------------------------------------------

fastsim2

----- Begin Fatal Exception 30-Oct-2021 16:16:05 CEST-----------------------
An exception of category 'EventSetupConflict' occurred while
   [0] Calling beginJob
Exception Message:
two EventSetup Sources want to deliver type="PPSAssociationCuts" label=""
 from record PPSAssociationCutsRcd. The two providers are 
1) type="PPSAssociationCutsESSource" label="ppsAssociationCutsESSource"
2) type="PoolDBESSource" label="GlobalTag"
Please either
   remove one of these Sources
   or find a way of configuring one of them so it does not deliver this data
   or use an es_prefer statement in the configuration to choose one.
----- End Fatal Exception -------------------------------------------------

hlt_mc_Fake1

----- Begin Fatal Exception 30-Oct-2021 16:20:41 CEST-----------------------
An exception of category 'EventSetupConflict' occurred while
   [0] Calling beginJob
Exception Message:
two EventSetup Sources want to deliver type="PPSAssociationCuts" label=""
 from record PPSAssociationCutsRcd. The two providers are 
1) type="PPSAssociationCutsESSource" label="ppsAssociationCutsESSource"
2) type="PoolDBESSource" label="GlobalTag"
Please either
   remove one of these Sources
   or find a way of configuring one of them so it does not deliver this data
   or use an es_prefer statement in the configuration to choose one.
----- End Fatal Exception -------------------------------------------------

Expand to see more addon errors ...

tvami · 2021-10-30T18:10:42Z

@cmsbuild , please test with #35766

cmsbuild · 2021-10-30T22:05:18Z

-1

Failed Tests: UnitTests
Summary: https://cmssdt.cern.ch/SDT/jenkins-artifacts/pull-request-integration/PR-47c0c9/20111/summary.html
COMMIT: 1c83752
CMSSW: CMSSW_12_2_X_2021-10-30-1100/slc7_amd64_gcc900
User test area: For local testing, you can use /cvmfs/cms-ci.cern.ch/week0/cms-sw/cmssw/35909/20111/install.sh to create a dev area with all the needed externals and cmssw changes.

The following merge commits were also included on top of IB + this PR after doing git cms-merge-topic:

You can see more details here:
https://cmssdt.cern.ch/SDT/jenkins-artifacts/pull-request-integration/PR-47c0c9/20111/git-recent-commits.json
https://cmssdt.cern.ch/SDT/jenkins-artifacts/pull-request-integration/PR-47c0c9/20111/git-merge-result

Unit Tests

I found errors in the following unit tests:

---> test TestDQMServicesDemo had ERRORS

Comparison Summary

@slava77 comparisons for the following workflows were not done due to missing matrix map:

/data/cmsbld/jenkins/workspace/compare-root-files-short-matrix/data/PR-47c0c9/149.0_QCD_Pt_80_120_13_PPREF+QCD_Pt_80_120_13_PPREF+DIGIPPREF2017+RECOPPREF2017+HARVESTPPREF2017

Summary:

No significant changes to the logs found
Reco comparison results: 0 differences found in the comparisons
DQMHistoTests: Total files compared: 42
DQMHistoTests: Total histograms compared: 2901440
DQMHistoTests: Total failures: 0
DQMHistoTests: Total nulls: 0
DQMHistoTests: Total successes: 2901418
DQMHistoTests: Total skipped: 22
DQMHistoTests: Total Missing objects: 0
DQMHistoSizes: Histogram memory added: 0.0 KiB( 41 files compared)
Checked 177 log files, 37 edm output root files, 42 DQM output files
TriggerResults: no differences found

malbouis · 2021-10-30T22:10:44Z

+alca

cmsbuild · 2021-10-30T22:11:06Z

This pull request is fully signed and it will be integrated in one of the next master IBs (but tests are reportedly failing). This pull request will now be reviewed by the release team before it's merged. @perrotta, @dpiparo, @qliphy (and backports should be raised in the release meeting by the corresponding L2)

tvami · 2021-10-30T22:13:42Z

Results are as expected

unit test fails due to Unit test failures in DQMServices/Demo and RecoTauTag/HLTProducers in the 12_1_X IBs #35868
Jenkins diffs are zero
more importantly 149.0 runs fine (this is he wf that uses the ppRef GT)

tvami · 2021-10-30T22:15:12Z

@tvami There is conflict to be resolved after merging #35914

Hi @qliphy conflict is resolved and tests have finished nicely (except for the known unit test issue)

qliphy · 2021-10-31T01:37:29Z

merge

Add ppRef modifier instead of a GT

6d80013

cmsbuild added this to the CMSSW_12_2_X milestone Oct 29, 2021

cmsbuild added alca-pending code-checks-pending orp-pending pending-signatures tests-pending labels Oct 29, 2021

cmsbuild added the requires-external label Oct 29, 2021

cmsbuild added code-checks-approved and removed code-checks-pending labels Oct 29, 2021

cmsbuild added tests-started and removed tests-pending labels Oct 29, 2021

cmsbuild mentioned this pull request Oct 29, 2021

Include PPS Association Cuts in data and MC GTs #35914

Merged

cmsbuild added tests-rejected and removed tests-started labels Oct 29, 2021

cmsbuild added alca-approved fully-signed and removed alca-pending pending-signatures labels Oct 30, 2021

Merge branch 'master' into ModifierForPPref

1c83752

cmsbuild removed the alca-approved label Oct 30, 2021

cmsbuild added tests-started and removed tests-pending labels Oct 30, 2021

cmsbuild added code-checks-approved and removed code-checks-pending labels Oct 30, 2021

cmsbuild added tests-rejected and removed tests-started labels Oct 30, 2021

cmsbuild added tests-started and removed tests-rejected labels Oct 30, 2021

cmsbuild added tests-rejected and removed tests-started labels Oct 30, 2021

cmsbuild added alca-approved fully-signed and removed alca-pending pending-signatures labels Oct 30, 2021

cmsbuild added orp-approved and removed orp-pending labels Oct 31, 2021

cmsbuild merged commit b35ff2e into cms-sw:master Oct 31, 2021

This was referenced Oct 31, 2021

Include reduced material DD4HEP geometry scenarios in Run-3 MC GTs #35920

Merged

rootmodule: Updated root to tip of branch master cms-sw/cmsdist#7421

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Add ppRef modifier instead of a GT #35909

Add ppRef modifier instead of a GT #35909

tvami commented Oct 29, 2021

tvami commented Oct 29, 2021

cmsbuild commented Oct 29, 2021

cmsbuild commented Oct 29, 2021

tvami commented Oct 29, 2021

cmsbuild commented Oct 29, 2021

malbouis commented Oct 30, 2021 •

edited

Loading

malbouis commented Oct 30, 2021

cmsbuild commented Oct 30, 2021

qliphy commented Oct 30, 2021

tvami commented Oct 30, 2021

cmsbuild commented Oct 30, 2021

cmsbuild commented Oct 30, 2021

cmsbuild commented Oct 30, 2021

tvami commented Oct 30, 2021

cmsbuild commented Oct 30, 2021

malbouis commented Oct 30, 2021

cmsbuild commented Oct 30, 2021

tvami commented Oct 30, 2021

tvami commented Oct 30, 2021

qliphy commented Oct 31, 2021

Add ppRef modifier instead of a GT #35909

Add ppRef modifier instead of a GT #35909

Conversation

tvami commented Oct 29, 2021

PR description:

PR validation:

if this PR is a backport please specify the original PR and why you need to backport that PR:

tvami commented Oct 29, 2021

cmsbuild commented Oct 29, 2021

cmsbuild commented Oct 29, 2021

tvami commented Oct 29, 2021

cmsbuild commented Oct 29, 2021

Unit Tests

Comparison Summary

malbouis commented Oct 30, 2021 • edited Loading

malbouis commented Oct 30, 2021

cmsbuild commented Oct 30, 2021

qliphy commented Oct 30, 2021

tvami commented Oct 30, 2021

cmsbuild commented Oct 30, 2021

cmsbuild commented Oct 30, 2021

cmsbuild commented Oct 30, 2021

Unit Tests

RelVals

RelVals-INPUT

AddOn Tests

tvami commented Oct 30, 2021

cmsbuild commented Oct 30, 2021

Unit Tests

Comparison Summary

malbouis commented Oct 30, 2021

cmsbuild commented Oct 30, 2021

tvami commented Oct 30, 2021

tvami commented Oct 30, 2021

qliphy commented Oct 31, 2021

malbouis commented Oct 30, 2021 •

edited

Loading