Include reduced material DD4HEP geometry scenarios in Run-3 MC GTs #35920

tvami · 2021-10-31T10:57:58Z

PR description:

Include reduced material DD4hep geometry scenarios in Run-3 MC GTs as requested in [1]

I've created new GTs

121X_mcRun3_2021_design_v16
https://cms-conddb.cern.ch/cmsDbBrowser/diff/Prod/gts/121X_mcRun3_2021_design_v15/121X_mcRun3_2021_design_v16
* 121X_mcRun3_2021_realistic_v18
https://cms-conddb.cern.ch/cmsDbBrowser/diff/Prod/gts/121X_mcRun3_2021_realistic_v17/121X_mcRun3_2021_realistic_v18
* 121X_mcRun3_2021cosmics_realistic_deco_v18
https://cms-conddb.cern.ch/cmsDbBrowser/diff/Prod/gts/121X_mcRun3_2021cosmics_realistic_deco_v17/121X_mcRun3_2021cosmics_realistic_deco_v18
121X_mcRun3_2021_realistic_HI_v18
https://cms-conddb.cern.ch/cmsDbBrowser/diff/Prod/gts/121X_mcRun3_2021_realistic_HI_v17/121X_mcRun3_2021_realistic_HI_v18
121X_mcRun3_2023_realistic_v17
https://cms-conddb.cern.ch/cmsDbBrowser/diff/Prod/gts/121X_mcRun3_2023_realistic_v16/121X_mcRun3_2023_realistic_v17
121X_mcRun3_2024_realistic_v17
https://cms-conddb.cern.ch/cmsDbBrowser/diff/Prod/gts/121X_mcRun3_2024_realistic_v16/121X_mcRun3_2024_realistic_v17

The diffs are only in the requested tags of

XMLFILE_Geometry_121DD4hepV2_Extended2021ZeroMaterial_mc, label: ZeroMaterial

XMLFILE_Geometry_121DD4hepV2_Extended2021FlatMinus05Percent_mc, label: FlatMinus05Percent

XMLFILE_Geometry_121DD4hepV2_Extended2021FlatMinus10Percent_mc, label: FlatMinus10Percent

XMLFILE_Geometry_121DD4hepV2_Extended2021FlatPlus05Percent_mc, label: FlatPlus05Percent

XMLFILE_Geometry_121DD4hepV2_Extended2021FlatPlus10Percent_mc, label: FlatPlus10Percent

[1] https://hypernews.cern.ch/HyperNews/CMS/get/calibrations/4510.html

PR validation:

The tags are not consumed by default but only in special campaigns.

if this PR is a backport please specify the original PR and why you need to backport that PR:

N/A

cc @cvuosalo

resolves cms-AlCaDB/AlCaTools#47

tvami · 2021-10-31T11:00:23Z

@cmsbuild , please test

cmsbuild · 2021-10-31T11:05:29Z

+code-checks

Logs: https://cmssdt.cern.ch/SDT/code-checks/cms-sw-PR-35920/26325

This PR adds an extra 12KB to repository

cmsbuild · 2021-10-31T11:05:52Z

A new Pull Request was created by @tvami (Tamas Vami) for master.

It involves the following packages:

Configuration/AlCa (alca)

@malbouis, @tvami, @yuanchao, @francescobrivio can you please review it and eventually sign? Thanks.
@Martin-Grunewald, @missirol, @mmusich, @fabiocos, @tocheng this is something you requested to watch as well.
@perrotta, @dpiparo, @qliphy you are the release manager for this.

cms-bot commands are listed here

francescobrivio · 2021-10-31T11:23:22Z

@tvami although for mcRun3 GTs the DD4HEP geometry is the de fault one, maybe it's worth specifying it also in the PR title/description that you added the dd4hep zero material tags. What do you think?

(The GT diffs look very clean BTW 😄)

tvami · 2021-10-31T11:30:37Z

@tvami although for mcRun3 GTs the DD4HEP geometry is the de fault one, maybe it's worth specifying it also in the PR title/description that you added the dd4hep zero material tags. What do you think?

Sure, done!

cmsbuild · 2021-10-31T12:40:06Z

-1

Failed Tests: UnitTests RelVals RelVals-INPUT AddOn
Summary: https://cmssdt.cern.ch/SDT/jenkins-artifacts/pull-request-integration/PR-82afc3/20115/summary.html
COMMIT: 40f2bb4
CMSSW: CMSSW_12_2_X_2021-10-31-0000/slc7_amd64_gcc900
User test area: For local testing, you can use /cvmfs/cms-ci.cern.ch/week1/cms-sw/cmssw/35920/20115/install.sh to create a dev area with all the needed externals and cmssw changes.

The following merge commits were also included on top of IB + this PR after doing git cms-merge-topic:

You can see more details here:
https://cmssdt.cern.ch/SDT/jenkins-artifacts/pull-request-integration/PR-82afc3/20115/git-recent-commits.json
https://cmssdt.cern.ch/SDT/jenkins-artifacts/pull-request-integration/PR-82afc3/20115/git-merge-result

Unit Tests

I found errors in the following unit tests:

---> test TestDQMOnlineClient-ctpps_dqm_sourceclient had ERRORS
---> test TestDQMOnlineClient-ecal_dqm_sourceclient had ERRORS
---> test TestDQMOnlineClient-csc_dqm_sourceclient had ERRORS
---> test TestDQMServicesDemo had ERRORS
and more ...

RelVals

135.4

----- Begin Fatal Exception 31-Oct-2021 12:45:27 CET-----------------------
An exception of category 'EventSetupConflict' occurred while
   [0] Calling beginJob
Exception Message:
two EventSetup Sources want to deliver type="PPSAssociationCuts" label=""
 from record PPSAssociationCutsRcd. The two providers are 
1) type="PPSAssociationCutsESSource" label="ppsAssociationCutsESSource"
2) type="PoolDBESSource" label="GlobalTag"
Please either
   remove one of these Sources
   or find a way of configuring one of them so it does not deliver this data
   or use an es_prefer statement in the configuration to choose one.
----- End Fatal Exception -------------------------------------------------

136.8311

----- Begin Fatal Exception 31-Oct-2021 12:46:02 CET-----------------------
An exception of category 'EventSetupConflict' occurred while
   [0] Calling beginJob
Exception Message:
two EventSetup Sources want to deliver type="PPSAssociationCuts" label=""
 from record PPSAssociationCutsRcd. The two providers are 
1) type="PPSAssociationCutsESSource" label="ppsAssociationCutsESSource"
2) type="PoolDBESSource" label="GlobalTag"
Please either
   remove one of these Sources
   or find a way of configuring one of them so it does not deliver this data
   or use an es_prefer statement in the configuration to choose one.
----- End Fatal Exception -------------------------------------------------

136.7611

----- Begin Fatal Exception 31-Oct-2021 12:46:02 CET-----------------------
An exception of category 'EventSetupConflict' occurred while
   [0] Calling beginJob
Exception Message:
two EventSetup Sources want to deliver type="PPSAssociationCuts" label=""
 from record PPSAssociationCutsRcd. The two providers are 
1) type="PPSAssociationCutsESSource" label="ppsAssociationCutsESSource"
2) type="PoolDBESSource" label="GlobalTag"
Please either
   remove one of these Sources
   or find a way of configuring one of them so it does not deliver this data
   or use an es_prefer statement in the configuration to choose one.
----- End Fatal Exception -------------------------------------------------

Expand to see more relval errors ...

RelVals-INPUT

4.64.6_MinimumBias2010A+MinimumBias2010A+RECOSKIMALCA+HARVESTDR1/step2_MinimumBias2010A+MinimumBias2010A+RECOSKIMALCA+HARVESTDR1.log
136.72411136.72411_RunJetHT2016B_reminiaodUL+RunJetHT2016B_reminiaodUL+REMINIAOD_data2016UL_HIPM+HARVESTDR2_REMINIAOD_data2016UL_HIPM/step2_RunJetHT2016B_reminiaodUL+RunJetHT2016B_reminiaodUL+REMINIAOD_data2016UL_HIPM+HARVESTDR2_REMINIAOD_data2016UL_HIPM.log
136.72412136.72412_RunJetHT2016B_reminiaodUL+RunJetHT2016B_reminiaodUL+REMININANO_data2016UL_HIPM+HARVESTDR2_REMININANO_data2016UL_HIPM/step2_RunJetHT2016B_reminiaodUL+RunJetHT2016B_reminiaodUL+REMININANO_data2016UL_HIPM+HARVESTDR2_REMININANO_data2016UL_HIPM.log

Expand to see more relval errors ...

AddOn Tests

fastsim1

----- Begin Fatal Exception 31-Oct-2021 12:37:23 CET-----------------------
An exception of category 'EventSetupConflict' occurred while
   [0] Calling beginJob
Exception Message:
two EventSetup Sources want to deliver type="PPSAssociationCuts" label=""
 from record PPSAssociationCutsRcd. The two providers are 
1) type="PPSAssociationCutsESSource" label="ppsAssociationCutsESSource"
2) type="PoolDBESSource" label="GlobalTag"
Please either
   remove one of these Sources
   or find a way of configuring one of them so it does not deliver this data
   or use an es_prefer statement in the configuration to choose one.
----- End Fatal Exception -------------------------------------------------

fastsim2

----- Begin Fatal Exception 31-Oct-2021 12:37:24 CET-----------------------
An exception of category 'EventSetupConflict' occurred while
   [0] Calling beginJob
Exception Message:
two EventSetup Sources want to deliver type="PPSAssociationCuts" label=""
 from record PPSAssociationCutsRcd. The two providers are 
1) type="PPSAssociationCutsESSource" label="ppsAssociationCutsESSource"
2) type="PoolDBESSource" label="GlobalTag"
Please either
   remove one of these Sources
   or find a way of configuring one of them so it does not deliver this data
   or use an es_prefer statement in the configuration to choose one.
----- End Fatal Exception -------------------------------------------------

hlt_mc_Fake1

----- Begin Fatal Exception 31-Oct-2021 12:41:33 CET-----------------------
An exception of category 'EventSetupConflict' occurred while
   [0] Calling beginJob
Exception Message:
two EventSetup Sources want to deliver type="PPSAssociationCuts" label=""
 from record PPSAssociationCutsRcd. The two providers are 
1) type="PPSAssociationCutsESSource" label="ppsAssociationCutsESSource"
2) type="PoolDBESSource" label="GlobalTag"
Please either
   remove one of these Sources
   or find a way of configuring one of them so it does not deliver this data
   or use an es_prefer statement in the configuration to choose one.
----- End Fatal Exception -------------------------------------------------

Expand to see more addon errors ...

tvami · 2021-10-31T12:52:00Z

@cmsbuild , please test with #35766

cmsbuild · 2021-10-31T15:21:29Z

-1

Failed Tests: UnitTests
Summary: https://cmssdt.cern.ch/SDT/jenkins-artifacts/pull-request-integration/PR-82afc3/20117/summary.html
COMMIT: 40f2bb4
CMSSW: CMSSW_12_2_X_2021-10-31-0000/slc7_amd64_gcc900
User test area: For local testing, you can use /cvmfs/cms-ci.cern.ch/week1/cms-sw/cmssw/35920/20117/install.sh to create a dev area with all the needed externals and cmssw changes.

The following merge commits were also included on top of IB + this PR after doing git cms-merge-topic:

You can see more details here:
https://cmssdt.cern.ch/SDT/jenkins-artifacts/pull-request-integration/PR-82afc3/20117/git-recent-commits.json
https://cmssdt.cern.ch/SDT/jenkins-artifacts/pull-request-integration/PR-82afc3/20117/git-merge-result

Unit Tests

I found errors in the following unit tests:

---> test TestDQMServicesDemo had ERRORS

Comparison Summary

There are some workflows for which there are errors in the baseline:
135.4 step 1
136.8311 step 2
136.7611 step 2
136.88811 step 2
4.22 step 3
140.56 step 2
4.53 step 3
136.731 step 3
136.793 step 3
136.874 step 3
2018.1 step 1
1001.0 step 2
1000.0 step 2
1306.0 step 3
1330.0 step 3
312.0 step 3
10042.0 step 3
10024.0 step 3
10824.0 step 3
25202.0 step 3
11634.0 step 3
11634.7 step 3
11634.914 step 3
10224.0 step 3
11634.911 step 3
12434.0 step 3
23234.0 step 3
34634.0 step 3
38634.0 step 3
28234.0 step 3
250202.181 step 4
34834.999 step 4
The results for the comparisons for these workflows could be incomplete
This means most likely that the IB is having errors in the relvals.The error does NOT come from this pull request

Summary:

You potentially added 22560 lines to the logs
Reco comparison results: 0 differences found in the comparisons
DQMHistoTests: Total files compared: 9
DQMHistoTests: Total histograms compared: 304467
DQMHistoTests: Total failures: 0
DQMHistoTests: Total nulls: 0
DQMHistoTests: Total successes: 304459
DQMHistoTests: Total skipped: 8
DQMHistoTests: Total Missing objects: 0
DQMHistoSizes: Histogram memory added: 0.0 KiB( 8 files compared)
Checked 118 log files, 16 edm output root files, 9 DQM output files
TriggerResults: no differences found

tvami · 2021-10-31T15:27:12Z

+alca

test results as expected
@francescobrivio already agreed to the GT changes

cmsbuild · 2021-10-31T15:27:35Z

This pull request is fully signed and it will be integrated in one of the next master IBs (but tests are reportedly failing). This pull request will now be reviewed by the release team before it's merged. @perrotta, @dpiparo, @qliphy (and backports should be raised in the release meeting by the corresponding L2)

perrotta · 2021-11-02T11:06:30Z

please test with #35941,#35921
(in order to have some meaningful test and comparison)

cmsbuild · 2021-11-02T15:02:51Z

+1

Summary: https://cmssdt.cern.ch/SDT/jenkins-artifacts/pull-request-integration/PR-82afc3/20181/summary.html
COMMIT: 40f2bb4
CMSSW: CMSSW_12_2_X_2021-11-01-2300/slc7_amd64_gcc900
User test area: For local testing, you can use /cvmfs/cms-ci.cern.ch/week1/cms-sw/cmssw/35920/20181/install.sh to create a dev area with all the needed externals and cmssw changes.

Comparison Summary

Summary:

No significant changes to the logs found
Reco comparison results: 0 differences found in the comparisons
DQMHistoTests: Total files compared: 42
DQMHistoTests: Total histograms compared: 2901890
DQMHistoTests: Total failures: 0
DQMHistoTests: Total nulls: 0
DQMHistoTests: Total successes: 2901868
DQMHistoTests: Total skipped: 22
DQMHistoTests: Total Missing objects: 0
DQMHistoSizes: Histogram memory added: 0.0 KiB( 41 files compared)
Checked 177 log files, 37 edm output root files, 42 DQM output files
TriggerResults: no differences found

perrotta · 2021-11-02T15:09:19Z

+1

The DD4hep Run 3 12_1 reduced material geometry scenarios are only used for calibrations and special studies: therefore, it is normal that there are no differences in test outputs while just adding them to the GT

Include reduced material geometry scenarios in Run-3 MC GTs

40f2bb4

cmsbuild added this to the CMSSW_12_2_X milestone Oct 31, 2021

cmsbuild added alca-pending code-checks-pending orp-pending pending-signatures tests-pending labels Oct 31, 2021

cmsbuild added tests-started and removed tests-pending labels Oct 31, 2021

cmsbuild added code-checks-approved and removed code-checks-pending labels Oct 31, 2021

tvami changed the title ~~Include reduced material geometry scenarios in Run-3 MC GTs~~ Include reduced material DD4HEP geometry scenarios in Run-3 MC GTs Oct 31, 2021

cmsbuild added tests-rejected and removed tests-started labels Oct 31, 2021

cmsbuild added requires-external tests-started and removed tests-rejected labels Oct 31, 2021

cmsbuild added tests-rejected and removed tests-started labels Oct 31, 2021

cmsbuild removed the alca-pending label Oct 31, 2021

cmsbuild added alca-approved fully-signed and removed pending-signatures labels Oct 31, 2021

cmsbuild added tests-started and removed tests-rejected labels Nov 2, 2021

cmsbuild added tests-approved and removed tests-started labels Nov 2, 2021

cmsbuild added orp-approved and removed orp-pending labels Nov 2, 2021

cmsbuild merged commit 4e40cd7 into cms-sw:master Nov 2, 2021

tvami mentioned this pull request Nov 5, 2021

Add reduced material scenarios to Run-3 MC GTs cms-AlCaDB/AlCaTools#47

Closed

mmusich mentioned this pull request Nov 5, 2021

A new ZeroMaterial GeometryFileRcd payload is needed for the Run-3 scenario #31550

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Include reduced material DD4HEP geometry scenarios in Run-3 MC GTs #35920

Include reduced material DD4HEP geometry scenarios in Run-3 MC GTs #35920

tvami commented Oct 31, 2021 •

edited

tvami commented Oct 31, 2021

cmsbuild commented Oct 31, 2021

cmsbuild commented Oct 31, 2021

francescobrivio commented Oct 31, 2021

tvami commented Oct 31, 2021

cmsbuild commented Oct 31, 2021

tvami commented Oct 31, 2021

cmsbuild commented Oct 31, 2021

tvami commented Oct 31, 2021

cmsbuild commented Oct 31, 2021

perrotta commented Nov 2, 2021

cmsbuild commented Nov 2, 2021

perrotta commented Nov 2, 2021

Include reduced material DD4HEP geometry scenarios in Run-3 MC GTs #35920

Include reduced material DD4HEP geometry scenarios in Run-3 MC GTs #35920

Conversation

tvami commented Oct 31, 2021 • edited

PR description:

PR validation:

if this PR is a backport please specify the original PR and why you need to backport that PR:

tvami commented Oct 31, 2021

cmsbuild commented Oct 31, 2021

cmsbuild commented Oct 31, 2021

francescobrivio commented Oct 31, 2021

tvami commented Oct 31, 2021

cmsbuild commented Oct 31, 2021

Unit Tests

RelVals

RelVals-INPUT

AddOn Tests

tvami commented Oct 31, 2021

cmsbuild commented Oct 31, 2021

Unit Tests

Comparison Summary

tvami commented Oct 31, 2021

cmsbuild commented Oct 31, 2021

perrotta commented Nov 2, 2021

cmsbuild commented Nov 2, 2021

Comparison Summary

perrotta commented Nov 2, 2021

tvami commented Oct 31, 2021 •

edited