Update GT for ultra legacy : MC ECAL, GEM eMap, PF calibration #26049

tocheng · 2019-03-03T10:22:59Z

The PR includes update of GTs in autoCond to document the conditons and support the developments for ultra legacy. The updates cover ECAL MC conditions for ultra legacy in 2016 and 2018, PF calibration including eta dependence, new GEM eMap for MC (>=2017).

Ecal condition changes can be referred to

ECAL 2016 : AlCa hn
ECAL 2018 : AlCa hn

PF calibration

PF calibration is to support PR Removing hard-coded eta dependency from PFEnergyCalibration.cc #25883
The change is only a change of dataformat. The formulas are the same (just putting the eta-dependent formulas from CMSSW to the condition in DB)
There will be a new commit or new PR to update the PF calibration used in HLT or the update will be part of PR Removing hard-coded eta dependency from PFEnergyCalibration.cc #25883**

GEM eMap is to support PR #25158

Run-1 MC

update PF calibration

Run-2 MC 2016

New ideal GT is
https://cms-conddb.cern.ch/cmsDbBrowser/list/Prod/gts/105X_mcRun2_design_v2
Diff
https://cms-conddb.cern.ch/cmsDbBrowser/diff/Prod/gts/105X_mcRun2_design_v2/105X_mcRun2_design_v1

update PF calibration

New GT for Stage-1 L1 trigger (2015)
https://cms-conddb.cern.ch/cmsDbBrowser/list/Prod/gts/105X_mcRun2_asymptotic_l1stage1_v1
Diff
https://cms-conddb.cern.ch/cmsDbBrowser/diff/Prod/gts/105X_mcRun2_asymptotic_l1stage1_v1/103X_mcRun2_asymptotic_l1stage1_v1

update PF calibration

New realistic GT and cosmic GT are
https://cms-conddb.cern.ch/cmsDbBrowser/list/Prod/gts/105X_mcRun2_asymptotic_v3
https://cms-conddb.cern.ch/cmsDbBrowser/list/Prod/gts/105X_mcRun2cosmics_startup_deco_v3
Diff
https://cms-conddb.cern.ch/cmsDbBrowser/diff/Prod/gts/105X_mcRun2_asymptotic_v3/105X_mcRun2_asymptotic_v1
https://cms-conddb.cern.ch/cmsDbBrowser/diff/Prod/gts/105X_mcRun2cosmics_startup_deco_v3/105X_mcRun2cosmics_startup_deco_v1

ECAL MC conditions for ultra legacy announced at AlCa-hn
PF calibration

Run-2 MC 2017

add new GEM MC eMap
PF calibration

Run-2 MC 2018

ECAL MC conditions for ultra legacy announced at AlCa-hn
add new GEM MC eMap
PF calibration

Run-2 heavy Ion MC 2018

New GTs are
https://cms-conddb.cern.ch/cmsDbBrowser/list/Prod/gts/105X_upgrade2018_realistic_HI_v3
Diff
https://cms-conddb.cern.ch/cmsDbBrowser/diff/Prod/gts/105X_upgrade2018_realistic_HI_v3/103X_upgrade2018_realistic_HI_v12

add new GEM MC eMap
PF calibration

Run-3/Phase-2 MC

New GTs are
https://cms-conddb.cern.ch/cmsDbBrowser/list/Prod/gts/105X_postLS2_design_v3
https://cms-conddb.cern.ch/cmsDbBrowser/list/Prod/gts/105X_postLS2_realistic_v5
https://cms-conddb.cern.ch/cmsDbBrowser/list/Prod/gts/105X_upgrade2023_realistic_v4
Diff
https://cms-conddb.cern.ch/cmsDbBrowser/diff/Prod/gts/105X_postLS2_design_v3/105X_postLS2_design_v1
https://cms-conddb.cern.ch/cmsDbBrowser/diff/Prod/gts/105X_postLS2_realistic_v5/105X_postLS2_realistic_v3
https://cms-conddb.cern.ch/cmsDbBrowser/diff/Prod/gts/105X_upgrade2023_realistic_v4/105X_upgrade2023_realistic_v2

add new GEM MC eMap
PF calibration

Data

New offline GT is
https://cms-conddb.cern.ch/cmsDbBrowser/list/Prod/gts/105X_dataRun2_v7
Diff
https://cms-conddb.cern.ch/cmsDbBrowser/diff/Prod/gts/105X_dataRun2_v7/105X_dataRun2_v6

PF calibration

New relval offline GT is
https://cms-conddb.cern.ch/cmsDbBrowser/list/Prod/gts/105X_dataRun2_relval_v7
Diff
https://cms-conddb.cern.ch/cmsDbBrowser/diff/Prod/gts/105X_dataRun2_relval_v7/105X_dataRun2_relval_v6

PF calibration

New prompt like GTs are
https://cms-conddb.cern.ch/cmsDbBrowser/list/Prod/gts/105X_dataRun2_PromptLike_v7
https://cms-conddb.cern.ch/cmsDbBrowser/list/Prod/gts/105X_dataRun2_PromptLike_HEfail_v6
Diff
https://cms-conddb.cern.ch/cmsDbBrowser/diff/Prod/gts/105X_dataRun2_PromptLike_v7/105X_dataRun2_PromptLike_v6
https://cms-conddb.cern.ch/cmsDbBrowser/diff/Prod/gts/105X_dataRun2_PromptLike_HEfail_v6/105X_dataRun2_PromptLike_HEfail_v5

PF calibration

New prompt like heavy Ion GT
https://cms-conddb.cern.ch/cmsDbBrowser/list/Prod/gts/105X_dataRun2_PromptLike_HI_v2
Diff
https://cms-conddb.cern.ch/cmsDbBrowser/diff/Prod/gts/105X_dataRun2_PromptLike_HI_v2/103X_dataRun2_PromptLike_HI_v3

Add PPS timing and LHC info
Freeze Hcal quality to be the condition with HE sector failure
new GEM eMap
PF calibration

cmsbuild · 2019-03-03T10:23:28Z

The code-checks are being triggered in jenkins.

cmsbuild · 2019-03-03T10:30:03Z

+code-checks

Logs: https://cmssdt.cern.ch/SDT/code-checks/cms-sw-PR-26049/8597

This PR adds an extra 12KB to repository

cmsbuild · 2019-03-03T10:30:26Z

A new Pull Request was created by @tocheng (Tongguang) for master.

It involves the following packages:

Configuration/AlCa

@cmsbuild, @pohsun, @franzoni, @tocheng can you please review it and eventually sign? Thanks.
@makortel, @Martin-Grunewald, @mmusich, @tocheng this is something you requested to watch as well.
@davidlange6, @slava77, @fabiocos you are the release manager for this.

cms-bot commands are listed here

tocheng · 2019-03-03T10:32:08Z

@cmsbuild please test workflow 1010, 1001.2, 1040, 140.56, 140.57, 150, 158

cmsbuild · 2019-03-03T10:32:34Z

The tests are being triggered in jenkins.
https://cmssdt.cern.ch/jenkins/job/ib-any-integration/33370/console Started: 2019/03/03 13:53

cmsbuild · 2019-03-03T16:37:26Z

+1
Tested at: f689a30
https://cmssdt.cern.ch/SDT/jenkins-artifacts/pull-request-integration/PR-26049/33370/summary.html

cmsbuild · 2019-03-03T16:37:30Z

Comparison job queued.

cmsbuild · 2019-03-03T19:36:27Z

Comparison is ready
https://cmssdt.cern.ch/SDT/jenkins-artifacts/pull-request-integration/PR-26049/33370/summary.html

@slava77 comparisons for the following workflows were not done due to missing matrix map:

/build/cmsbld/jenkins/workspace/compare-root-files-short-matrix/results/JR-comparison/PR-26049/1001.2_RunZeroBias2017F+RunZeroBias2017F+TIER0EXPRUN2+ALCAEXPRUN2+ALCAHARVDSIPIXELCAL
/build/cmsbld/jenkins/workspace/compare-root-files-short-matrix/results/JR-comparison/PR-26049/1010.0_TestEnableEcalHCAL2017B+TestEnableEcalHCAL2017B+TIER0EXPTE+ALCAEXPTE+ALCAHARVDTE
/build/cmsbld/jenkins/workspace/compare-root-files-short-matrix/results/JR-comparison/PR-26049/1040.0_RunZeroBias2017F+RunZeroBias2017F+TIER0RAWSIPIXELCAL+ALCASPLITSIPIXELCAL+ALCAHARVDSIPIXELCAL

Comparison Summary:

No significant changes to the logs found
Reco comparison results: 8633 differences found in the comparisons
DQMHistoTests: Total files compared: 32
DQMHistoTests: Total histograms compared: 3114826
DQMHistoTests: Total failures: 26832
DQMHistoTests: Total nulls: 1
DQMHistoTests: Total successes: 3087796
DQMHistoTests: Total skipped: 197
DQMHistoTests: Total Missing objects: 0
DQMHistoSizes: Histogram memory added: 0.638 KiB( 31 files compared)
DQMHistoSizes: changed ( 25202.0 ): -0.129 KiB SiStrip/MechanicalView
DQMHistoSizes: changed ( 250202.181 ): 0.767 KiB SiStrip/MechanicalView
Checked 133 log files, 14 edm output root files, 32 DQM output files

tocheng · 2019-03-03T20:12:35Z

@fabiocos Sorry for submitting the PR not in time. I am wondering whether the PR can be backported to 10_5_X to enter 10_5_0 to document the ultra legacy condition update for ECAL.
The PR looks fine from point of you,
just the new formatted PF calibration for HLT is not ready,
I pinged @spandeyehep people working on new PF calibration #25883 , but the PF calibration for HLT is managed by another group of people. So I am not sure how to proceed.
We may ignore the PF calibration at HLT for the moment and let the people working on #25883 change GT when needed.

fabiocos · 2019-03-04T09:34:31Z

@tocheng if this PR is ok for you we can of course have it in 10_6_0_pre1 and test it there. But if this is needed for the production with 10_5_X we need to understand with @srimanob and @franzoni how to manage its validation before adding it to 10_5_0, as this missed 10_5_0_pre2.

srimanob · 2019-03-04T09:47:42Z

@fabiocos
This we need to discuss on how to process with 2016 and 2018. The last update was targeted for 2017.
The original plan is to keep 10_5 alive to produce small sets of MC for PF Calibration for all 3 years. So we can expect the change to continue coming, the is what @tocheng is putting is here. We may end up like
10_5_0: for 2017 PF Calibration
10_5_3: for 2018/2016 PF Calibration
Data conditions will come later too.

@tocheng
Could you please confirm , for 2017, both changes (PFCalibration, GEM) will have no effect in PFCalibration campaign. For PFCalibration, I see that it's only format change to support eta-dependent. Why do we need this in 2017 10_5? It should not harm, anyway.

tocheng · 2019-03-04T09:54:17Z

@tocheng
Could you please confirm , for 2017, both changes (PFCalibration, GEM) will have no effect in PFCalibration campaign. For PFCalibration, I see that it's only format change to support eta-dependent. Why do we need this in 2017 10_5? It should not harm, anyway.

GEM eMap doesn't hurt and has no impact. Including it is just to keep consistency with data where the new GEM eMap is already in 10_5 release.

For PF calibration, there is no impact either in 10_5, either. There may be some misunderstanding between me and developers of new PF calibration #25883 @spandeyehep .
If #25883 will only enter 10_6 release, there is no need to add new PF calibration.
I keep the change of PF calibration into a single committee, it is very easy to revert.

spandeyehep · 2019-03-04T11:11:18Z

Hi all,

adding @bkansal & @seemasharmafnal into the loop.
The new tag will have no effect in the RECO workflow from the offline PFHadronCalibration point of view.

We wanted to have the new payload in the GT so that remaining tests and validation of code changes can be done as soon as possible for #25883 . We are also in contact with HLT group and they are working in the direction of payload creation consistent with the PR #25883. @milosdjordjevic

More details can be found out in the comments section of #25883 (comment)

tocheng · 2019-03-04T11:15:14Z

Hi all,

adding @bkansal & @seemasharmafnal into the loop.
The new tag will have no effect in the RECO workflow from the offline PFHadronCalibration point of view.

We wanted to have the new payload in the GT so that remaining tests and validation of code changes can be done as soon as possible for #25883 . We are also in contact with HLT group and they are working in the direction of payload creation consistent with the PR #25883. @milosdjordjevic

More details can be found out in the comments section of #25883 (comment)

Hello @spandeyehep , the question is whether you want your PR for new PF calibration dataformat to enter CMSSW 10_5 release. If not, there is no need for new PF calibration entering 10_5_X GTs although no effect.

fabiocos · 2019-03-07T08:20:03Z

@tocheng the failure is due to a problem independent on this PR that is causing a massive failure of the IB. We are going to fix it for next IB, please just wait for that

fabiocos · 2019-03-07T21:58:06Z

please test workflow 1010, 1001.2, 1040, 140.55, 140.56, 140.57, 150, 158

cmsbuild · 2019-03-07T21:58:34Z

The tests are being triggered in jenkins.
https://cmssdt.cern.ch/jenkins/job/ib-any-integration/33470/console Started: 2019/03/07 23:00

cmsbuild · 2019-03-08T01:13:55Z

+1
Tested at: 45053b8
https://cmssdt.cern.ch/SDT/jenkins-artifacts/pull-request-integration/PR-26049/33470/summary.html

cmsbuild · 2019-03-08T01:14:00Z

Comparison job queued.

cmsbuild · 2019-03-08T04:36:25Z

Comparison is ready
https://cmssdt.cern.ch/SDT/jenkins-artifacts/pull-request-integration/PR-26049/33470/summary.html

@slava77 comparisons for the following workflows were not done due to missing matrix map:

/build/cmsbld/jenkins/workspace/compare-root-files-short-matrix/results/JR-comparison/PR-26049/1001.2_RunZeroBias2017F+RunZeroBias2017F+TIER0EXPRUN2+ALCAEXPRUN2+ALCAHARVDSIPIXELCAL
/build/cmsbld/jenkins/workspace/compare-root-files-short-matrix/results/JR-comparison/PR-26049/1010.0_TestEnableEcalHCAL2017B+TestEnableEcalHCAL2017B+TIER0EXPTE+ALCAEXPTE+ALCAHARVDTE
/build/cmsbld/jenkins/workspace/compare-root-files-short-matrix/results/JR-comparison/PR-26049/1040.0_RunZeroBias2017F+RunZeroBias2017F+TIER0RAWSIPIXELCAL+ALCASPLITSIPIXELCAL+ALCAHARVDSIPIXELCAL

Comparison Summary:

No significant changes to the logs found
Reco comparison results: 8633 differences found in the comparisons
DQMHistoTests: Total files compared: 32
DQMHistoTests: Total histograms compared: 3114826
DQMHistoTests: Total failures: 28629
DQMHistoTests: Total nulls: 4
DQMHistoTests: Total successes: 3085996
DQMHistoTests: Total skipped: 197
DQMHistoTests: Total Missing objects: 0
DQMHistoSizes: Histogram memory added: 3.698 KiB( 31 files compared)
DQMHistoSizes: changed ( 25202.0 ): -0.129 KiB SiStrip/MechanicalView
DQMHistoSizes: changed ( 250202.181 ): 0.767 KiB SiStrip/MechanicalView
DQMHistoSizes: changed ( 11624.0 ): 2.903 KiB Hcal/DigiRunHarvesting
DQMHistoSizes: changed ( 11624.0 ): 0.125 KiB Hcal/DigiTask
DQMHistoSizes: changed ( 11624.0 ): 0.016 KiB Hcal/EventInfo
DQMHistoSizes: changed ( 11624.0 ): 0.016 KiB Hcal/RecoRunHarvesting
Checked 133 log files, 14 edm output root files, 32 DQM output files

tocheng · 2019-03-08T08:38:57Z

+1

cmsbuild · 2019-03-08T08:39:25Z

This pull request is fully signed and it will be integrated in one of the next master IBs (tests are also fine). This pull request will now be reviewed by the release team before it's merged. @davidlange6, @slava77, @smuzaffar, @fabiocos (and backports should be raised in the release meeting by the corresponding L2)

fabiocos · 2019-03-08T09:44:54Z

+1

as discussed with @tocheng further fixes in support of #25883 will move to a new PR, as I understand this problem is not affecting anything else in the IB we agreed to move forward to close the pre-release,
and support the backport to 10_5_X of the subset used for calibration studies

cmsbuild added this to the CMSSW_10_6_X milestone Mar 3, 2019

cmsbuild added alca-pending code-checks-pending comparison-pending orp-pending pending-signatures tests-pending labels Mar 3, 2019

cmsbuild added code-checks-approved and removed code-checks-pending labels Mar 3, 2019

cmsbuild added tests-started and removed tests-pending labels Mar 3, 2019

cmsbuild added tests-approved and removed tests-started labels Mar 3, 2019

cmsbuild added comparison-available and removed comparison-pending labels Mar 3, 2019

tocheng mentioned this pull request Mar 3, 2019

GEM unpacker for VFat V3 #25158

Merged

cmsbuild added comparison-notrun tests-rejected and removed tests-started labels Mar 7, 2019

cmsbuild added comparison-pending tests-pending and removed comparison-notrun tests-rejected labels Mar 7, 2019

cmsbuild added tests-started and removed tests-pending labels Mar 7, 2019

cmsbuild added tests-approved and removed tests-started labels Mar 8, 2019

cmsbuild added comparison-available and removed comparison-pending labels Mar 8, 2019

cmsbuild added alca-approved fully-signed and removed alca-pending pending-signatures labels Mar 8, 2019

cmsbuild added orp-approved and removed orp-pending labels Mar 8, 2019

cmsbuild merged commit 097a76f into cms-sw:master Mar 8, 2019

abdoulline mentioned this pull request Mar 11, 2019

Switching HCAL Run3 MC to DB conditions #26139

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Update GT for ultra legacy : MC ECAL, GEM eMap, PF calibration #26049

Update GT for ultra legacy : MC ECAL, GEM eMap, PF calibration #26049

tocheng commented Mar 3, 2019 •

edited

cmsbuild commented Mar 3, 2019

cmsbuild commented Mar 3, 2019

cmsbuild commented Mar 3, 2019

tocheng commented Mar 3, 2019

cmsbuild commented Mar 3, 2019 •

edited

cmsbuild commented Mar 3, 2019

cmsbuild commented Mar 3, 2019

cmsbuild commented Mar 3, 2019

tocheng commented Mar 3, 2019

fabiocos commented Mar 4, 2019

srimanob commented Mar 4, 2019 •

edited

tocheng commented Mar 4, 2019

spandeyehep commented Mar 4, 2019

tocheng commented Mar 4, 2019

fabiocos commented Mar 7, 2019

fabiocos commented Mar 7, 2019

cmsbuild commented Mar 7, 2019 •

edited

cmsbuild commented Mar 8, 2019

cmsbuild commented Mar 8, 2019

cmsbuild commented Mar 8, 2019

tocheng commented Mar 8, 2019

cmsbuild commented Mar 8, 2019

fabiocos commented Mar 8, 2019

Update GT for ultra legacy : MC ECAL, GEM eMap, PF calibration #26049

Update GT for ultra legacy : MC ECAL, GEM eMap, PF calibration #26049

Conversation

tocheng commented Mar 3, 2019 • edited

The PR includes update of GTs in autoCond to document the conditons and support the developments for ultra legacy. The updates cover ECAL MC conditions for ultra legacy in 2016 and 2018, PF calibration including eta dependence, new GEM eMap for MC (>=2017).

Ecal condition changes can be referred to

PF calibration

GEM eMap is to support PR #25158

Run-1 MC

Run-2 MC 2016

Run-2 MC 2017

Run-2 MC 2018

Run-2 heavy Ion MC 2018

Run-3/Phase-2 MC

Data

cmsbuild commented Mar 3, 2019

cmsbuild commented Mar 3, 2019

cmsbuild commented Mar 3, 2019

tocheng commented Mar 3, 2019

cmsbuild commented Mar 3, 2019 • edited

cmsbuild commented Mar 3, 2019

cmsbuild commented Mar 3, 2019

cmsbuild commented Mar 3, 2019

tocheng commented Mar 3, 2019

fabiocos commented Mar 4, 2019

srimanob commented Mar 4, 2019 • edited

tocheng commented Mar 4, 2019

spandeyehep commented Mar 4, 2019

tocheng commented Mar 4, 2019

fabiocos commented Mar 7, 2019

fabiocos commented Mar 7, 2019

cmsbuild commented Mar 7, 2019 • edited

cmsbuild commented Mar 8, 2019

cmsbuild commented Mar 8, 2019

cmsbuild commented Mar 8, 2019

tocheng commented Mar 8, 2019

cmsbuild commented Mar 8, 2019

fabiocos commented Mar 8, 2019

tocheng commented Mar 3, 2019 •

edited

cmsbuild commented Mar 3, 2019 •

edited

srimanob commented Mar 4, 2019 •

edited

cmsbuild commented Mar 7, 2019 •

edited