Adding HGCal configuration in electron MC Validations #20934

archiron · 2017-10-17T08:36:39Z

A few modifications have been made for HGCal Phase2 integration into electron MC Validations.

2 small modifcations into Validation/RecoEgamma/plugins/ ('ElectronMcFakeValidator.cc & ElectronMcSignalValidator.cc) related to the scale limits of one histo.
there is modifications into the _cfi files to take HGcal Pahse2 modifications into account (ElectronMcFakeValidator_gedGsfElectrons_cfi.py, ElectronMcSignalValidatorPt1000_gedGsfElectrons_cfi.py & ElectronMcSignalValidator_gedGsfElectrons_cfi.py).

The config files (_cfg.py files) that we use have been rewritten to take modifications into account :
ElectronMcFakePostValidation_cfg.py, ElectronMcFakeValidation_gedGsfElectrons_cfg.py,
ElectronMcSignalPostValidationMiniAOD_cfg.py, ElectronMcSignalValidationMiniAOD_cfg.py,
ElectronMcSignalPostValidationPt1000_cfg.py, ElectronMcSignalValidationPt1000_gedGsfElectrons_cfg.py,
ElectronMcSignalPostValidation_cfg.py, ElectronMcSignalValidation_gedGsfElectrons_cfg.py

We have added 3 files for run simplification :
relval_hgcal.tcsh -> used to launch local computations & web page histos
OvalFile.HGCAL -> used to create web pages histos
electronValidationCheck_Env.py -> used for RECO/HGCal initialization. It is a new file wich is called by our config (_cfg.py) files when we create histograms web pages.

Tests have been made with runTheMatrix without any encountered pbms.

@beaudett @fcouderc @rovere

#CMSSW_9_3_X

cmsbuild · 2017-10-17T08:36:59Z

The code-checks are being triggered in jenkins.

cmsbuild · 2017-10-17T08:39:53Z

-code-checks

Logs: https://cmssdt.cern.ch/SDT/code-checks/PR-20934/1481

Code check has found code style and quality issues which could be resolved by applying a patch in https://cmssdt.cern.ch/SDT/code-checks/PR-20934/1481/git-diff.patch
e.g. curl https://cmssdt.cern.ch/SDT/code-checks/PR-20934/1481/git-diff.patch | patch -p1

You can run scram build code-checks to apply code checks directly

cmsbuild · 2017-10-17T09:55:47Z

The code-checks are being triggered in jenkins.

cmsbuild · 2017-10-17T09:58:45Z

+code-checks

Logs: https://cmssdt.cern.ch/SDT/code-checks/PR-20934/1485

cmsbuild · 2017-10-17T09:59:07Z

A new Pull Request was created by @archiron (Chiron) for master.

It involves the following packages:

Validation/RecoEgamma

@kmaeshima, @cmsbuild, @vanbesien, @vazzolini, @dmitrijus can you please review it and eventually sign? Thanks.
@davidlange6, @slava77 you are the release manager for this.

cms-bot commands are listed here

rovere · 2017-10-23T07:46:19Z

Hi all,
do we really need 6+ days to even trigger the tests? It seems too much to me.
@archiron Quickly looking into the code it seems that you switched the monitoring of gedElectrons with electronsFromMultiCl, which is correct since it's the only working collection we have. I was wondering if it could be better to add a cloned version of the usual validation sequence instead of replacing the inputTag, so that the moment we will have working gedGsfElectrons no changes will be needed. Let me know what you think.

beaudett · 2017-10-23T09:13:06Z

Hi Marco,
may I ask you more information on the change you propose ? Are you saying that within the if (phase2) switch, the validation sequence is cloned or are you proposing to introduce a new specific config file with the Phase2 validation sequence ?
Probably, I am missing something, but as far I understand, when we switch to default gedGsfElectrons for Phase 2, we will have to change an InputTag somewhere, no ? Does your proposal go in the direction of automatically catching the change ? How ? Thanks.

rovere · 2017-10-23T09:34:34Z

Validation/RecoEgamma/python/ElectronMcFakeValidator_gedGsfElectrons_cfi.py

-
+from Configuration.Eras.Modifier_phase2_hgcal_cff import phase2_hgcal
+phase2_hgcal.toModify(
+    electronMcFakeValidator,


@beaudett
here we switch from gedGsfElectrons to ecalDrivenGsfElectronsFromMultiCl et all for all related collection.
What I was proposing was to clone the original analyzer and run them both at the same time so that gedGsfElectrons are left as they are (broken for the time being) and FromMultiCl added. The moment we fix the gedGsfElectrons there's nothing else/more to touch here. Would that make sense?

I see. It would be indeed convenient. But we will have duplicate set of histograms then, one broken (with empty histograms) and one filled. Aren't the broken ones take disk-space for nothing ? If you tell me that it is not an issue, then I buy your solution !

@beaudett they will take up both memory and disk space: the latter is very limited, the formes is somehow quite expensive these days. Just to have a rough understanding, of how much histograms are we talking about? I'd then put the histograms in 2 separate folders. That could also be useful to have a comparison later on between the 2 collections.

Hi Marco,

we are speaking of > 1K histograms, with a good fraction of them being 2D.
ElectronMcFakeValidator : 360 histos
ElectronMcSignalValidator : 494 histos
ElectronMcSignalValidatorPt1000 : 494 histos
ElectronMcSignalValidatorMiniAOD : 41 histos

Arnaud reminded me that we had been asked in the past to reduce the number of histograms, and we had to struggle a bit when we introduced the Pt1000 duplicated directory (which is not empty) and for which the ranges of the histograms are adapted.
Therefore, and after some thinking, we are a bit reluctant to clone the sequence and to put a new directory in place. We are aware that it implies that we manually switch back to the regular collection once it is in place.
We hope it is agreeable to you.

Florian & Arnaud

rovere · 2017-10-23T12:21:05Z

In that case, I totally agree: I thought we were talking about much fewer histograms. Ciao, -- Marco.

…

___________ Marco Rovere Marco.Rovere@cern.ch CERN EP-CMG | room 40 3-A28 | tel +41227671209 (71209)

On Mon, Oct 23, 2017 at 4:40 AM, Florian Beaudette ***@***.*** > wrote: ***@***.**** commented on this pull request. ------------------------------ In Validation/RecoEgamma/python/ElectronMcFakeValidator_ gedGsfElectrons_cfi.py <#20934 (comment)>: > @@ -57,7 +57,18 @@ histosCfg = cms.PSet(electronMcFakeHistosCfg) ) - - - - +from Configuration.Eras.Modifier_phase2_hgcal_cff import phase2_hgcal +phase2_hgcal.toModify( + electronMcFakeValidator, Hi Marco, we are speaking of > 1K histograms, with a good fraction of them being 2D. ElectronMcFakeValidator : 360 histos ElectronMcSignalValidator : 494 histos ElectronMcSignalValidatorPt1000 : 494 histos ElectronMcSignalValidatorMiniAOD : 41 histos Arnaud reminded me that we had been asked in the past to reduce the number of histograms, and we had to struggle a bit when we introduced the Pt1000 duplicated directory (which is not empty) and for which the ranges of the histograms are adapted. Therefore, and after some thinking, we are a bit reluctant to clone the sequence and to put a new directory in place. We are aware that it implies that we manually switch back to the regular collection once it is in place. We hope it is agreeable to you. Florian & Arnaud — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#20934 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/ABHaR7cbs4RWR6QCRPpJ0wDdRL4Y-oVCks5svHs8gaJpZM4P7xe6> .

beaudett · 2017-10-25T12:31:08Z

Hello
have the test been triggered in the end ?
Thanks F.

rovere · 2017-10-25T13:37:45Z

I'm begging for this but, apparently, I have no luck.

beaudett · 2017-10-25T13:39:48Z

I now understand why CMS is so well ranked by github on the amount of discussions :-)

slava77 · 2017-10-25T13:44:56Z

@cmsbuild please test

cmsbuild · 2017-10-25T13:45:17Z

The tests are being triggered in jenkins.
https://cmssdt.cern.ch/jenkins/job/ib-any-integration/23981/console Started: 2017/10/25 18:07

cmsbuild · 2017-10-25T20:04:12Z

New categories assigned: upgrade

@kpedro88 you have been requested to review this Pull request/Issue and eventually sign? Thanks

kpedro88 · 2017-10-25T20:09:50Z

@slava77 I also find this quite puzzling. Modifications of the Validation subsystem should not be able to affect reco. In the 2023D19 workflow, the timing-related changes are enabled - maybe this problem is related to #20621? @lgray @bendavid

slava77 · 2017-10-26T04:31:59Z

On 10/25/17 10:09 PM, Kevin Pedro wrote: @slava77 <https://github.com/slava77> I also find this quite puzzling. Modifications of the Validation subsystem should not be able to affect reco. In the 2023D19 workflow, the timing-related changes are enabled - maybe this problem is related to #20621 <#20621>? @lgray <https://github.com/lgray> @bendavid <https://github.com/bendavid>

It looks like the same is showing up in other PRs e.g. #20682 (comment) Is this workflow hitting some uninitialized values or did it get some extra randomization?

…

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#20934 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AEdcbq1NHSVL_LqZApSIwb7TduZWEnO3ks5sv5WPgaJpZM4P7xe6>.

kpedro88 · 2017-10-26T14:26:18Z

@slava77 I'll run valgrind...

davidlange6 · 2017-10-26T14:33:14Z

see first the GitHub thread https://hypernews.cern.ch/HyperNews/CMS/get/pixelOfflineSW/1335.html #20397 on these histograms

…

On Oct 26, 2017, at 4:26 PM, Kevin Pedro ***@***.***> wrote: @slava77 I'll run valgrind... — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub, or mute the thread.

kpedro88 · 2017-10-26T14:36:55Z

@davidlange6 I had noticed the irreproducibility in Phase1 pixel validation, but I'm not sure if it is related to what was observed here (changes in handful of reco validation plots only in 20434.0, 2023D19 workflow).

slava77 · 2017-10-26T14:49:16Z

On 10/26/17 4:36 PM, Kevin Pedro wrote: @davidlange6 <https://github.com/davidlange6> I had noticed the irreproducibility in Phase1 pixel validation, but I'm not sure if it is related to what was observed here (changes in handful of reco validation plots only in 20434.0, 2023D19 workflow).

I don't see how phase-1 pixel validation can change RPC rechit content in reco.

davidlange6 · 2017-10-26T14:52:22Z

indeed - sorry for the noise.. this one is different.

…

On Oct 26, 2017, at 4:36 PM, Kevin Pedro ***@***.***> wrote: @davidlange6 I had noticed the irreproducibility in Phase1 pixel validation, but I'm not sure if it is related to what was observed here (changes in handful of reco validation plots only in 20434.0, 2023D19 workflow). — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub, or mute the thread.

kpedro88 · 2017-10-26T21:43:42Z

@slava77 @davidlange6 see #21036

dmitrijus · 2017-11-02T08:32:41Z

+1

kpedro88 · 2017-11-02T14:24:48Z

+1

cmsbuild · 2017-11-02T14:25:08Z

This pull request is fully signed and it will be integrated in one of the next master IBs (tests are also fine). This pull request will now be reviewed by the release team before it's merged. @davidlange6, @slava77, @smuzaffar (and backports should be raised in the release meeting by the corresponding L2)

davidlange6 · 2017-11-03T07:28:42Z

+1

Adding HGCal configuration in electron MC Validations

9dc10c3

cmsbuild added this to the CMSSW_9_4_X milestone Oct 17, 2017

cmsbuild added code-checks-pending comparison-pending dqm-pending orp-pending pending-signatures tests-pending labels Oct 17, 2017

cmsbuild added code-checks-rejected and removed code-checks-pending labels Oct 17, 2017

code-checks correction & forgotten line (cfi)

579934c

cmsbuild added code-checks-pending and removed code-checks-rejected labels Oct 17, 2017

cmsbuild added code-checks-approved and removed code-checks-pending labels Oct 17, 2017

rovere reviewed Oct 23, 2017

View reviewed changes

cmsbuild removed the tests-pending label Oct 25, 2017

cmsbuild added the upgrade-pending label Oct 25, 2017

smuzaffar modified the milestones: CMSSW_9_4_X, CMSSW_10_0_X Oct 26, 2017

cmsbuild modified the milestones: CMSSW_10_0_X, CMSSW_9_4_X Oct 26, 2017

kpedro88 mentioned this pull request Oct 26, 2017

Uninitialized memory in RPC simulation #21036

Closed

cmsbuild added dqm-approved and removed dqm-pending labels Nov 2, 2017

cmsbuild added fully-signed upgrade-approved and removed pending-signatures upgrade-pending labels Nov 2, 2017

cmsbuild added orp-approved and removed orp-pending labels Nov 3, 2017

cmsbuild merged commit 4b1d1d6 into cms-sw:master Nov 3, 2017

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Adding HGCal configuration in electron MC Validations #20934

Adding HGCal configuration in electron MC Validations #20934

archiron commented Oct 17, 2017

cmsbuild commented Oct 17, 2017

cmsbuild commented Oct 17, 2017

cmsbuild commented Oct 17, 2017

cmsbuild commented Oct 17, 2017

cmsbuild commented Oct 17, 2017

rovere commented Oct 23, 2017

beaudett commented Oct 23, 2017

rovere Oct 23, 2017

beaudett Oct 23, 2017

rovere Oct 23, 2017

beaudett Oct 23, 2017

rovere commented Oct 23, 2017 via email

beaudett commented Oct 25, 2017

rovere commented Oct 25, 2017

beaudett commented Oct 25, 2017

slava77 commented Oct 25, 2017

cmsbuild commented Oct 25, 2017 •

edited

cmsbuild commented Oct 25, 2017

kpedro88 commented Oct 25, 2017

slava77 commented Oct 26, 2017 via email

kpedro88 commented Oct 26, 2017

davidlange6 commented Oct 26, 2017 via email

kpedro88 commented Oct 26, 2017

slava77 commented Oct 26, 2017 via email

davidlange6 commented Oct 26, 2017 via email

kpedro88 commented Oct 26, 2017

dmitrijus commented Nov 2, 2017

kpedro88 commented Nov 2, 2017

cmsbuild commented Nov 2, 2017

davidlange6 commented Nov 3, 2017

Adding HGCal configuration in electron MC Validations #20934

Adding HGCal configuration in electron MC Validations #20934

Conversation

archiron commented Oct 17, 2017

cmsbuild commented Oct 17, 2017

cmsbuild commented Oct 17, 2017

cmsbuild commented Oct 17, 2017

cmsbuild commented Oct 17, 2017

cmsbuild commented Oct 17, 2017

rovere commented Oct 23, 2017

beaudett commented Oct 23, 2017

rovere Oct 23, 2017

Choose a reason for hiding this comment

beaudett Oct 23, 2017

Choose a reason for hiding this comment

rovere Oct 23, 2017

Choose a reason for hiding this comment

beaudett Oct 23, 2017

Choose a reason for hiding this comment

rovere commented Oct 23, 2017 via email

beaudett commented Oct 25, 2017

rovere commented Oct 25, 2017

beaudett commented Oct 25, 2017

slava77 commented Oct 25, 2017

cmsbuild commented Oct 25, 2017 • edited

cmsbuild commented Oct 25, 2017

kpedro88 commented Oct 25, 2017

slava77 commented Oct 26, 2017 via email

kpedro88 commented Oct 26, 2017

davidlange6 commented Oct 26, 2017 via email

kpedro88 commented Oct 26, 2017

slava77 commented Oct 26, 2017 via email

davidlange6 commented Oct 26, 2017 via email

kpedro88 commented Oct 26, 2017

dmitrijus commented Nov 2, 2017

kpedro88 commented Nov 2, 2017

cmsbuild commented Nov 2, 2017

davidlange6 commented Nov 3, 2017

cmsbuild commented Oct 25, 2017 •

edited