Replace fastSim.isChosen() with toModify/toReplaceWith/makeProcessModifier #21389

makortel · 2017-11-20T16:02:47Z

This PR is an attempt to get rid of the (disallowed) Modifier.isChosen() for FastSim customizations. Mostly it was a matter of using toModify() or toReplaceWith(), but the "digi aliases" required ProcessModifier plus two hacks to be able to track whether the job is classical mixing, premixing step1, or premixing step2 (from "which _cff files have been loaded").

I also catched two typos in the FastSim customization in Validation/RecoEgamma/python/photonPostProcessor_cff.py.

Tested in 10_0_0_pre1, no changes expected (except possibly from the photonPostProcessor_cff.py fix).

@Dr15Jones

This is still ugly, but hopefully bit less than before.

cmsbuild · 2017-11-20T16:03:12Z

The code-checks are being triggered in jenkins.

cmsbuild · 2017-11-20T16:11:12Z

+code-checks

Logs: https://cmssdt.cern.ch/SDT/code-checks/PR-21389/2092

cmsbuild · 2017-11-20T16:11:41Z

A new Pull Request was created by @makortel (Matti Kortelainen) for master.

It involves the following packages:

Configuration/EventContent
Configuration/StandardSequences
FastSimulation/Configuration
HLTriggerOffline/Btag
HLTriggerOffline/Common
RecoTracker/FinalTrackSelectors
SimCalorimetry/Configuration
SimGeneral/DataMixingModule
SimGeneral/MixingModule
SimGeneral/TrackingAnalysis
SimMuon/CSCDigitizer
SimMuon/DTDigitizer
SimMuon/MCTruth
SimTracker/TrackAssociatorProducers
Validation/Configuration
Validation/DTRecHits
Validation/EcalDigis
Validation/EcalHits
Validation/EcalRecHits
Validation/HcalDigis
Validation/HcalRecHits
Validation/MuonCSCDigis
Validation/MuonDTDigis
Validation/MuonHits
Validation/MuonIdentification
Validation/MuonRPCDigis
Validation/RPCRecHits
Validation/RecoEgamma
Validation/RecoMuon
Validation/RecoTrack

@perrotta, @civanch, @vazzolini, @lveldere, @kmaeshima, @ssekmen, @mdhildreth, @dmitrijus, @cmsbuild, @franzoni, @jfernan2, @slava77, @vanbesien, @davidlange6 can you please review it and eventually sign? Thanks.
@ghellwig, @felicepantaleo, @abbiendi, @argiro, @Martin-Grunewald, @imarches, @threus, @calderona, @acaudron, @sdevissc, @jhgoh, @HuguesBrun, @ferencek, @trocino, @rociovilar, @GiacomoSguazzoni, @rovere, @VinInn, @ptcox, @wmtford, @mschrode, @ebrondol, @mtosi, @dgulhan, @swertz, @battibass, @JyothsnaKomaragiri, @mverzett, @gpetruc, @matt-komm, @pvmulder this is something you requested to watch as well.
@davidlange6, @slava77 you are the release manager for this.

cms-bot commands are listed here

makortel · 2017-11-20T16:13:48Z

@cmsbuild, please test

cmsbuild · 2017-11-20T16:14:09Z

The tests are being triggered in jenkins.
https://cmssdt.cern.ch/jenkins/job/ib-any-integration/24560/console Started: 2017/11/20 18:35

cmsbuild · 2017-11-20T20:05:51Z

+1
Tested at: e8ed1aa
https://cmssdt.cern.ch/SDT/jenkins-artifacts/pull-request-integration/PR-21389/24560/summary.html

The following merge commits were also included on top of IB + this PR after doing git cms-merge-topic:
71a61e3
You can see more details here:
https://cmssdt.cern.ch/SDT/jenkins-artifacts/pull-request-integration/PR-21389/24560/git-log-recent-commits
https://cmssdt.cern.ch/SDT/jenkins-artifacts/pull-request-integration/PR-21389/24560/git-merge-result

cmsbuild · 2017-11-20T20:05:54Z

Comparison job queued.

cmsbuild · 2017-11-20T21:46:08Z

Comparison is ready
https://cmssdt.cern.ch/SDT/jenkins-artifacts/pull-request-integration/PR-21389/24560/summary.html

Comparison Summary:

No significant changes to the logs found
Reco comparison results: 0 differences found in the comparisons
DQMHistoTests: Total files compared: 27
DQMHistoTests: Total histograms compared: 2833444
DQMHistoTests: Total failures: 1
DQMHistoTests: Total nulls: 0
DQMHistoTests: Total successes: 2833265
DQMHistoTests: Total skipped: 178
DQMHistoTests: Total Missing objects: 0
DQMHistoSizes: Histogram memory added: 1.51000000004 KiB( 23 files compared)
Checked 111 log files, 8 edm output root files, 27 DQM output files

perrotta · 2017-11-24T15:55:58Z

Validation/EcalDigis/python/ecalSelectiveReadoutValidation_cfi.py

-    ecalSelectiveReadoutValidation.EeSimHitCollection = cms.InputTag("famosSimHits","EcalHitsEE")
+fastSim.toModify(ecalSelectiveReadoutValidation,
+    EbSimHitCollection = "famosSimHits:EcalHitsEB",
+    EeSimHitCollection = "famosSimHits;EcalHitsEE"


Typo: ";" -> ":"

cmsbuild · 2017-11-28T19:15:05Z

-1

Tested at: da08427

The following merge commits were also included on top of IB + this PR after doing git cms-merge-topic:
f493624
You can see more details here:
https://cmssdt.cern.ch/SDT/jenkins-artifacts/pull-request-integration/PR-21389/24727/git-log-recent-commits
https://cmssdt.cern.ch/SDT/jenkins-artifacts/pull-request-integration/PR-21389/24727/git-merge-result

You can see the results of the tests here:
https://cmssdt.cern.ch/SDT/jenkins-artifacts/pull-request-integration/PR-21389/24727/summary.html

I found follow errors while testing this PR

Failed tests: Build

Build:

I found an error when building:

trackingParticleNumberOfLayersProducer
--- Registered EDM Plugin: SimGeneralTrackingAnalysisPlugins
--- Registered EDM Plugin: SimGeneralMixingModulePlugins
@@@@ Running edmWriteConfigs for SimTrackerTrackAssociation_plugins
>> Building big plugin tmp/slc6_amd64_gcc630/src/BigProducts/Simulation/pluginSimulation.so
c++: error: external/slc6_amd64_gcc630/objs-full/SimG4CoreCustomPhysics.obj: No such file or directory
gmake: *** [tmp/slc6_amd64_gcc630/src/BigProducts/Simulation/pluginSimulation.so] Error 1
>> Subsystem Configuration built
>> Subsystem RecoBTag built
>> Subsystem ElectroWeakAnalysis built
>> Leaving Package Validation/HcalDigis

The following merge commits were also included on top of IB + this PR after doing git cms-merge-topic:
f493624
You can see more details here:
https://cmssdt.cern.ch/SDT/jenkins-artifacts/pull-request-integration/PR-21389/24727/git-log-recent-commits
https://cmssdt.cern.ch/SDT/jenkins-artifacts/pull-request-integration/PR-21389/24727/git-merge-result

cmsbuild · 2017-11-28T19:15:08Z

Comparison not run due to Build errors (RelVals and Igprof tests were also skipped)

perrotta · 2017-11-29T06:10:43Z

please test
(Last CMSSW_10_0_X_2017-11-28-2300 IB should be fixed now)

cmsbuild · 2017-11-29T06:11:05Z

The tests are being triggered in jenkins.
https://cmssdt.cern.ch/jenkins/job/ib-any-integration/24742/console Started: 2017/11/29 13:07

cmsbuild · 2017-11-29T14:33:43Z

+1
Tested at: da08427
https://cmssdt.cern.ch/SDT/jenkins-artifacts/pull-request-integration/PR-21389/24742/summary.html

The following merge commits were also included on top of IB + this PR after doing git cms-merge-topic:
bd23f8c
You can see more details here:
https://cmssdt.cern.ch/SDT/jenkins-artifacts/pull-request-integration/PR-21389/24742/git-log-recent-commits
https://cmssdt.cern.ch/SDT/jenkins-artifacts/pull-request-integration/PR-21389/24742/git-merge-result

cmsbuild · 2017-11-29T14:33:45Z

Comparison job queued.

cmsbuild · 2017-11-29T17:26:29Z

Comparison is ready
https://cmssdt.cern.ch/SDT/jenkins-artifacts/pull-request-integration/PR-21389/24742/summary.html

There are some workflows for which there are errors in the baseline:
10824.0 step 5
The results for the comparisons for these workflows could be incomplete
This means most likely that the IB is having errors in the relvals.The error does NOT come from this pull request

Comparison Summary:

No significant changes to the logs found
Reco comparison results: 0 differences found in the comparisons
DQMHistoTests: Total files compared: 26
DQMHistoTests: Total histograms compared: 2605338
DQMHistoTests: Total failures: 1
DQMHistoTests: Total nulls: 0
DQMHistoTests: Total successes: 2605166
DQMHistoTests: Total skipped: 171
DQMHistoTests: Total Missing objects: 0
DQMHistoSizes: Histogram memory added: 0.860000000117 KiB( 21 files compared)
Checked 106 log files, 8 edm output root files, 26 DQM output files

perrotta · 2017-11-29T21:12:17Z

+1

Comparing the configuration dumps with and without this PR, I see no changes, except the two typos in photonPostProcessor_cff.py which are fixed here
No changes in jenkins tests, as expected

davidlange6 · 2017-12-01T15:56:10Z

merge

makortel added 2 commits November 20, 2017 15:57

Replace fastSim.isChosen() with toModify()/toReplaceWith() where trivial

aeaebb7

Replace fastSim.isChosen() with ProcessModifiers for the digi aliases

e8ed1aa

This is still ugly, but hopefully bit less than before.

cmsbuild added this to the CMSSW_10_0_X milestone Nov 20, 2017

cmsbuild added code-checks-pending comparison-pending dqm-pending fastsim-pending operations-pending orp-pending pending-signatures reconstruction-pending simulation-pending tests-pending labels Nov 20, 2017

cmsbuild added code-checks-approved and removed code-checks-pending labels Nov 20, 2017

cmsbuild added tests-started and removed tests-pending labels Nov 20, 2017

cmsbuild added tests-approved and removed tests-started labels Nov 20, 2017

cmsbuild added comparison-available and removed comparison-pending labels Nov 20, 2017

perrotta reviewed Nov 24, 2017

View reviewed changes

cmsbuild added the tests-started label Nov 28, 2017

cmsbuild added comparison-notrun tests-rejected and removed comparison-pending tests-started labels Nov 28, 2017

cmsbuild added comparison-pending tests-pending and removed comparison-notrun tests-rejected labels Nov 29, 2017

cmsbuild added tests-started and removed tests-pending labels Nov 29, 2017

cmsbuild added tests-approved and removed tests-started labels Nov 29, 2017

cmsbuild added comparison-available and removed comparison-pending labels Nov 29, 2017

cmsbuild added reconstruction-approved and removed reconstruction-pending labels Nov 29, 2017

cmsbuild added orp-approved and removed orp-pending labels Dec 1, 2017

cmsbuild merged commit 4111261 into cms-sw:master Dec 1, 2017

makortel deleted the removeIsChosenFastSim branch February 12, 2018 13:04

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Replace fastSim.isChosen() with toModify/toReplaceWith/makeProcessModifier #21389

Replace fastSim.isChosen() with toModify/toReplaceWith/makeProcessModifier #21389

makortel commented Nov 20, 2017

cmsbuild commented Nov 20, 2017

cmsbuild commented Nov 20, 2017

cmsbuild commented Nov 20, 2017

makortel commented Nov 20, 2017

cmsbuild commented Nov 20, 2017 •

edited

cmsbuild commented Nov 20, 2017

cmsbuild commented Nov 20, 2017

cmsbuild commented Nov 20, 2017

perrotta Nov 24, 2017

cmsbuild commented Nov 28, 2017

cmsbuild commented Nov 28, 2017

perrotta commented Nov 29, 2017

cmsbuild commented Nov 29, 2017 •

edited

cmsbuild commented Nov 29, 2017

cmsbuild commented Nov 29, 2017

cmsbuild commented Nov 29, 2017

perrotta commented Nov 29, 2017

davidlange6 commented Dec 1, 2017

Replace fastSim.isChosen() with toModify/toReplaceWith/makeProcessModifier #21389

Replace fastSim.isChosen() with toModify/toReplaceWith/makeProcessModifier #21389

Conversation

makortel commented Nov 20, 2017

cmsbuild commented Nov 20, 2017

cmsbuild commented Nov 20, 2017

cmsbuild commented Nov 20, 2017

makortel commented Nov 20, 2017

cmsbuild commented Nov 20, 2017 • edited

cmsbuild commented Nov 20, 2017

cmsbuild commented Nov 20, 2017

cmsbuild commented Nov 20, 2017

perrotta Nov 24, 2017

Choose a reason for hiding this comment

cmsbuild commented Nov 28, 2017

cmsbuild commented Nov 28, 2017

perrotta commented Nov 29, 2017

cmsbuild commented Nov 29, 2017 • edited

cmsbuild commented Nov 29, 2017

cmsbuild commented Nov 29, 2017

cmsbuild commented Nov 29, 2017

perrotta commented Nov 29, 2017

davidlange6 commented Dec 1, 2017

cmsbuild commented Nov 20, 2017 •

edited

cmsbuild commented Nov 29, 2017 •

edited