Better era imports #16001

kpedro88 · 2016-09-27T18:12:23Z

This PR addresses ~90% of the instances of importing Configuration.StandardSequences.Eras, replacing those imports with the modifier-specific cff files now available. Most of the edits were made automatically using a Python script, with a few corner cases fixed by hand afterward.

Remaining types of Era imports still to be dealt with:

In various Tracking cff files, there is a call to getattr(eras,era) in a general loop.
cmsRun configs generated by cmsDriver use the import to initialize the cms.Process object. (Most of the tracked configs are stored in test directories, so aren't picked up by checkdeps.)
Only in Configuration/DataProcessing, import Configuration.StandardSequences.Eras as eras.

These special cases of imports should probably be addressed by someone who understands the expected function of the code.

This PR is going to touch a huge number of categories, but I would appreciate merging it quickly once it's confirmed not to cause any regressions, in order to avoid never-ending rebases. We should start enforcing the new way of importing specific modifiers in open PRs. I tested this PR using the short matrix in the latest IB and all tests passed.

attn: @davidlange6, @Dr15Jones, @slava77

cmsbuild · 2016-09-27T18:13:02Z

A new Pull Request was created by @kpedro88 (Kevin Pedro) for CMSSW_8_1_X.

It involves the following packages:

CalibCalorimetry/HcalPlugins
CalibMuon/CSCCalibration
CommonTools/RecoAlgos
Configuration/EventContent
Configuration/StandardSequences
DQM/BeamMonitor
DQM/HcalTasks
DQM/L1TMonitor
DQM/Physics
DQM/SiPixelCommon
DQM/SiPixelMonitorCluster
DQM/SiPixelMonitorDigi
DQM/SiPixelMonitorRawData
DQM/SiPixelMonitorRecHit
DQM/SiPixelMonitorTrack
DQM/SiStripMonitorClient
DQMOffline/Configuration
DQMOffline/JetMET
DQMOffline/L1Trigger
DQMOffline/Muon
EventFilter/CSCRawToDigi
EventFilter/HcalRawToDigi
EventFilter/RawDataCollector
EventFilter/SiPixelRawToDigi
FastSimulation/Calorimetry
FastSimulation/TrajectoryManager
Geometry/CSCGeometryBuilder
HLTriggerOffline/Btag
HLTriggerOffline/Common
HLTriggerOffline/SUSYBSM
IOMC/RandomEngine
L1Trigger/CSCTrackFinder
L1Trigger/CSCTriggerPrimitives
L1Trigger/Configuration
L1Trigger/L1ExtraFromDigis
L1Trigger/L1TCalorimeter
L1Trigger/L1TGlobal
L1Trigger/L1TMuon
PhysicsTools/PatAlgos
RecoEcal/Configuration
RecoEcal/EgammaClusterProducers
RecoEgamma/Configuration
RecoEgamma/EgammaElectronProducers
RecoEgamma/EgammaPhotonProducers
RecoEgamma/PhotonIdentification
RecoJets/Configuration
RecoJets/JetProducers
RecoLocalCalo/Configuration
RecoLocalMuon/CSCRecHitD
RecoLocalMuon/Configuration
RecoLocalMuon/RPCRecHit
RecoLocalTracker/Configuration
RecoLocalTracker/Phase2ITPixelClusterizer
RecoLocalTracker/SiPixelClusterizer
RecoLocalTracker/SiPixelRecHits
RecoLocalTracker/SiStripClusterizer
RecoLocalTracker/SiStripZeroSuppression
RecoLuminosity/LumiProducer
RecoMET/Configuration
RecoMET/METFilters
RecoMuon/Configuration
RecoMuon/GlobalTrackingTools
RecoMuon/MuonIdentification
RecoMuon/TrackingTools
RecoParticleFlow/Configuration
RecoParticleFlow/PFClusterProducer
RecoParticleFlow/PFProducer
RecoParticleFlow/PFTracking
RecoPixelVertexing/Configuration
RecoPixelVertexing/PixelLowPtUtilities
RecoPixelVertexing/PixelTrackFitting
RecoPixelVertexing/PixelTriplets
RecoTracker/ConversionSeedGenerators
RecoTracker/FinalTrackSelectors
RecoTracker/IterativeTracking
RecoTracker/MeasurementDet
RecoTracker/SpecialSeedGenerators
RecoTracker/TkSeedGenerator
RecoTracker/TkSeedingLayers
RecoTracker/TkTrackingRegions
RecoTracker/TrackProducer
RecoTracker/TransientTrackingRecHit
RecoVertex/Configuration
RecoVertex/PrimaryVertexProducer
SLHCUpgradeSimulations/Configuration
SimCalorimetry/Configuration
SimCalorimetry/EcalTrigPrimProducers
SimCalorimetry/HcalSimProducers
SimCalorimetry/HcalZeroSuppressionProducers
SimG4Core/Application
SimG4Core/Configuration
SimGeneral/Configuration
SimGeneral/DataMixingModule
SimGeneral/MixingModule
SimGeneral/TrackingAnalysis
SimMuon/CSCDigitizer
SimMuon/Configuration
SimMuon/DTDigitizer
SimMuon/MCTruth
SimMuon/RPCDigitizer
SimTracker/TrackAssociatorProducers
TrackingTools/TrackAssociator
Validation/Configuration
Validation/DTRecHits
Validation/EcalDigis
Validation/EcalHits
Validation/EcalRecHits
Validation/HcalDigis
Validation/HcalHits
Validation/HcalRecHits
Validation/MuonCSCDigis
Validation/MuonDTDigis
Validation/MuonHits
Validation/MuonIdentification
Validation/MuonRPCDigis
Validation/RPCRecHits
Validation/RecoEgamma
Validation/RecoMuon
Validation/RecoTau
Validation/RecoTrack
Validation/RecoVertex

@ghellwig, @lveldere, @ianna, @cerminar, @rekovic, @vanbesien, @civanch, @monttj, @cmsbuild, @davidlange6, @smuzaffar, @Dr15Jones, @cvuosalo, @mdhildreth, @slava77, @mommsen, @mmusich, @mulhearn, @ssekmen, @emeschi, @dmitrijus, @franzoni can you please review it and eventually sign? Thanks.
@ghellwig, @TaiSakuma, @gouskos, @yslai, @forthommel, @yduhm, @rappoccio, @argiro, @Martin-Grunewald, @imarches, @fioriNTU, @ahinzmann, @lgray, @abbiendi, @dgulhan, @venturia, @mmarionncern, @hdelanno, @calderona, @makortel, @threus, @acaudron, @sdevissc, @jhgoh, @amagitte, @valuev, @jdolen, @jlagram, @HuguesBrun, @ferencek, @OlivierBondu, @trocino, @rociovilar, @Sam-Harper, @barvic, @wmtan, @GiacomoSguazzoni, @rafaellopesdesa, @tocheng, @VinInn, @bellan, @nhanvtran, @gkasieczka, @rovere, @deguio, @schoef, @mschrode, @idebruyn, @ptcox, @richard-cms, @nickmccoll, @battibass, @gbenelli, @wddgit, @JyothsnaKomaragiri, @mverzett, @dkotlins, @wmtford, @cbernet, @gpetruc, @matt-komm, @mariadalfonso, @pvmulder, @bachtis this is something you requested to watch as well.
@slava77, @smuzaffar you are the release manager for this.

cms-bot commands are list here #13028

kpedro88 · 2016-09-27T18:13:50Z

@cmsbuild please test

cmsbuild · 2016-09-27T18:14:26Z

The tests are being triggered in jenkins.
https://cmssdt.cern.ch/jenkins/job/ib-any-integration/15410/console

kpedro88 · 2016-09-27T18:15:22Z

Since the bot ran out of tags:
@HuguesBrun, @ferencek, @OlivierBondu, @trocino, @rociovilar, @Sam-Harper, @barvic, @wmtan, @GiacomoSguazzoni, @rafaellopesdesa, @tocheng, @VinInn, @bellan, @nhanvtran, @gkasieczka, @rovere, @deguio, @schoef, @mschrode, @idebruyn, @ptcox, @richard-cms, @nickmccoll, @battibass, @gbenelli, @wddgit, @JyothsnaKomaragiri, @mverzett, @dkotlins, @wmtford, @cbernet, @gpetruc, @matt-komm, @mariadalfonso, @pvmulder, @bachtis this is something you requested to watch as well.

slava77 · 2016-09-27T18:22:51Z

@kpedro88
which tests have passed, actually? Did you see that the matrix ran, or did you check comparisons as well?

Please check expanded configs in several standard workflows to confirm that your changes provide identical results.

kpedro88 · 2016-09-27T18:24:43Z

@slava77 I saw that all workflows in the short matrix ran to completion with no errors. I was planning to rely on the comparisons from the bot. Is there a way that I can run the comparison generation script myself?

slava77 · 2016-09-27T18:29:31Z

comparisons are not enough.
There are many things that are not monitored.
So, the best check is the expanded configuration (python dump)

We post a link to Reco tools in every RECO/AT meeting.
comparisons are described in https://twiki.cern.ch/twiki/bin/viewauth/CMS/RecoIntegration#Compare_outputs_FWLite_style and a similar DQM.
If you want to run RelMon-based comparisons, you can find the individual wflow scripts in the cms-bot outputs (next to the output directory)

kpedro88 · 2016-09-27T18:31:29Z

Ah, I see what you mean now. I can write a script to call edmConfigDump on everything and diff vs. a clean IB.

makortel · 2016-09-27T19:36:10Z

In various Tracking cff files, there is a call to getattr(eras,era) in a general loop.

I can take care of this (after this PR gets merged).

kpedro88 · 2016-09-27T19:40:45Z

@makortel thanks! (the second commit in this PR shows which files have that usage)

slava77 · 2016-09-27T21:22:34Z

On 9/27/16 1:32 PM, Kevin Pedro wrote:

@slava77 https://github.com/slava77 I wrote a script to run
|edmConfigDump| on all the .py files generated by the short matrix in my
working area and in a clean IB |CMSSW_8_1_X_2016-09-27-1100|. I then ran
a script to |diff| each pair of config dumps. The diffs showed that all
the dumps are identical. (I also checked a few random ones by hand to
confirm.)

Nice.
Thank you for checking.

—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
#16001 (comment), or
mute the thread
https://github.com/notifications/unsubscribe-auth/AEdcbqJZWQQph1i325isR3XmlnsOc3Lfks5quX1lgaJpZM4KH_AY.

Dr15Jones · 2016-09-27T21:28:37Z

+1

mulhearn · 2016-09-27T23:05:53Z

+1

cmsbuild · 2016-09-28T00:00:08Z

+1
Tested at: 1eb4481
https://cmssdt.cern.ch/SDT/jenkins-artifacts/pull-request-integration/PR-16001/15410/summary.html

cmsbuild · 2016-09-28T01:05:11Z

Comparison is ready
https://cmssdt.cern.ch/SDT/jenkins-artifacts/pull-request-integration/PR-16001/15410/summary.html

kpedro88 · 2016-09-28T15:45:10Z

Comparisons look good to me at a glance.

Again, would prefer to merge quickly to avoid conflicts popping up... (this will also make it quicker to test any other PRs that modify Era-related code)

davidlange6 · 2016-09-28T15:50:24Z

I agree. All - any additional checks needed beyond what can be done over the next few hours (eg, as to merge by the next IB)?

On Sep 28, 2016, at 5:45 PM, Kevin Pedro notifications@github.com wrote:

Comparisons look good to me at a glance.

Again, would prefer to merge quickly to avoid conflicts popping up... (this will also make it quicker to test any other PRs that modify Era-related code)

—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub, or mute the thread.

davidlange6 · 2016-09-28T19:29:28Z

please comment if not ok even if now merged

makortel · 2016-09-29T07:16:29Z

(this is probably the wrong place to comment but anyway)

Interestingly, in CMSSW_8_1_X_2016-09-28-2300 after a single edit in Configuration/StandardSequences/python/Eras.py git cms-checkdeps -a still adds gazillion packages. Checking /cvmfs/cms-ib.cern.ch/week0/slc6_amd64_gcc530/cms/cmssw-patch/CMSSW_8_1_X_2016-09-28-2300/etc/dependencies/pyusedby.out.gz (where the dependence information is apparently read by git-cms-checkdeps), it indeed lists many files "depending" on Eras.py.

Are we missing something (additional) here?

davidlange6 · 2016-09-29T07:20:10Z

i guess it will change once we have a full build of the IB instead of a patch. we’ll try to trigger a full build today

On Sep 29, 2016, at 9:16 AM, Matti Kortelainen notifications@github.com wrote:

(this is probably the wrong place to comment but anyway)

Interestingly, in CMSSW_8_1_X_2016-09-28-2300 after a single edit in Configuration/StandardSequences/python/Eras.py git cms-checkdeps -a still adds gazillion packages. Checking /cvmfs/cms-ib.cern.ch/week0/slc6_amd64_gcc530/cms/cmssw-patch/CMSSW_8_1_X_2016-09-28-2300/etc/dependencies/pyusedby.out.gz (where the dependence information is apparently read by git-cms-checkdeps), it indeed lists many files "depending" on Eras.py.

Are we missing something (additional) here?

—
You are receiving this because you modified the open/close state.
Reply to this email directly, view it on GitHub, or mute the thread.

makortel · 2016-09-29T07:22:09Z

Ok, so the list needs a full build (I bit of suspected that), thanks.

kpedro88 · 2016-09-30T19:52:14Z

@makortel CMSSW_8_1_X_2016-09-29-2300 was a full build. I tried modifying Configuration.StandardSequences.Eras and running checkdeps, and now it only finds 10 packages. Hurray!

Configuration/Applications (python)
Configuration/DataProcessing (python)
DQM/Integration (python)
DQM/TrackingMonitorSource (python)
HLTrigger/Configuration (python)
L1Trigger/L1TNtuples (python)
L1Trigger/TrackTrigger (python)
RecoTracker/IterativeTracking (python)
Validation/RecoTau (python)
Validation/RecoTrack (python)

slava77 · 2016-09-30T20:10:47Z

On 9/30/16 12:52 PM, Kevin Pedro wrote:

@makortel https://github.com/makortel |CMSSW_8_1_X_2016-09-29-2300|
was a full build. I tried modifying Configuration.StandardSequences.Eras
and running checkdeps, and now it only finds 10 packages. Hurray!

nice

|Configuration/Applications (python) Configuration/DataProcessing
(python) DQM/Integration (python) DQM/TrackingMonitorSource (python)
HLTrigger/Configuration (python) L1Trigger/L1TNtuples (python)
L1Trigger/TrackTrigger (python) RecoTracker/IterativeTracking (python)
Validation/RecoTau (python) Validation/RecoTrack (python) |

—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
#16001 (comment), or
mute the thread
https://github.com/notifications/unsubscribe-auth/AEdcbhl-9JCniFNwUysd2hnjaU0BHp_dks5qvWhvgaJpZM4KH_AY.

makortel · 2016-10-03T07:28:23Z

@kpedro88 Nice indeed :) I'll attack now on the remaining imports in tracking.

kpedro88 added 2 commits September 27, 2016 13:01

remove (most) direct importing of Configuration.StandardSequences.Eras

183087b

restore eras import in tracking cffs (needs to be addressed separately)

1eb4481

cmsbuild added this to the Next CMSSW_8_1_X milestone Sep 27, 2016

cmsbuild added alca-pending analysis-pending comparison-pending core-pending daq-pending dqm-pending fastsim-pending geometry-pending l1-pending operations-pending orp-pending pending-signatures reconstruction-pending simulation-pending tests-pending labels Sep 27, 2016

cmsbuild added tests-started and removed tests-pending labels Sep 27, 2016

cmsbuild added core-approved geometry-approved and removed core-pending geometry-pending labels Sep 27, 2016

cmsbuild added l1-approved and removed l1-pending labels Sep 27, 2016

cmsbuild added tests-approved and removed tests-started labels Sep 28, 2016

cmsbuild added comparison-available and removed comparison-pending labels Sep 28, 2016

davidlange6 merged commit df1fc8b into cms-sw:CMSSW_8_1_X Sep 28, 2016

jbsauvan mentioned this pull request Sep 29, 2016

Add HGCAL L1 trigger in SimL1Emulator and event content #15952

Merged

makortel mentioned this pull request Sep 29, 2016

Clean up temporary 2017 workflows and eras, add "2017 design" workflow #15994

Merged

slava77 mentioned this pull request Sep 29, 2016

Address residual code review for Timing PRs #15938

Merged

makortel mentioned this pull request Oct 4, 2016

Remove Eras.py imports from tracking #16086

Merged

kpedro88 mentioned this pull request Oct 25, 2016

[Core] Split Eras.py into multiple files #14895

Open

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Better era imports #16001

Better era imports #16001

kpedro88 commented Sep 27, 2016

cmsbuild commented Sep 27, 2016

kpedro88 commented Sep 27, 2016

cmsbuild commented Sep 27, 2016 •

edited

kpedro88 commented Sep 27, 2016

slava77 commented Sep 27, 2016

kpedro88 commented Sep 27, 2016

slava77 commented Sep 27, 2016

kpedro88 commented Sep 27, 2016

makortel commented Sep 27, 2016

kpedro88 commented Sep 27, 2016

slava77 commented Sep 27, 2016

Dr15Jones commented Sep 27, 2016

mulhearn commented Sep 27, 2016

cmsbuild commented Sep 28, 2016

cmsbuild commented Sep 28, 2016

kpedro88 commented Sep 28, 2016

davidlange6 commented Sep 28, 2016

davidlange6 commented Sep 28, 2016

makortel commented Sep 29, 2016

davidlange6 commented Sep 29, 2016

makortel commented Sep 29, 2016

kpedro88 commented Sep 30, 2016

slava77 commented Sep 30, 2016

makortel commented Oct 3, 2016

Better era imports #16001

Better era imports #16001

Conversation

kpedro88 commented Sep 27, 2016

cmsbuild commented Sep 27, 2016

kpedro88 commented Sep 27, 2016

cmsbuild commented Sep 27, 2016 • edited

kpedro88 commented Sep 27, 2016

slava77 commented Sep 27, 2016

kpedro88 commented Sep 27, 2016

slava77 commented Sep 27, 2016

kpedro88 commented Sep 27, 2016

makortel commented Sep 27, 2016

kpedro88 commented Sep 27, 2016

slava77 commented Sep 27, 2016

Dr15Jones commented Sep 27, 2016

mulhearn commented Sep 27, 2016

cmsbuild commented Sep 28, 2016

cmsbuild commented Sep 28, 2016

kpedro88 commented Sep 28, 2016

davidlange6 commented Sep 28, 2016

davidlange6 commented Sep 28, 2016

makortel commented Sep 29, 2016

davidlange6 commented Sep 29, 2016

makortel commented Sep 29, 2016

kpedro88 commented Sep 30, 2016

slava77 commented Sep 30, 2016

makortel commented Oct 3, 2016

cmsbuild commented Sep 27, 2016 •

edited