Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Migrate legacy EDModules to thread-efficient alternatives #36404

Closed
makortel opened this issue Dec 7, 2021 · 36 comments
Closed

Migrate legacy EDModules to thread-efficient alternatives #36404

makortel opened this issue Dec 7, 2021 · 36 comments

Comments

@makortel
Copy link
Contributor

makortel commented Dec 7, 2021

Legacy EDModules have been effectively deprecated since the start of Run 2, and we are now phasing out their support.

Current ways to locate legacy EDModules:

  • Static analyzer lists those not in test/ directories
  • CMSDEPRECATED IB lists all of them as compilation warnings (accompanied with code that needs to be migrated to esConsumes, Migrate ED modules to use esConsumes #31061)
  • cmsRun jobs print out a warning message listing legacy EDModules run in that job
@cmsbuild
Copy link
Contributor

cmsbuild commented Dec 7, 2021

A new Issue was created by @makortel Matti Kortelainen.

@Dr15Jones, @perrotta, @dpiparo, @makortel, @smuzaffar, @qliphy can you please review it and eventually sign/assign? Thanks.

cms-bot commands are listed here

@makortel
Copy link
Contributor Author

makortel commented Dec 7, 2021

assign analysis,code,db,dqm,generators,l1,upgrade

(starting with categories whose legacy EDModules are being run in IBs)

@cmsbuild
Copy link
Contributor

cmsbuild commented Dec 7, 2021

New categories assigned: upgrade,db,analysis,generators,l1,dqm

@jfernan2,@ahmad3213,@AdrianoDee,@mkirsano,@rvenditti,@emanueleusai,@ggovi,@srimanob,@alberto-sanchez,@SiewYan,@santocch,@rekovic,@GurpreetSinghChahal,@francescobrivio,@pbo0,@malbouis,@tvami,@Saptaparna,@cecilecaillol,@agrohsje,@pmandrik you have been requested to review this Pull request/Issue and eventually sign? Thanks

@makortel
Copy link
Contributor Author

makortel commented Dec 7, 2021

As a step towards being able to change the warning message to an exception (goal is by 13_0_0) I gathered EDModules listed by the warning messages in full runTheMatrix and unit test logs, and classified the EDModules by their packages and L2 areas. Below is a list from CMSSW_12_3_X_2021-12-06-2300 (the name(s) after the C++ type name are the module labels in the jobs they were run).

@cms-sw/analysis-l2

PhysicsTools/HepMCCandAlgos:
- ParticleListDrawer printDecaySubset
PhysicsTools/UtilAlgos:
- WrappedEDMuonAnalyzer muonAnalyzer
TopQuarkAnalysis/TopEventProducers:
- TopDecaySubset decaySubset
- TopInitSubset initSubset
- TtFullHadEvtBuilder ttFullHadEvent
- TtFullLepEvtBuilder ttFullLepEvent
- TtGenEventReco genEvt
- TtSemiLepEvtBuilder ttSemiLepEvent
TopQuarkAnalysis/TopHitFit:
- TtSemiLepHitFitProducerMuon hitFitTtSemiLepEvent,hitFitTtSemiLepEventHypothesis
TopQuarkAnalysis/TopKinFitter:
- TtFullHadKinFitProducer kinFitTtFullHadEvent,kinFitTtFullHadEventHypothesis
- TtFullLepKinSolutionProducer kinSolutionTtFullLepEvent,kinSolutionTtFullLepEventHypothesis
- TtSemiLepKinFitProducerMuon kinFitTtSemiLepEvent,kinFitTtSemiLepEventHypothesis,kinFitTtSemiLepEventSelection
TopQuarkAnalysis/TopJetCombination:
- TtFullHadHypGenMatch ttFullHadHypGenMatch
- TtFullHadHypKinFit ttFullHadHypKinFit
- TtFullLepHypGenMatch ttFullLepHypGenMatch
- TtFullLepHypKinSolution ttFullLepHypKinSolution
- TtSemiLepHypGenMatch ttSemiLepHypGenMatch
- TtSemiLepHypGeom ttSemiLepHypGeom
- TtSemiLepHypHitFit ttSemiLepHypHitFit
- TtSemiLepHypKinFit ttSemiLepHypKinFit
- TtSemiLepHypMaxSumPtWMass ttSemiLepHypMaxSumPtWMass
- TtSemiLepHypMVADisc ttSemiLepHypMVADisc
- TtSemiLepHypWMassDeltaTopMass ttSemiLepHypWMassDeltaTopMass
- TtSemiLepHypWMassMaxSumPt ttSemiLepHypWMassMaxSumPt
- TtSemiLepJetCombGeom findTtSemiLepJetCombGeom
- TtSemiLepJetCombMaxSumPtWMass findTtSemiLepJetCombMaxSumPtWMass
- TtSemiLepJetCombWMassDeltaTopMass findTtSemiLepJetCombWMassDeltaTopMass
- TtSemiLepJetCombWMassMaxSumPt findTtSemiLepJetCombWMassMaxSumPt
TopQuarkAnalysis/TopSkimming:
- TtDecayChannelFilter ttFullHadronicFilter,ttFullLeptonicFilter,ttSemiLeptonicFilter
TopQuarkAnalysis/TopTools:
- TtFullHadJetPartonMatch ttFullHadJetPartonMatch
- TtFullLepJetPartonMatch ttFullLepJetPartonMatch
- TtSemiLepJetPartonMatch ttSemiLepJetPartonMatch

@cms-sw/core-l2

FWCore/Framework:
- IntLegacyProducer one,two
- TestBeginEndJobAnalyzer m1
- TestFailuresAnalyzer m1
FWCore/Integration:
- HierarchicalEDProducer h
- ProdigalAnalyzer analyzer

@cms-sw/db-l2

OnlineDB/SiStripO2O:
- SiStripO2OBadStrip siStripO2O

@cms-sw/dqm-l2

DQM/BeamMonitor:
- AlcaBeamMonitorClient AlcaBeamMonitorClient
DQM/EcalCommon:
- EcalMonitorPrescaler ecalPhysicsFilter
DQM/EcalPreshowerMonitorModule:
- ESDaqInfoTask ecalPreshowerDaqInfoTask
- ESDataCertificationTask ecalPreshowerDataCertificationTask
- ESDcsInfoTask ecalPreshowerDcsInfoTask
DQM/SiStripMonitorClient:
- SiStripAnalyser SiStripAnalyser
- SiStripCertificationInfo siStripCertificationInfo
- SiStripDaqInfo siStripDaqInfo
- SiStripDcsInfo siStripDcsInfo
DQMOffline/Trigger:
- DQMOfflineHLTEventInfoClient HLTOverallSummary
- EgHLTOfflineSummaryClient egHLTOffDQMSummaryClient
- HLTInclusiveVBFClient hltInclusiveVBFClient
DQMServices/Components:
- DQMFEDIntegrityClient dqmFEDIntegrity
- DQMMessageLoggerClient DQMMessageLoggerClient
DQMServices/Demo:
- TestLegacyEDAnalyzer testlegacy
- TestLegacyFillRunEDAnalyzer testlegacyfillrun
- TestLegacyFillLumiEDAnalyzer testlegacyfilllumi
- TestLegacyHarvester testlegacyharvester
DQMServices/FwkIO:
- DummyReadDQMStore reader
Validation/RecoEgamma:
- ConversionPostprocessing conversionPostprocessing
- PhotonPostprocessing pfPhotonPostprocessing,photonPostprocessing
Validation/RecoTau:
- DQMHistNormalizer normalizePlotsZTT

@cms-sw/generators-l2

GeneratorInterface/LHEInterface:
- DummyLHEAnalyzer lhedump

@cms-sw/l1-l2

DataFormats/L1TrackTrigger:
- TTTrackTrackWordDummyOneAnalyzer dummyAna
EventFilter/CSCTFRawToDigi:
- CSCTFSingleGen csctfsinglegen
- CSCTFAnalyzer csctfanalyzer

@cms-sw/upgrade-l2

DataFormats/L1TrackTrigger:
- TTTrackTrackWordDummyOneAnalyzer dummyAna

@makortel
Copy link
Contributor Author

makortel commented Dec 7, 2021

assign core

@cmsbuild
Copy link
Contributor

cmsbuild commented Dec 7, 2021

New categories assigned: core

@Dr15Jones,@smuzaffar,@makortel you have been requested to review this Pull request/Issue and eventually sign? Thanks

@agrohsje
Copy link

agrohsje commented Dec 8, 2021

Hi @SiewYan . Do you mind updating
GeneratorInterface/LHEInterface:

  • DummyLHEAnalyzer lhedump ?
    Thanks!

@SiewYan
Copy link
Contributor

SiewYan commented Dec 8, 2021

@agrohsje , i will see to it!

@jfernan2
Copy link
Contributor

jfernan2 commented Dec 9, 2021

Thanks @makortel
Regarding the DQM modules, some of them (see below) are running on Online DQM (single thread) or in dedicated test configs or Unit Tests. Anyway, could you please remind in this thread some information about which are the alternatives and some instructions?

I found these two twikis but I am not sure if they are up to date:
https://twiki.cern.ch/twiki/bin/view/CMSPublic/FWMultithreadedThreadSafeDataStructures
https://twiki.cern.ch/twiki/bin/view/CMSPublic/FWMultithreadedAnalysisEDAnalyzer
Thank you

ecalPhysicsFilter
SiStripAnalyser
egHLTOffDQMSummaryClient
testlegacy*
DummyReadDQMStore reader

@makortel
Copy link
Contributor Author

makortel commented Dec 9, 2021

@jfernan2 Those two links are largely up to date. In addition, the different module types (globla/stream/one) are documented in https://twiki.cern.ch/twiki/bin/view/CMSPublic/FWMultithreadedFrameworkModuleTypes and links in there.

Some general rules of thumb

(there are many cases that do not fit in these categories, but I'd expect those to be easiest to discuss case-by-case)

@makortel
Copy link
Contributor Author

makortel commented Dec 9, 2021

@jfernan2 Looking at the specific cases you listed

  • EcalMonitorPrescaler ecalPhysicsFilter could be migrated to edm::global::EDFilter<edm::RunCache<...>> when using atomics for the counters (or without atomics to edm::one with RunCache, but it would be less efficient)
  • SiStripAnalyser SiStripAnalyser looks complicated, it also accesses DQMStore directly
  • EgHLTOfflineSummaryClient egHLTOffDQMSummaryClient I'd try to migrate to DQMEDHarvester
  • TestLegacy* testlegacy* eventually remove?
  • DummyReadDQMStore reader I'd try edm::one, similarly to DQMEDHarvester (but likely fewer extensions are needed)

@makortel
Copy link
Contributor Author

Around 2022-05-02 the deprecation annotations in CMSDEPRECATED_X were updated such that they should now flag reliably all classes that derive from legacy EDModule (or at least more reliably than earlier).

I also went through the logs of CMSSW_12_5_X_2022-05-17-2300 matrix tests and unit tests, and the following legacy EDModules are being run there for sure (there can be more that my log parsing approach fails to find)

@cms-sw/analysis-l2

PhysicsTools/UtilAlgos:
- WrappedEDMuonAnalyzer muonAnalyzer
TopQuarkAnalysis/TopKinFitter:
- TtFullLepKinSolutionProducer kinSolutionTtFullLepEvent,kinSolutionTtFullLepEventHypothesis

@cms-sw/core-l2

FWCore/Framework:
- IntLegacyProducer one,two

@cms-sw/dqm-l2

DQM/BeamMonitor:
- AlcaBeamMonitorClient AlcaBeamMonitorClient
DQM/EcalCommon:
- EcalMonitorPrescaler ecalPhysicsFilter
DQM/EcalPreshowerMonitorModule:
- ESDaqInfoTask ecalPreshowerDaqInfoTask
- ESDataCertificationTask ecalPreshowerDataCertificationTask
- ESDcsInfoTask ecalPreshowerDcsInfoTask
DQM/SiStripMonitorClient:
- SiStripAnalyser SiStripAnalyser
- SiStripCertificationInfo siStripCertificationInfo
- SiStripDaqInfo siStripDaqInfo
- SiStripDcsInfo siStripDcsInfo
DQMOffline/Trigger:
- DQMOfflineHLTEventInfoClient HLTOverallSummary
- EgHLTOfflineSummaryClient egHLTOffDQMSummaryClient
- HLTInclusiveVBFClient hltInclusiveVBFClient
DQMServices/Demo:
- TestLegacyEDAnalyzer testlegacy
- TestLegacyFillRunEDAnalyzer testlegacyfillrun
- TestLegacyFillLumiEDAnalyzer testlegacyfilllumi
- TestLegacyHarvester testlegacyharvester
DQMServices/FwkIO:
- DummyReadDQMStore reader
Validation/RecoTau:
- DQMHistNormalizer normalizePlotsZTT

Thanks for the help! The list continues to shrink.

@makortel
Copy link
Contributor Author

I updated the list of legacy EDModules still being run as part of runTheMatrix or unit tests for sure (there can be more that my log parsing approach fails to find) in CMSSW_12_6_X_2022-09-27-2300

Former @cms-sw/analysis-l2

PhysicsTools/UtilAlgos:
- WrappedEDMuonAnalyzer muonAnalyzer

@cms-sw/core-l2

FWCore/Framework:
- IntLegacyProducer one,two

@cms-sw/dqm-l2

DQM/BeamMonitor:
- AlcaBeamMonitorClient AlcaBeamMonitorClient
DQM/EcalPreshowerMonitorModule:
- ESDaqInfoTask ecalPreshowerDaqInfoTask
- ESDataCertificationTask ecalPreshowerDataCertificationTask
- ESDcsInfoTask ecalPreshowerDcsInfoTask
DQM/SiStripMonitorClient:
- SiStripAnalyser SiStripAnalyser
- SiStripCertificationInfo siStripCertificationInfo
- SiStripDaqInfo siStripDaqInfo
- SiStripDcsInfo siStripDcsInfo
DQMServices/Demo:
- TestLegacyEDAnalyzer testlegacy
- TestLegacyFillRunEDAnalyzer testlegacyfillrun
- TestLegacyFillLumiEDAnalyzer testlegacyfilllumi
- TestLegacyHarvester testlegacyharvester
DQMServices/FwkIO:
- DummyReadDQMStore reader
Validation/RecoTau:
- DQMHistNormalizer normalizePlotsZTT

@tvami
Copy link
Contributor

tvami commented Sep 28, 2022

I happen to do a tk offline shift this week and started to deal with

DQM/SiStripMonitorClient:

  • SiStripAnalyser SiStripAnalyser
  • SiStripCertificationInfo siStripCertificationInfo
  • SiStripDaqInfo siStripDaqInfo
  • SiStripDcsInfo siStripDcsInfo

in #39518

@tvami
Copy link
Contributor

tvami commented Sep 28, 2022

For

DQM/BeamMonitor:

  • AlcaBeamMonitorClient AlcaBeamMonitorClient

@dzuolo can you fix this please?

@wddgit
Copy link
Contributor

wddgit commented Sep 28, 2022

I'm working on these now:

  • DQM/EcalPreshowerMonitorModule:
  • DQMServices/FwkIO

@Dr15Jones
Copy link
Contributor

Dr15Jones commented Oct 3, 2022

I did
CommonTools/RecoUtils #39584
DataFormat/Math #39585
DQM/DataScouting #39586
EventFilter/EcalRawToDigi #39587
EventFilter/GctRawToDigi #39588
EventFilter/L1GlobalTriggerRawToDigi #39589

EventFilter/LTCRawToDigi #39603
EventFilter/RawDataCollector #39604
HLTrigger/btau #39610
IORawData/DTCommissioning #39613
JetMETCorrections/JetVertexAssociation #39614

JetMETCorrections/Type1MET #39628
L1Trigger/GlobalCaloTrigger #39629
L1Trigger/L1ExtraFromDigis #39632
L1Trigger/TextToDigi #39637
PhysicsTools/CandAlgos #39639
RecoBTag/SoftLepton #39640
RecoHI/HiJetAlgos #39643

RecoHI/HiTracking #39654
RecoJets/JetAssociationProducers #39657
RecoJets/JetProducers #39658
RecoLocalCalo/HcalLaserReco #39660
RecoMuon/CosmicMuonProducer #39661
RecoMuon/GlobalMuonProducer #39662
RecoMuon/GlobalTrackingTools #39663

RecoMuon/MuonIdentification
RecoMuon/MuonIsolation
RecoMuon/MuonIsolationProducers #39674

RecoTBCalo/EcalTBHodoscopeReconstructor
RecoTBCalo/EcalTBTDCReconstructor
RecoTBCalo/HcalTBObjectUnpacker #39696

RecoTracker/DebugTools
RecoTracker/SpecialSeedGenerators
RecoVertex/AdaptiveVertexFinder #39698

SimPPS/RPDigiProducer #39699
TopQuarkAnalysis/TopEventProducers #39701

@wddgit
Copy link
Contributor

wddgit commented Oct 4, 2022

These are done and the PRs are submitted now:

DQM/EcalPreshowerMonitorModule (PR #39616):

  • ESDaqInfoTask ecalPreshowerDaqInfoTask
  • ESDataCertificationTask ecalPreshowerDataCertificationTask
  • ESDcsInfoTask ecalPreshowerDcsInfoTask

DQMServices/FwkIO (PR #39541):

  • DummyReadDQMStore reader

@makortel
Copy link
Contributor Author

For

DQM/BeamMonitor:

  • AlcaBeamMonitorClient AlcaBeamMonitorClient

@dzuolo can you fix this please?

@tvami @dzuolo Any news on this? Thanks!

@makortel
Copy link
Contributor Author

DQMServices/Demo:
- TestLegacyEDAnalyzer testlegacy
- TestLegacyFillRunEDAnalyzer testlegacyfillrun
- TestLegacyFillLumiEDAnalyzer testlegacyfilllumi
- TestLegacyHarvester testlegacyharvester

@cms-sw/dqm-l2 I suppose these modules could just be removed. When do you think that could be done? Could you take care of that? Thanks!

@emanueleusai
Copy link
Member

ok I will take care of it this week

@tvami
Copy link
Contributor

tvami commented Oct 17, 2022

DQM/BeamMonitor:
AlcaBeamMonitorClient AlcaBeamMonitorClient
@dzuolo can you fix this please?
@tvami @dzuolo Any news on this? Thanks!

Nothing yet, @francescobrivio do we have anybody else in the beamspot team who could do this? Simone maybe?

I should also note @makortel that technically this is a dqm signature module, not alca

@dzuolo
Copy link
Contributor

dzuolo commented Oct 17, 2022

DQM/BeamMonitor:

AlcaBeamMonitorClient AlcaBeamMonitorClient

@dzuolo can you fix this please?

@tvami @dzuolo Any news on this? Thanks!

Nothing yet, @francescobrivio do we have anybody else in the beamspot team who could do this? Simone maybe?

I should also note @makortel that technically this is a dqm signature module, not alca

Dear colleagues,
Unfortunately at the moment nobody from the beamspot team is available to work on this topic due to a lack of time and/or knowledge of the core software. @gennai is for sure the most qualified person but he is busy with other activities. Would like to add something simone?

@Dr15Jones
Copy link
Contributor

Just did

#39746
L1TriggerConfig/CSCTFConfigProducers
L1TriggerConfig/DTTPGConfigProducers
L1TriggerConfig/L1GeometryProducers
L1TriggerConfig/RCTConfigProducers

@wddgit
Copy link
Contributor

wddgit commented Oct 17, 2022

I just did L1Trigger/L1TMuonOverlap. I was just starting L1TriggerConfig, your comment was just in time. I'll do RecoJets/JetAnalyzers next.

@Dr15Jones
Copy link
Contributor

Dr15Jones commented Oct 20, 2022

Just did

#39798
L1Trigger/DTTrackFinder
L1Trigger/DTTrigger
L1Trigger/GlobalTrigger
L1Trigger/L1GctAnalyzer
L1Trigger/L1TCalorimeter
L1Trigger/L1TGEM
L1Trigger/L1TMuonEndCap

#39801
RecoEgamma/EgammaElectronProducers RecoEgamma/EgammaIsolationAlgos RecoEgamma/EgammaTools

@Dr15Jones
Copy link
Contributor

Dr15Jones commented Oct 21, 2022

Did

#39819

JetMETCorrections/
       IsolatedParticles  MCJet  TauJet

HLTrigger/HLTanalyzers #39821

@makortel
Copy link
Contributor Author

DQM/BeamMonitor:
AlcaBeamMonitorClient AlcaBeamMonitorClient

@dzuolo can you fix this please?

@tvami @dzuolo Any news on this? Thanks!

Nothing yet, @francescobrivio do we have anybody else in the beamspot team who could do this? Simone maybe?

AlcaBeamMonitorClient is now migrated in #39888 (in a suboptimal way though, that does not use DQM base classes and prevents concurrent lumis).

I should also note @makortel that technically this is a dqm signature module, not alca

I know, but I replied to your (@tvami) comment in #36404 (comment).

@makortel
Copy link
Contributor Author

makortel commented Nov 3, 2022

DQMServices/Demo:
- TestLegacyEDAnalyzer testlegacy
- TestLegacyFillRunEDAnalyzer testlegacyfillrun
- TestLegacyFillLumiEDAnalyzer testlegacyfilllumi
- TestLegacyHarvester testlegacyharvester

@cms-sw/dqm-l2 I suppose these modules could just be removed. When do you think that could be done? Could you take care of that? Thanks!

ok I will take care of it this week

@emanueleusai These are now one of the few last legacy modules. Would you be able to take care of them soon? Or should we (core) make an attempt to remove them?

@makortel
Copy link
Contributor Author

DQMServices/Demo:
- TestLegacyEDAnalyzer testlegacy
- TestLegacyFillRunEDAnalyzer testlegacyfillrun
- TestLegacyFillLumiEDAnalyzer testlegacyfilllumi
- TestLegacyHarvester testlegacyharvester

@cms-sw/dqm-l2 I suppose these modules could just be removed. When do you think that could be done? Could you take care of that? Thanks!

ok I will take care of it this week

@emanueleusai These are now one of the few last legacy modules. Would you be able to take care of them soon? Or should we (core) make an attempt to remove them?

@emanueleusai Would you have any update when these modules could be removed (or otherwise be taken care of)? Or should we make an attempt?

@makortel
Copy link
Contributor Author

DQMServices/Demo:
- TestLegacyEDAnalyzer testlegacy
- TestLegacyFillRunEDAnalyzer testlegacyfillrun
- TestLegacyFillLumiEDAnalyzer testlegacyfilllumi
- TestLegacyHarvester testlegacyharvester

These are now migrated in #40409.

@makortel
Copy link
Contributor Author

+core

All legacy EDModules in CMSSW have been migrated, and the support for legacy EDModules has been removed in #39710 and #40466

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

No branches or pull requests