Storing Generator Filter Information in NanoAOD #34169
Conversation
|
-code-checks Logs: https://cmssdt.cern.ch/SDT/code-checks/cms-sw-PR-34169/23376
Code check has found code style and quality issues which could be resolved by applying following patch(s)
|
mzarucki
force-pushed
the
nanoAOD_genFilterEff_120X
branch
from
0966e78
to
1eb6263
Compare
|
-code-checks Logs: https://cmssdt.cern.ch/SDT/code-checks/cms-sw-PR-34169/23377
Code check has found code style and quality issues which could be resolved by applying following patch(s)
|
|
As discussed in yesterday's X-POG meeting, there might still be some open questions/comments:
I leave the above considerations to the experts. Cheers, |
| @@ -271,3 +272,134 @@ class FirstObjectSimpleFlatTableProducer : public SimpleFlatTableProducerBase<T, | |||
| return out; | |||
| } | |||
| }; | |||
|
|
|||
| template <typename T, typename TProd> | |||
| class SimpleFlatTableProducerBaseLumi : public edm::one::EDProducer<edm::EndLuminosityBlockProducer> { | |||
There was a problem hiding this comment.
Why does this module (template) need to be edm::one? Could it be edm::stream?
If it needs to stay edm::one for some reason, please use also edm::LuminosityBlockCache extension to tell the framework the the module can process events from multiple LuminosityBlocks concurrently.
There was a problem hiding this comment.
Dear @makortel,
At the time of development I tried to use the edm::stream (as in SimpleFlatTableProducerBase) guided by the FWMultithreadedFrameworkStreamModuleInterface Twiki however, this resulted in a series of compilation errors that I was not able to resolve. Using edm::one solved the issues.
If you have any suggestions on how to make edm::stream work, I would be grateful.
Best,
Mateusz
There was a problem hiding this comment.
What kind of compilation errors did you get?
There was a problem hiding this comment.
The first one is: /afs/cern.ch/work/m/mzarucki/nanoAOD/CMSSW_12_0_0_pre3/src/PhysicsTools/NanoAOD/interface/SimpleFlatTableProducer.h:315:8: error: 'void SimpleFlatTableProducerBaseLumi<T, TProd>::endLuminosityBlockProduce(edm::LuminosityBlock&, const edm::EventSetup&) [with T = GenFilterInfo; TProd = GenFilterInfo]' marked 'final', but is not virtual
which can be solved by just removing the final specifier for the endLuminosityBlockProduce method.
A following compilation error is:
/cvmfs/cms.cern.ch/slc7_amd64_gcc900/cms/cmssw/CMSSW_12_0_0_pre3/src/FWCore/Framework/interface/stream/callAbilities.h:455:43: error: 'globalEndLuminosityBlockProduce' is not a member of 'LumiSingletonSimpleFlatTableProducer<GenFilterInfo>' 455 | T::globalEndLuminosityBlockProduce(Lumi, iES, iRC);
There was a problem hiding this comment.
My bad, after further thought stream module type actually wouldn't help. I was thinking we could use the "streams" to work around the thread-unsafety of StringObjectFunction and StringCutObjectSelector, but the LuminosityBlock transitions of stream modules are global (i.e. can occur concurrently).
So the best way to support "processing Events from multiple LuminosityBlocks concurrently" would be to use the edm::LuminosityBlockCache extension. Since you don't really need the cache, you could declare it as e.g. edm::LuminosityBlockCache<int> and in globalBeginLuminosityBlock() return nullptr.
There was a problem hiding this comment.
Dear @makortel,
Therefore, is edm:one fine to use for this module?
As suggested, I have added the edm:LuminosityBlockCache extension as follows:
class SimpleFlatTableProducerBaseLumi : public edm::one::EDProducer<edm::EndLuminosityBlockProducer, edm::LuminosityBlockCache<int>> {
I have also added the globalBeginLuminosityBlock method which returns nullptr.
Unfortunately, I get the following compilation errors:
from /afs/cern.ch/work/m/mzarucki/nanoAOD/CMSSW_12_0_0_pre3/src/PhysicsTools/NanoAOD/plugins/SimpleFlatTableProducerPlugins.cc:1:
/cvmfs/cms.cern.ch/slc7_amd64_gcc900/external/gcc/9.3.0/include/c++/9.3.0/bits/unique_ptr.h: In instantiation of 'typename std::_MakeUniq<_Tp>::__single_object std::make_unique(_Args&& ...) [with _Tp = LumiSingletonSimpleFlatTableProducer<GenFilterInfo>; _Args = {const edm::ParameterSet&}; typename std::_MakeUniq<_Tp>::__single_object = std::unique_ptr<LumiSingletonSimpleFlatTableProducer<GenFilterInfo>, std::default_delete<LumiSingletonSimpleFlatTableProducer<GenFilterInfo> > >]':
/cvmfs/cms.cern.ch/slc7_amd64_gcc900/cms/cmssw/CMSSW_12_0_0_pre3/src/FWCore/Framework/src/MakeModuleHelper.h:39:40: required from 'static std::unique_ptr<_Tp> edm::MakeModuleHelper<Base>::makeModule(const edm::ParameterSet&) [with T = LumiSingletonSimpleFlatTableProducer<GenFilterInfo>; Base = edm::one::EDProducerBase]'
/cvmfs/cms.cern.ch/slc7_amd64_gcc900/cms/cmssw/CMSSW_12_0_0_pre3/src/FWCore/Framework/src/WorkerMaker.h:83:107: required from 'std::shared_ptr<edm::maker::ModuleHolder> edm::WorkerMaker<T>::makeModule(const edm::ParameterSet&) const [with T = LumiSingletonSimpleFlatTableProducer<GenFilterInfo>]'
/cvmfs/cms.cern.ch/slc7_amd64_gcc900/cms/cmssw/CMSSW_12_0_0_pre3/src/FWCore/Framework/src/WorkerMaker.h:77:40: required from here
/cvmfs/cms.cern.ch/slc7_amd64_gcc900/external/gcc/9.3.0/include/c++/9.3.0/bits/unique_ptr.h:857:30: error: invalid new-expression of abstract class type 'LumiSingletonSimpleFlatTableProducer<GenFilterInfo>'
857 | { return unique_ptr<_Tp>(new _Tp(std::forward<_Args>(__args)...)); }
| ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
Is there something obvious that I might be missing here?
Best regards,
Mateusz
There was a problem hiding this comment.
The use of LuminosityBlockCache requires overriding also void SimpleFlatTableProducerBaseLumi::globalEndLuminosityBlock(edm::LuminosityBlock const&, edm::EventSetup const&) override {}.
(sorry for not mentioning it earlier)
https://twiki.cern.ch/twiki/bin/view/CMSPublic/FWMultithreadedFrameworkOneModuleInterface#edm_LuminosityBlockCacheT
There was a problem hiding this comment.
Thanks for the clarification!
I have made the discussed updates to support processing events from multiple LuminosityBlocks concurrently in a4a7f44
I would be grateful if you could have a look whether this is what you were expecting.
Cheers,
Mateusz
There was a problem hiding this comment.
I would be grateful if you could have a look whether this is what you were expecting.
Yes, it looks good now. Thanks!
|
in addition to makortel comments please run the following
|
|
+code-checks Logs: https://cmssdt.cern.ch/SDT/code-checks/cms-sw-PR-34169/23395
|
|
A new Pull Request was created by @mzarucki (Mateusz Zarucki) for master. It involves the following packages: PhysicsTools/NanoAOD @cmsbuild, @mariadalfonso, @gouskos, @fgolf can you please review it and eventually sign? Thanks. cms-bot commands are listed here |
|
please test |
|
+1 Summary: https://cmssdt.cern.ch/SDT/jenkins-artifacts/pull-request-integration/PR-84e32b/16108/summary.html Comparison SummarySummary:
|
cmsbuild
added
code-checks-pending
tests-pending
and removed
code-checks-approved
labels
|
Dear @kdlong, @mariadalfonso, I have added the I ran the workflows: Best regards, |
|
+code-checks Logs: https://cmssdt.cern.ch/SDT/code-checks/cms-sw-PR-34169/27267
|
|
Pull request #34169 was updated. @cmsbuild, @mariadalfonso, @gouskos, @fgolf can you please check and sign again. |
|
please test workflow 546,547,548 |
|
+1 Summary: https://cmssdt.cern.ch/SDT/jenkins-artifacts/pull-request-integration/PR-84e32b/21129/summary.html Comparison Summary@slava77 comparisons for the following workflows were not done due to missing matrix map:
Summary:
|
|
+xpog
|
|
This pull request is fully signed and it will be integrated in one of the next master IBs (tests are also fine). This pull request will now be reviewed by the release team before it's merged. @perrotta, @dpiparo, @qliphy (and backports should be raised in the release meeting by the corresponding L2) |
cmsbuild
added
fully-signed
xpog-approved
and removed
pending-signatures
xpog-pending
labels
|
+1
|
PR description:
SimpleFlatTableProducerBaseLumi(edm::one::EDProducer) that stores the generator filter information in a flat tableLuminosityBlockstree in miniAODnumEventsTotal,numEventsPassed,filterEfficiency,filterEfficiencyErrorgenFilterTableis added to theglobalTablesMCsequence in globals_cff.pynanoaod::FlatTableis written into theLuminosityBlockstree as new branches via the addedLumiOutputBranchestypePR validation:
The code has been tested in CMSSW_12_0_0_pre1, running a standard nanoAOD workflow on a miniAOD signal MC sample with a generator filter (SMS-T2tt_dM-10to80_genHT-160_genMET-80_mWMin-0p1), over several thousand events (several LS). The output root file correctly has the requested generator filter information stored.
Following rebasing on the latest master branch, the code has passes the nanoAOD unit tests that run over data (
runTheMatrix.py -l 136.8523) in the latest IB CMSSW_12_0_X_2021-06-17-1100. The MC equivalent (runTheMatrix.py -l 1325.81) andscram b runtestsfail due to a DAS error not being able to access the file. I believe it would still important to inspect the output of the nanoAOD MC unit tests.