Hydjet Interface External #28764

wouf · 2020-01-20T16:31:41Z

PR description:

Hydjet inerface for using as external with cms-sw/cmsdist#5412

PR validation:

This is not final version, we are trying to solve #28636

cmsbuild · 2020-01-20T16:32:10Z

The code-checks are being triggered in jenkins.

cmsbuild · 2020-01-20T16:38:57Z

-code-checks

Logs: https://cmssdt.cern.ch/SDT/code-checks/cms-sw-PR-28764/13414

This PR adds an extra 32KB to repository

Code check has found code style and quality issues which could be resolved by applying following patch(s)

code-format:
https://cmssdt.cern.ch/SDT/code-checks/cms-sw-PR-28764/13414/code-format.patch
e.g. curl https://cmssdt.cern.ch/SDT/code-checks/cms-sw-PR-28764/13414/code-format.patch | patch -p1
You can also run scram build code-format to apply code format directly

cmsbuild · 2020-01-21T08:02:51Z

The code-checks are being triggered in jenkins.

cmsbuild · 2020-01-21T08:04:30Z

The code-checks are being triggered in jenkins.

cmsbuild · 2020-01-21T08:09:36Z

-code-checks

Logs: https://cmssdt.cern.ch/SDT/code-checks/cms-sw-PR-28764/13418

This PR adds an extra 32KB to repository

Code check has found code style and quality issues which could be resolved by applying following patch(s)

code-format:
https://cmssdt.cern.ch/SDT/code-checks/cms-sw-PR-28764/13418/code-format.patch
e.g. curl https://cmssdt.cern.ch/SDT/code-checks/cms-sw-PR-28764/13418/code-format.patch | patch -p1
You can also run scram build code-format to apply code format directly

cmsbuild · 2020-01-22T07:32:16Z

The code-checks are being triggered in jenkins.

cmsbuild · 2020-01-22T07:38:15Z

The code-checks are being triggered in jenkins.

cmsbuild · 2020-01-22T07:42:31Z

-code-checks

Logs: https://cmssdt.cern.ch/SDT/code-checks/cms-sw-PR-28764/13433

This PR adds an extra 32KB to repository

Code check has found code style and quality issues which could be resolved by applying following patch(s)

code-format:
https://cmssdt.cern.ch/SDT/code-checks/cms-sw-PR-28764/13433/code-format.patch
e.g. curl https://cmssdt.cern.ch/SDT/code-checks/cms-sw-PR-28764/13433/code-format.patch | patch -p1
You can also run scram build code-format to apply code format directly

cmsbuild · 2020-01-23T11:59:29Z

The tests are being triggered in jenkins.
Tested with other pull request(s) cms-sw/cmsdist#5412
Test Parameters:

MATRIX_EXTRAS = 150
https://cmssdt.cern.ch/jenkins/job/ib-run-pr-tests/4329/console Started: 2020/01/23 14:12

cmsbuild · 2020-01-23T14:46:38Z

+1
Tested at: a6eede4
https://cmssdt.cern.ch/SDT/jenkins-artifacts/pull-request-integration/PR-927a64/4329/summary.html
CMSSW: CMSSW_11_1_X_2020-01-23-1100
SCRAM_ARCH: slc7_amd64_gcc820

cmsbuild · 2020-01-23T14:46:47Z

Comparison job queued.

cmsbuild · 2020-01-23T16:14:19Z

Comparison is ready
https://cmssdt.cern.ch/SDT/jenkins-artifacts/pull-request-integration/PR-927a64/4329/summary.html

Comparison Summary:

No significant changes to the logs found
Reco comparison results: 0 differences found in the comparisons
DQMHistoTests: Total files compared: 34
DQMHistoTests: Total histograms compared: 2697090
DQMHistoTests: Total failures: 1
DQMHistoTests: Total nulls: 0
DQMHistoTests: Total successes: 2696743
DQMHistoTests: Total skipped: 346
DQMHistoTests: Total Missing objects: 0
DQMHistoSizes: Histogram memory added: 0.0 KiB( 33 files compared)
Checked 147 log files, 16 edm output root files, 34 DQM output files

silviodonato · 2020-01-28T15:25:14Z

Kind reminder to @alberto-sanchez @SiewYan @qliphy @efeyazgan @mkirsano @agrohsje

makortel · 2020-01-28T15:40:54Z

GeneratorInterface/HydjetInterface/interface/HydjetHadronizer.h

@@ -32,7 +36,7 @@ namespace HepMC {
 namespace gen {
  class Pythia6Service;

-  class HydjetHadronizer : public BaseHadronizer {
+  class HydjetHadronizer : public BaseHadronizer, public edm::EDConsumerBase {


This does not work as intended. The framework will not magically get the consumes() information by arbitrary components declaring to inherit from edm::EDConsumerBase. The edm::ConsumesCollector should be used instead to call consumes() (it needs to be percolated to the constructor of HydjectHadronizer from the constructors of all the EDModules that use it).

@makortel I assumed to use e.setConsumer(this) after getEDMEvent() or setEDMEvent() (which we have to add somewhere). In tests it looks worked, but I can be wrong...

That is not sufficient (anyway Event::setConsumer() mostly framework internal function). Framework needs to know the consumes information (=data dependencies) at the module construction time.

@makortel Isn't it?
Sorry if I'm being stupid, but could You please to explain Your suggestion to replace edm::EDConsumerBase by edm::ConsumesCollector? I mean how to implement it?

Isn't it?

No. The call to the EDConsumerBase constructor is far from sufficient. In short the framework asks from the EDConsumerBase objects it knows (=EDModules constructed by the framework) which products are consumed. The framework is not aware of arbitrary classes that decide to inherit from EDConsumerBase.

Sorry if I'm being stupid, but could You please to explain Your suggestion to replace edm::EDConsumerBase by edm::ConsumesCollector? I mean how to implement it?

See https://twiki.cern.ch/twiki/bin/view/CMSPublic/SWGuideEDMGetDataFromEvent#Consumes_and_Helpers

Essentially it should be something along the EDModule using HydjectHadronizer, e.g. edm::GeneratorFilter<HAD, DEC> doing

template <class HAD, class DEC> GeneratorFilter<HAD, DEC>::GeneratorFilter(ParameterSet const& ps) : EDFilter(), hadronizer_(ps, consumesCollector()), decayer_(nullptr), nEventsInLumiBlock_(0) {

and then HydjectHadronizer doing

HydjetHadronizer::HydjetHadronizer(const ParameterSet& pset, ConsumesCollector iC) ... { ... if (embedding_) { cflag_ = 0; src_ = iC.consumes<HepMCProduct>( pset.getUntrackedParameter<edm::InputTag>("backgroundLabel", edm::InputTag("generator", "unsmeared"))); } ... }

And yes, it does imply that all hadronizers that are used in conjunction with edm::GeneratorFilter<...> need to be modified to take edm::ConsumesCollector argument (that they can ignore).

@makortel Thanks! It works, but with an exception:

Exception Message: Unrunnable schedule Module run order problem found: generatorSmeared after VtxSmeared [path generation_step], VtxSmeared after mix [path generation_step], mix after randomEngineStateProducer [path generation_step], randomEngineStateProducer after generator [path generation_step], generator consumes generatorSmeared Running in the threaded framework would lead to indeterminate results. Please change order of modules in mentioned Path(s) to avoid inconsistent module ordering.

I think we have firstly to solve the issue.

It works,

Good.

I think we have firstly to solve the issue.

Correct.

Isn't it?

No. The call to the EDConsumerBase constructor is far from sufficient. In short the framework asks from the EDConsumerBase objects it knows (=EDModules constructed by the framework) which products are consumed. The framework is not aware of arbitrary classes that decide to inherit from EDConsumerBase.

Sorry if I'm being stupid, but could You please to explain Your suggestion to replace edm::EDConsumerBase by edm::ConsumesCollector? I mean how to implement it?

See https://twiki.cern.ch/twiki/bin/view/CMSPublic/SWGuideEDMGetDataFromEvent#Consumes_and_Helpers

Essentially it should be something along the EDModule using HydjectHadronizer, e.g. edm::GeneratorFilter<HAD, DEC> doing

template <class HAD, class DEC> GeneratorFilter<HAD, DEC>::GeneratorFilter(ParameterSet const& ps) : EDFilter(), hadronizer_(ps, consumesCollector()), decayer_(nullptr), nEventsInLumiBlock_(0) {

I changed all generators for this, but have a trouble with Pythia8:

'''

Compiling edm plugin /afs/cern.ch/work/w/wouf/HYDJET/1.9.1/EmbeddingProj/CMSSW_11_1_0_pre5/src/GeneratorInterface/Pythia8Interface/plugins/Pythia8Hadronizer.cc
In file included from /cvmfs/cms.cern.ch/slc7_amd64_gcc820/external/dire/2.004-bcolbf/include/Dire/SplitInfo.h:17,
from /cvmfs/cms.cern.ch/slc7_amd64_gcc820/external/dire/2.004-bcolbf/include/Dire/Splittings.h:15,
from /cvmfs/cms.cern.ch/slc7_amd64_gcc820/external/dire/2.004-bcolbf/include/Dire/SplittingLibrary.h:15,
from /cvmfs/cms.cern.ch/slc7_amd64_gcc820/external/dire/2.004-bcolbf/include/Dire/Dire.h:8,
from /afs/cern.ch/work/w/wouf/HYDJET/1.9.1/EmbeddingProj/CMSSW_11_1_0_pre5/src/GeneratorInterface/Pythia8Interface/plugins/Pythia8Hadronizer.cc:15:
/cvmfs/cms.cern.ch/slc7_amd64_gcc820/external/dire/2.004-bcolbf/include/Dire/DireBasics.h:108:7: warning: 'class Pythia8::DireFunction' has virtual functions and accessible non-virtual destructor [-Wnon-virtual-dtor]
class DireFunction {
^~~~~~~~~~~~
In file included from /afs/cern.ch/work/w/wouf/HYDJET/1.9.1/EmbeddingProj/CMSSW_11_1_0_pre5/src/GeneratorInterface/Pythia8Interface/plugins/Pythia8Hadronizer.cc:61:
/afs/cern.ch/work/w/wouf/HYDJET/1.9.1/EmbeddingProj/CMSSW_11_1_0_pre5/src/GeneratorInterface/Core/interface/ConcurrentHadronizerFilter.h: In instantiation of 'edm::gen::StreamCache<HAD, DEC>::StreamCache(const edm::ParameterSet&) [with HAD = Pythia8Hadronizer; DEC = gen::ConcurrentExternalDecayDriver]':
/cvmfs/cms.cern.ch/slc7_amd64_gcc820/external/gcc/8.2.0-pafccj/include/c++/8.3.1/bits/unique_ptr.h:831:30: required from 'typename std::_MakeUniq<_Tp>::__single_object std::make_unique(_Args&& ...) [with _Tp = edm::gen::StreamCache<Pythia8Hadronizer, gen::ConcurrentExternalDecayDriver>; _Args = {const edm::ParameterSet&}; typename std::_MakeUniq<_Tp>::_single_object = std::unique_ptr<edm::gen::StreamCache<Pythia8Hadronizer, gen::ConcurrentExternalDecayDriver>, std::default_delete<edm::gen::StreamCache<Pythia8Hadronizer, gen::ConcurrentExternalDecayDriver> > >]'
/afs/cern.ch/work/w/wouf/HYDJET/1.9.1/EmbeddingProj/CMSSW_11_1_0_pre5/src/GeneratorInterface/Core/interface/ConcurrentHadronizerFilter.h:176:62: required from 'std::unique_ptr<edm::gen::StreamCache<HAD, DEC> > edm::ConcurrentHadronizerFilter<HAD, DEC>::beginStream(edm::StreamID) const [with HAD = Pythia8Hadronizer; DEC = gen::ConcurrentExternalDecayDriver]'
/afs/cern.ch/work/w/wouf/HYDJET/1.9.1/EmbeddingProj/CMSSW_11_1_0_pre5/src/GeneratorInterface/Core/interface/ConcurrentHadronizerFilter.h:175:47: required from here
/afs/cern.ch/work/w/wouf/HYDJET/1.9.1/EmbeddingProj/CMSSW_11_1_0_pre5/src/GeneratorInterface/Core/interface/ConcurrentHadronizerFilter.h:70:65: error: no matching function for call to 'Pythia8Hadronizer::Pythia8Hadronizer()'
StreamCache(ParameterSet const& iPSet) : hadronizer{iPSet} {}
^
/afs/cern.ch/work/w/wouf/HYDJET/1.9.1/EmbeddingProj/CMSSW_11_1_0_pre5/src/GeneratorInterface/Pythia8Interface/plugins/Pythia8Hadronizer.cc:170:1: note: candidate: 'Pythia8Hadronizer::Pythia8Hadronizer(const edm::ParameterSet&, edm::ConsumesCollector&&)'
Pythia8Hadronizer::Pythia8Hadronizer(const edm::ParameterSet &params, edm::ConsumesCollector&& iC)
^~~~~~~~~~~~~~~~~
/afs/cern.ch/work/w/wouf/HYDJET/1.9.1/EmbeddingProj/CMSSW_11_1_0_pre5/src/GeneratorInterface/Pythia8Interface/plugins/Pythia8Hadronizer.cc:170:1: note: candidate expects 2 arguments, 1 provided
'''

@makortel @mkirsano Could You please help to fix it?

makortel · 2020-01-28T15:41:57Z

GeneratorInterface/HydjetInterface/src/HydjetHadronizer.cc

-    src_ = pset.getParameter<edm::InputTag>("backgroundLabel");
+  if (embedding_) {
+    cflag_ = 0;
+    src_ = mayConsume<HepMCProduct>(


Why mayConsume()? Below the HepMCProduct is read always when embedding_ is true, so this could be just consumes().

I see, thanks.

makortel · 2020-01-28T15:42:29Z

GeneratorInterface/HydjetInterface/src/HydjetHadronizer.cc

+  if (embedding_) {
+    cflag_ = 0;
+    src_ = mayConsume<HepMCProduct>(
+        pset.getUntrackedParameter<edm::InputTag>("backgroundLabel", edm::InputTag("generator", "unsmeared")));


Shouldn't this parameter stay as tracked? It affects physics and therefore should to be included in the event provenance.

@makortel It does not included to the gen fragment by default, so to avoid changing all configs I added default value. Anyway for current underlying sample it can't be various. In case of different label (then underlying sample has) the generation fails.

silviodonato · 2020-02-07T13:58:14Z

This PR is related to the issue #28636

silviodonato · 2020-05-26T12:33:12Z

@wouf after the long discussion on #28636, it seems you have all pieces to run hydjet as externals. Are you going to update this PR or do you prefer to open a new one?

wouf · 2020-05-27T07:13:01Z

@wouf after the long discussion on #28636, it seems you have all pieces to run hydjet as externals.

Not really, at this moment we are trying to solve issue with OscarMTProducer warnings:

%MSG-w SimG4CoreGenerator: in event :  OscarMTProducer:g4SimHits  23-Apr-2020 09:05:14 CEST Run: 1 Event: 10
10 a particle  pdgid= 2101 has status=2 but has no decay vertex, so will be fully tracked by Geant4
%MSG

which is coming from PYTHIA6's incorrect childes information in some cases. In that cases PYJETS common block contains:

#           1  PDG:        2212  st:        21  mother:           0  childes:            0 -           0
#           2  PDG:        2112  st:        21  mother:           0  childes:            0 -           0
#           3  PDG:          21  st:          21  mother:           1  childes:            0 -           0
#           4  PDG:          21  st:          21  mother:           2  childes:            0 -           0
#           5  PDG:          21  st:          21  mother:           3  childes:            0 -           0
#           6  PDG:          21  st:          21  mother:           4  childes:            0 -           0 
#           7  PDG:          21  st:          21  mother:           0  childes:            0 -           0 
#           8  PDG:          21  st:          21  mother:          0  childes:            0 -           0 
#           9  PDG:           2  st:          13  mother:           7  childes:    200100000 -           0
#          10  PDG:          21  st:          13  mother:           0  childes:    200110000 -   200090000
#          11  PDG:          21  st:          13  mother:           8  childes:    200120000 -   200100000
#          12  PDG:          21  st:          13  mother:           8  childes:    200130000 -   200110000

At this moment we are trying to work around.

Are you going to update this PR or do you prefer to open a new one?

I think the new one would be better.

silviodonato · 2020-06-01T13:56:31Z

I think the new one would be better.
Let me close this PR then. If/when you want to reopen it, just let me know and I will do it.

wouf · 2020-06-10T21:54:56Z

Thank You @silviodonato , all!
The new PR in #30185

Hydjet Interface External

a0ac8f5

cmsbuild added this to the CMSSW_11_1_X milestone Jan 20, 2020

cmsbuild added code-checks-pending comparison-pending generators-pending orp-pending pending-signatures tests-pending labels Jan 20, 2020

wouf mentioned this pull request Jan 20, 2020

Access edm::event from GeneratorInterface #28636

Closed

cmsbuild added code-checks-rejected and removed code-checks-pending labels Jan 20, 2020

Update HydjetHadronizer.h

8604cd6

cmsbuild added code-checks-pending and removed code-checks-rejected labels Jan 21, 2020

Update HydjetHadronizer.cc

e57e5eb

cmsbuild added code-checks-rejected and removed code-checks-pending labels Jan 21, 2020

Update HydjetHadronizer.cc

6dd97d1

cmsbuild added code-checks-pending and removed code-checks-rejected labels Jan 22, 2020

Update HydjetAnalyzer.cc

366de12

cmsbuild removed the code-checks-pending label Jan 22, 2020

cmsbuild added comparison-pending tests-pending and removed comparison-notrun tests-rejected labels Jan 23, 2020

cmsbuild mentioned this pull request Jan 23, 2020

Hydjet integration as external cms-sw/cmsdist#5412

Merged

cmsbuild added requires-external tests-started and removed tests-pending labels Jan 23, 2020

cmsbuild added tests-approved and removed tests-started labels Jan 23, 2020

cmsbuild added comparison-available and removed comparison-pending labels Jan 23, 2020

makortel reviewed Jan 28, 2020

View reviewed changes

This was referenced May 5, 2020

Clean up BuildFiles under GeneratorInterface/ #29656

Merged

Restructured code of againstElectronDeadECAL tauID #29627

Merged

cmsbuild mentioned this pull request May 18, 2020

Apply existing clang-tidy checks #29866

Closed

smuzaffar modified the milestones: CMSSW_11_1_X, CMSSW_11_2_X May 22, 2020

silviodonato closed this Jun 1, 2020

wouf mentioned this pull request Jun 10, 2020

HYDJET, PYQUEN and the new embedding #30185

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Hydjet Interface External #28764

Hydjet Interface External #28764

wouf commented Jan 20, 2020

cmsbuild commented Jan 20, 2020

cmsbuild commented Jan 20, 2020

cmsbuild commented Jan 21, 2020

cmsbuild commented Jan 21, 2020

cmsbuild commented Jan 21, 2020

cmsbuild commented Jan 22, 2020

cmsbuild commented Jan 22, 2020

cmsbuild commented Jan 22, 2020

cmsbuild commented Jan 23, 2020 •

edited

cmsbuild commented Jan 23, 2020

cmsbuild commented Jan 23, 2020

cmsbuild commented Jan 23, 2020

silviodonato commented Jan 28, 2020

makortel Jan 28, 2020

wouf Jan 28, 2020 •

edited

makortel Jan 28, 2020

wouf Jan 29, 2020

makortel Jan 29, 2020

wouf Jan 31, 2020 •

edited

makortel Jan 31, 2020

wouf Apr 15, 2020

makortel Jan 28, 2020

wouf Jan 28, 2020

makortel Jan 28, 2020

wouf Jan 28, 2020

silviodonato commented Feb 7, 2020

silviodonato commented May 26, 2020

wouf commented May 27, 2020

silviodonato commented Jun 1, 2020

wouf commented Jun 10, 2020

Hydjet Interface External #28764

Hydjet Interface External #28764

Conversation

wouf commented Jan 20, 2020

PR description:

PR validation:

cmsbuild commented Jan 20, 2020

cmsbuild commented Jan 20, 2020

cmsbuild commented Jan 21, 2020

cmsbuild commented Jan 21, 2020

cmsbuild commented Jan 21, 2020

cmsbuild commented Jan 22, 2020

cmsbuild commented Jan 22, 2020

cmsbuild commented Jan 22, 2020

cmsbuild commented Jan 23, 2020 • edited

cmsbuild commented Jan 23, 2020

cmsbuild commented Jan 23, 2020

cmsbuild commented Jan 23, 2020

silviodonato commented Jan 28, 2020

Choose a reason for hiding this comment

wouf Jan 28, 2020 • edited

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

wouf Jan 31, 2020 • edited

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

silviodonato commented Feb 7, 2020

silviodonato commented May 26, 2020

wouf commented May 27, 2020

silviodonato commented Jun 1, 2020

wouf commented Jun 10, 2020

cmsbuild commented Jan 23, 2020 •

edited

wouf Jan 28, 2020 •

edited

wouf Jan 31, 2020 •

edited