re-check GenJet clustering for GenHFHadronMatcher

[jpata]

making an issue so I know to solve GenJet stuff from #67 which is closed.

from Andrea:
those are the open points in my view:

we should not remake the gen jets as they should be the same but I understand that for the hadron matching this could be actually needed? can you confirm?
in output we should have a single gen jet having all needed features, namely:
the standard flavour coming from PAT
the hadron matched flavour
the genNu correction
if hadron matcher needs to remake its own jets, then we have to back match those to the original collection and just set a had-flavour attribute

With spring15, the genjets still need to be re-clustered. As far as I know, the original authors (Johannes Hauk, @bartonaz) are taking a look since June.

This is the error we get halfway through the processing.

Begin processing the 171st record. Run 1, Event 2982953, LumiSection 29870 at 28-Aug-2015 11:20:24.060 CEST
----- Begin Fatal Exception 28-Aug-2015 11:20:24 CEST-----------------------
An exception of category 'ProductNotFound' occurred while
   [0] Processing run: 1 lumi: 29870 event: 2982953
   [1] Running path 'endpath'
   [2] Calling event method for module PoolOutputModule/'OUT'
   [3] Calling produce method for unscheduled module GenHFHadronMatcher/'matchGenBHadron'
   [4] Calling produce method for unscheduled module JetFlavourClustering/'genJetFlavourPlusLeptonInfos'
Exception Message:
InvalidID get by product ID: invalid ProductID supplied
   Additional Info:
      [a] If you wish to continue processing events after a ProductNotFound exception,
add "SkipEvent = cms.untracked.vstring('ProductNotFound')" to the "options" PSet in the configuration.

----- End Fatal Exception -------------------------------------------------

[arizzi]

that's the error you get if you do not recluster or what? do you have an example producing such error to look at?

[jpata]

Exactly, this is without reclustering:

to reproduce, change this to "slimmedGenJets"
https://github.com/vhbb/cmssw/blob/vhbbHeppy74X/VHbbAnalysis/Heppy/test/combined_cmssw.py#L393

also needs to be changed here. i'll make it configurable in a PR which fixes GenJet stuff: https://github.com/vhbb/cmssw/blob/vhbbHeppy74X/VHbbAnalysis/Heppy/python/GenHFHadronMatcher.py#L67

[bartonaz]

@jpata , sorry for a rather long silence from our side; we were very busy lately with many other things.
As you've probably seen yourself, the code still crashes with SPRING15 samples. Apparently, the issue with missing jet constituents was not fixed yet. We have brought this question again to the experts and hopefully will have some update on this soon.

The three points that you've raised as support for matching to the slimmedGenJets are not very clear to me. The standard flavour in PAT does not rely on genJets any more, therefore it is irrelevant now. The hadron-matched flavour is implemented in the JetFlavour tool, which is available independent of the actual genJet collection used. Finally, neutrinos indeed should be excluded from the reclustered genJets, which I have already implemented, and can submit as a PR in coming days. Since matching the reclustered jets to slimmedGenJets is rather tedious, I would prefer waiting a few days for an answer from the JetFlavour experts, and then decide what to do. In the meantime, reclustered genJets without neutrinos should be almost identical to the slimmedGenJets.

[jpata]

@bartonas, thanks for following up nevertheless and bringing it again to the experts. Could you point me again to the discussion about matching to slimmedGenJets and what you meant by "hadron-matched flavour is implemented in the JetFlavour tool"? I honestly no longer remember any such discussion...

[bartonaz]

@jpata, I was refering to your first post in this issue discussion. The hadron-matched jet flavour is the standard definition implemented in the JetFlavour tool, which is then used internally by the GenHFHadronMatcher. This is exactly what we are reproducing by reclustering jets, and running the JetFlavour code on them. Thus, jets and their flavour are in principle identical to the ones stored in miniAOD. I will perform additional checks to see how large are differences between reclustered jets and jets in miniAOD.

[jpata]

@bartonaz ah OK, thanks for the clarification. If the re-clustered jets are identical, then this matching is only for our convenience not to store gen-jets twice and still keep the original miniAOD definition of genjets. In short I'd simply like to fix what will be the GenHFHadronMatcher recipe: re-clustered jets or not, but I suppose we are still waiting for feedback from the experts.

[jpata]

OK, so from what I understand we will need to re-cluster and we are safe in matching original slimmedGenJets to reclustered until cms-sw#11081 is merged.

@arizzi: do you prefer to pull this into vhbb/cmssw or go with re-clustering?

Below a mail from @bartonaz describing the situation.

Thanks all.

1. The missing slimmedGenJets constituents are present intentionally and will not be removed in future.
2. With support from Andrea Rizzi and Dinko Ferencek I have implemented the protection against missing jet constituents, which allows to use slimmedGenJets in the JetFlavour tool, and consequently in the GenHFHadronMatcher.
3. Tests showed that the difference between slimmedGenJets and reclustered jets is very minor, and only slightly affects jet-hadron matching, mainly for jets with Pt < b-hadron Pt, which should correspond to very low-Pt b-jets. 
4. Tests also showed that the final ttbar event categorisation is identical between the two jet collections. Therefore, it is safe to use slimmedGenJets directly.
5. Neutrinos were excluded from reclustered jets in AOD in order to match the ones in miniAOD.
6. All the above changes are already implemented and a corresponding pull request is already submitted.
7. The plugin for ttbar event categorisation, implemented privately by Johannes, was organised now as part of the TopQuarkAnalysis/TopTools module. The corresponding pull request is awaiting merging.

Once all the pull requests are merged and available in a new release, the TWiki will updated correspondingly.


the protections were implemented directly in the code of the JetFlavour tool, which performs the ghost-hadron injection, and is used internally in the GenHFHadronMatcher.
Since this is just code running on existing jets, the fix will work with existing miniAOD samples right after the pull request is merged. There is no need to wait for a new round of miniAOD production.

[arizzi]

I'd say to fix the hadron flavour importing the 11081 so that we run faster (one clustering step less).
In any case, next miniAOD central reprocessing should have the hadronFlavour precomputed (so let's import from 11081 only the c++ fix of checking if the reference isAvailable)

[jpata]

OK, I'll make a single PR containing the isAvailable fix and updated vhbb-side hadron matching/gen-jet code against vhbb today.