EGM GED bug fixes for pre12 (AOD size, cluster double counting, rechit fraction usage)

[lgray]

Built ontop of pre11.

This PR includes various bugfixes and improvements to AOD size:

• Fix to properly use rechit fractions for eta/phi width
• Fix to produce BasicCluster collections for PF-SCs (instead of ref'ing back to particleFlowClusterECAL)
• Fix photon pixel seed veto not being properly filled.
• Fix cut on photon SC eT to be consistent across all GED photon producers.
• Fix instance of cluster double counting that could happen in PFEGammaAlgo.
• Consistent renaming of SuperCluster accessors.
• Cut to reduce PF-ECAL SuperCluster collection.
• Reduced size to interesting rechit collections for PF-ECAL SCs
• Import PFClusters of PFSCs by edm::Ref instead of by fraction and overlap

[cmsbuild]

A new Pull Request was created by @lgray (Lindsey Gray) for CMSSW_7_0_X.

EGM GED bug fixes for pre12 (AOD size, cluster double counting, rechit fraction usage)

It involves the following packages:

DQMOffline/EGamma
DataFormats/EgammaCandidates
DataFormats/EgammaReco
DataFormats/ParticleFlowCandidate
DataFormats/PatCandidates
RecoEcal/Configuration
RecoEcal/EgammaClusterAlgos
RecoEcal/EgammaClusterProducers
RecoEcal/EgammaCoreTools
RecoEgamma/Configuration
RecoEgamma/EgammaElectronAlgos
RecoEgamma/EgammaElectronProducers
RecoEgamma/EgammaPhotonAlgos
RecoEgamma/EgammaPhotonProducers
RecoEgamma/EgammaTools
RecoEgamma/Examples
RecoParticleFlow/Configuration
RecoParticleFlow/PFClusterProducer
RecoParticleFlow/PFClusterTools
RecoParticleFlow/PFProducer
RecoTauTag/RecoTau
TauAnalysis/MCEmbeddingTools
Validation/RecoEgamma

@civanch, @thspeer, @danduggan, @mdhildreth, @cmsbuild, @anton-a, @nclopezo, @rovere, @deguio, @slava77, @vadler, @Degano, @ojeda can you please review it and eventually sign? Thanks.
@richard-cms this is something you requested to watch as well.
You can sign-off by replying to this message having '+1' in the first line of your reply.
You can reject by replying to this message having '-1' in the first line of your reply.
@ktf you are the release manager for this.

[slava77]

There is a problem running unit tests (various PAT tests, in fact)

Running script: $CMSSW_BASE/src/PhysicsTools/PatAlgos/test/runtests.sh
globaltag = START70_V4::All
11-Jan-2014 00:22:38 CET  Initiating request to open file root://eoscms//eos/cms/store/relval/CMSSW_7_0_0_pre7/RelValProdTTbar/AODSIM/PRE_ST62_V8-v2/00000/863E36FF-2046-E311-9526-003048678C26.root?svcClass=default
11-Jan-2014 00:22:40 CET  Successfully opened file root://eoscms//eos/cms/store/relval/CMSSW_7_0_0_pre7/RelValProdTTbar/AODSIM/PRE_ST62_V8-v2/00000/863E36FF-2046-E311-9526-003048678C26.root?svcClass=default
Begin processing the 1st record. Run 1, Event 1, LumiSection 1 at 11-Jan-2014 00:22:55.156 CET
Begin processing the 2nd record. Run 1, Event 2, LumiSection 1 at 11-Jan-2014 00:23:32.180 CET
----- Begin Fatal Exception 11-Jan-2014 00:23:32 CET-----------------------
An exception of category 'InvalidReference' occurred while
   [0] Processing run: 1 lumi: 1 event: 2
   [1] Running path 'outpath'
   [2] Calling event method for module PoolOutputModule/'out'
   [3] Calling produce method for unscheduled module PATElectronSelector/'selectedPatElectrons'
   [4] Calling produce method for unscheduled module PATElectronProducer/'patElectrons'
Exception Message:
BadRefCore RefCore: Request to resolve a null or invalid reference to a product of type 'std::vector<reco::SuperCluster>' has been detected.
Please modify the calling code to test validity before dereferencing.
----- End Fatal Exception -------------------------------------------------

[bendavid]

Hi Slava,
These PAT tests take as input some semi-random old relval files from here
https://github.com/cms-sw/cmssw/blob/CMSSW_7_0_X/PhysicsTools/PatAlgos/python/patInputFiles_cff.py

I confirm that producing a new ttbar aod file with this branch and running the test on top of that, things work fine.
(produced with runTheMatrix.py -l 2.0 --useInput=all -e --command="-n 100")

(Does this test even run out of the box in pre11? Because I would have guessed that the change which broke compatibility with the old 700pre7 input files specified here was actually the switch to the new ged sequence as default in pre11)

[cmsbuild]

Pull request #1986 was updated. @civanch, @thspeer, @danduggan, @mdhildreth, @cmsbuild, @anton-a, @nclopezo, @rovere, @deguio, @slava77, @vadler, @Degano, @ojeda can you please check and sign again.

[civanch]

+1

[vadler]

+1

[slava77]

working on it (second iteration)

[cmsbuild]

+1
https://cmssdt.cern.ch/SDT/jenkins-artifacts/pull-request-integration/PR-1986/2076/summary.html

[slava77]

From the last changes (b8e96e0..4100617) , photons are changed the most, as expected, around 10 GeV:
(these are expected)

particleFlow is quite resilient here

[slava77]

There are more messages of the following kind

%MSG-w PFEGammaAlgo::mergeROsByAnyLink:  PFEGammaProducer:particleFlowEGamma  14-Jan-2014 22:21:51 CET Run: 1 Event: 251
Need to implement proper merging of two gsf candidates!

Based on the increase of the final objects, I'm assuming these are happening more often as expected.
What are the plans to fix this thing?

[lgray]

Hi Slava,

Fixing this particular warning requires some fairly careful physics studies in the POGs/PAGs, close-by electrons and early conversions are pretty difficult to disentangle correctly.

If it is too annoying, we can demote it to a LogInfo in another PR to avoid re-signing for a one line change.

[slava77]

Hi Lindsey,

I'm just taking this as an opportunity to remind that it's there ;)

[lgray]

Then it is doing its job! :-)

[slava77]

this is

Vector deltaR = new_sc.position()-beamSpot_->position();
float et = new_sc.energy()/deltaR.mag()*deltaR.perp();

should be quite a bit lighter than conversion to eta and back

[slava77]

Just to note, I'm expecting a follow up on the code review in a separate PR, as discussed

[deguio]

+1

[slava77]

+1

tested 4100617 in CMSSW_7_0_X_2014-01-10-1400 (sign293a test area)

I'll post some more notes later.
Changes in physics as expected.
AOD event content expansion made in pre11 is now recovered

[cmsbuild]

This pull request is fully signed and it will be integrated in one of the next IBs unless changes (tests are also fine). @ktf can you please take care of it?

[slava77]

This PR is for 70X
@davidlange6 , I'm guessing, you will need to +1 the ORP signature

[slava77]

Here are some more notes for the record:

AOD size, only RECO process in ttbarPU 202.0 (200 events). Numbers are bytes per event

• gone: recoPFClusters_particleFlowClusterPS__RECO. 54121.7
• gone: recoPFClusters_particleFlowClusterECAL__RECO. 46983.8
• gone: recoPhotons_mustachePhotons__RECO. 487.585
• gone: recoPhotonCores_mustachePhotonCore__RECO. 42.535
• new: recoCaloClusters_particleFlowSuperClusterECAL_particleFlowBasicClusterECALBarrel_RECO. 2573.01
• new: recoCaloClusters_particleFlowSuperClusterECAL_particleFlowBasicClusterECALPreshower_RECO. 2259.61
• new: recoCaloClusters_particleFlowEGamma_EBEEClusters_RECO. 1540.79
• new: recoCaloClusters_particleFlowSuperClusterECAL_particleFlowBasicClusterECALEndcap_RECO. 1272.76
• new: recoCaloClusters_particleFlowEGamma_ESClusters_RECO. 1003.67
• Total went from 448 to 355 kB/evt (added 9kB, removed 102kB)

Timing (look at above 1s/job, or 5ms/evt, 200 events):

• ~ x2.5 decrease in time of conversionTrackCandidates (30 sec/job now)
• ~ x2 increase in time for gedPhotons (3sec/job now)

General on physics changes:

• photon yields increased around 10 GeV (fixed up thresholds)
• slight increase in passing electrons, in particular in background samples, coincides with MVA becoming more signal-like (this is from the fixed cluster shapes)

Some notable plots
TTbarPU

• more signal-like MVA (not significant though)
  
• cluster sizes reduced
  
  
  

Ele 35 response slightly decreased (less corrections, since cluster shapes are now changed). The effect for the bulk is pretty tiny.

[bendavid]

Hi Slava,
A few points FYI.

The decrease in timing for the conversions is due to the fact that the ecal seeding for the conversion tracks is now only running on the reduced collection of clusters from the particleFlowSuperClusterECAL producer rather than the full PFCluster collection.

I'm not sure the x2 increase in gedPhotons is fully expected, but given that the total time is small, this can probably be accounted for by the small effective change in threshold by cutting on the parentSCluster Et rather than the (non-corrected at this stage) refined supercluster Et.

If the foreseen changes and cleanup happen for the refined SuperCluster in 71x, the size of the two collections
new: recoCaloClusters_particleFlowEGamma_EBEEClusters_RECO. 1540.79
new: recoCaloClusters_particleFlowEGamma_ESClusters_RECO. 1003.67

can be expected to become probably an order of magnitude smaller, since they will contain only the relatively small number of CaloClusters added in addition to what is already in the mustache SC. (Whereas for 70x these are simply being duplicated between the two collections as has been discussed)