Differences between mkFit standalone validation vs MTV

[kmcdermo]

Main Issue

This has been an open issue for awhile now, and I wanted to make one place where we have some plots for reference.

The main hypothesis is that we are losing on short tracks with non-positive definite covariance matrices. Namely, the loss in efficiency from mkFit tracks compared to CMSSW tracks as seen in MTV is due to these tracks being dropped in the producer that interfaces between mkFit output and MTV input. These wacky covariances come from the backward fit within mkFit, and in particular seem to affect shorter tracks greater than longer ones.

The MTV results are in stark contrast to our standalone validation in which we see the near identical performance above pT > 0.9 GeV between mkFit and CMSSW and significantly better performance in mkFit compared to CMSSW in the barrel for tracks pT > 0 GeV.

To illustrate this hypothesis, I shamelessly am stealing some slides from Allie's talk for CMS Week that highlight the differences in definitions between mkFit standalone validation and MTV, as well as some plots of efficiency vs eta and pT. In addition, I have attached the efficiency vs number of layers from MTV from Giuseppe.

Varying Requirements with mkFit Validation

I also made some plots varying the definition of matching and good tracks. Keeping the definition of hit matching the same, the results are below:

• Simval using the defaults: 50 percent matching after seed, min 12 sim layers, min 10 reco layers
• Simval for setting the minimum sim layers to 5 and minimum reco layers to 5

I also changed the definition of the hit matching from 50% after the seed, to 75% including the seed (to better approximate MTV), with the results below:

• min 12 sim layers, 10 reco layers
• min 5 sim layers, 5 reco layers

I made some very quick slides demonstrating the different minimum layer requirements and hit matching schemes. As can be seen, we definitely do not do as well with lower layer requirements on tracks, as well as with the 75% matching criteria. This is most notable at low pT, but definitely affects the full pT and eta spectrum.

Main takeaways

• mkFit struggles to find short sim tracks (i.e. tracks with ~10 layers including the seed)
• mkFit tends to add bad hits for longer tracks, as seen in worsening efficiency with the 75% matching threshold with all reco hits

Proposed studies to determine where mkFit starts to fall off w.r.t. CMSSW

• Dedicated study scanning the efficiency by varying the hit matching threshold with and without the seed
• Dedicated study scanning the efficiency by varying the nLayers requirements for sim and reco tracks

Recipes

To produce the SimVal plots like those above for quick comparisons, simply run the validation script as normal (using the forConf parameter):

./val_scripts/validation-cmssw-benchmarks.sh forConf

You can drop the CMSSWVal to save on time by and then drop the irrelevant directories and plots in the web scripts by using the diffs below:

• Only SimVal is run
• Only collect SimVal plots

Below is a list of diffs used to make changes to minimum layer requirements and hit matching:

• Change the number of minimum sim layers needed (12->5)
• Change the number of minimum reco layers needed (10->5)
• Change the definition of the hit matching to be 75% with the seed included

All of the diffs are .txt files (but really should .patch files, but GH Markdown won't let me upload that extension). To apply a patch file directly, do:

git patch <name_of_text_file>

[kmcdermo]

This issue has been addressed by @cerati with MTV-like validation in PR #204, and adding nLayers eff in PR #211. Closing.