Activate IrradiationBiasCorrection in Pixel CPE generic for Run3

[mmusich]

PR description:

In this PR the IrradiationBiasCorrection parameter for the Pixel CPE Generic algorithm is activated for the Run 3 era.
The motivation for activating the correction is due to the expected comparatively large loss of resolution along the local y [global-z in barrel] direction for an heavily irradiated detector (as expected for the end of run3 and extended end or run3 simulation scenarios), in which carrier trapping significantly reduces Lorentz drift induced charge sharing.
As shown in this presentation, without this correction most clusters are reconstructed at ±100-200μm from true y, caused by charge loss from the backplane side.

As pointed out in the review (see #27183 (comment)) the Pixel CPE conditions need to be updated to take into account the parameter is switched on.
The phase1_2021_realistic and phase1_2021_cosmics are updated accordingly.
The diff of the two Global Tags is the following:

$ conddb diff 106X_upgrade2021_realistic_Candidate_2019_06_17_18_50_50 106X_upgrade2021_realistic_v5
[2019-06-17 21:26:38,571] INFO: Connecting to pro [frontier://PromptProd/CMS_CONDITIONS]
Record                        Label        pro::106X_upgrade2021_realistic_Candidate_2019_06_17_18_50_50 Tag  pro::106X_upgrade2021_realistic_v5 Tag                          
----------------------------  -----------  -----------------------------------------------------------------  --------------------------------------------------------------  
SiPixel2DTemplateDBObjectRcd  denominator  SiPixel2DTemplateDBObject_phase1_BoR3_HV350_Tr1300_den             SiPixel2DTemplateDBObject_phase1_38T_mc_v4_den_unirradiated     
SiPixel2DTemplateDBObjectRcd  numerator    SiPixel2DTemplateDBObject_phase1_BoR3_HV350_Tr1300_num             SiPixel2DTemplateDBObject_phase1_38T_2018_ultralegacymc_v1_num  
SiPixelGenErrorDBObjectRcd    -            SiPixelGenErrorDBObject_phase1_BoR3_HV350_Tr1300                   SiPixelGenErrorDBObject_phase1_38T_2018_ultralegacymc_v1        
SiPixelLorentzAngleRcd        -            SiPixelLorentzAngle_phase1_BoR3_HV350_Tr1300                       SiPixelLorentzAngle_phase1_2018_ultralegacymc_v1                
SiPixelLorentzAngleSimRcd     -            SiPixelLorentzAngleSim_phase1_BoR3_HV350_Tr1300                    SiPixelLorentzAngleSim_phase1_mc_v4                             
SiPixelTemplateDBObjectRcd    -            SiPixelTemplateDBObject_phase1_BoR3_HV350_Tr1300                   SiPixelTemplateDBObject_phase1_38T_2018_ultralegacymc_v1      

$ conddb diff 106X_upgrade2021cosmics_realistic_deco_Candidate_2019_06_17_19_08_59 106X_upgrade2021cosmics_realistic_deco_v2[2019-06-17 21:27:39,951] INFO: Connecting to pro [frontier://PromptProd/CMS_CONDITIONS]
Record                        Label        pro::106X_upgrade2021cosmics_realistic_deco_Candidate_2019_06_17_19_08_59 Tag  pro::106X_upgrade2021cosmics_realistic_deco_v2 Tag              
----------------------------  -----------  -----------------------------------------------------------------------------  --------------------------------------------------------------  
SiPixel2DTemplateDBObjectRcd  denominator  SiPixel2DTemplateDBObject_phase1_BoR3_HV350_Tr1300_den                         SiPixel2DTemplateDBObject_phase1_38T_mc_v4_den_unirradiated     
SiPixel2DTemplateDBObjectRcd  numerator    SiPixel2DTemplateDBObject_phase1_BoR3_HV350_Tr1300_num                         SiPixel2DTemplateDBObject_phase1_38T_2018_ultralegacymc_v1_num  
SiPixelGenErrorDBObjectRcd    -            SiPixelGenErrorDBObject_phase1_BoR3_HV350_Tr1300                               SiPixelGenErrorDBObject_phase1_38T_2018_ultralegacymc_v1        
SiPixelLorentzAngleRcd        -            SiPixelLorentzAngle_phase1_BoR3_HV350_Tr1300                                   SiPixelLorentzAngle_phase1_2018_ultralegacymc_v1                
SiPixelLorentzAngleSimRcd     -            SiPixelLorentzAngleSim_phase1_BoR3_HV350_Tr1300                                SiPixelLorentzAngleSim_phase1_mc_v4                             
SiPixelTemplateDBObjectRcd    -            SiPixelTemplateDBObject_phase1_BoR3_HV350_Tr1300                               SiPixelTemplateDBObject_phase1_38T_2018_ultralegacymc_v1   

PR validation:

Code compiles and the dumping of cmsDriver output configuration confirms the desired parameter value is assigned.
The activation of the parameter has been tested (together with the changes proposed in #27181) with:

cmsDriver.py -s GEN,SIM,DIGI,L1,DIGI2RAW,RAW2DIGI,RECO --evt_type SingleMuPt10_pythia8_cfi --fileout file:GENSIMRECO_MuPt10.root --python_filename=Run3_2023_generic_noIBC.py -n 10 --no_exec --era Run3 --beamspot Run3RoundOptics25ns13TeVHighSigmaZ --nThreads=4 —customise_commands="process.mix.digitizers.pixel.ThresholdInElectrons_BPix_L1 = cms.double(1300.0);  process.mix.digitizers.pixel.ThresholdInElectrons_BPix_L2 = cms.double(1600.0);process.mix.digitizers.pixel.ThresholdInElectrons_BPix = cms.double(1600.0); process.mix.digitizers.pixel.ThresholdInElectrons_FPix = cms.double(1600.0);process.PixelCPEGenericESProducer.IrradiationBiasCorrection = True”   --conditions 106X_mcRun3_2023_realistic_Candidate_2019_06_07_21_52_54 

There is not yet a physics output based validation (in progress), We will update the description once comparisons are available.
Physics validation has been shown in this presentation at the RECO / AT meeting of 14 Jun 2019 link.

if this PR is a backport please specify the original PR:

This PR is not a backport

cc: @tsusa @tvami @leaca @pmaksim1 @cmantill @OzAmram

[cmsbuild]

The code-checks are being triggered in jenkins.

[cmsbuild]

+code-checks

Logs: https://cmssdt.cern.ch/SDT/code-checks/cms-sw-PR-27183/10338

• This PR adds an extra 12KB to repository

[cmsbuild]

A new Pull Request was created by @mmusich (Marco Musich) for master.

It involves the following packages:

RecoLocalTracker/SiPixelRecHits

@perrotta, @cmsbuild, @slava77 can you please review it and eventually sign? Thanks.
@makortel, @felicepantaleo, @GiacomoSguazzoni, @rovere, @VinInn, @dkotlins, @gpetruc, @ebrondol, @threus this is something you requested to watch as well.
@davidlange6, @slava77, @fabiocos you are the release manager for this.

cms-bot commands are listed here

[perrotta]

please test

[cmsbuild]

The tests are being triggered in jenkins.
https://cmssdt.cern.ch/jenkins/job/ib-run-pr-tests/905/console Started: 2019/06/12 14:53

[cmsbuild]

-1

Tested at: 33eb134

You can see the results of the tests here:
https://cmssdt.cern.ch/SDT/jenkins-artifacts/pull-request-integration/PR-ef1fd2/905/summary.html

I found follow errors while testing this PR

Failed tests: RelVals

• RelVals:

When I ran the RelVals I found an error in the following workflows:
20034.0 step3

runTheMatrix-results/20034.0_TTbar_14TeV+TTbar_14TeV_TuneCUETP8M1_2023D17_GenSimHLBeamSpotFull14+DigiFullTrigger_2023D17+RecoFullGlobal_2023D17+HARVESTFullGlobal_2023D17/step3_TTbar_14TeV+TTbar_14TeV_TuneCUETP8M1_2023D17_GenSimHLBeamSpotFull14+DigiFullTrigger_2023D17+RecoFullGlobal_2023D17+HARVESTFullGlobal_2023D17.log

21234.0 step3

runTheMatrix-results/21234.0_TTbar_14TeV+TTbar_14TeV_TuneCUETP8M1_2023D21_GenSimHLBeamSpotFull14+DigiFullTrigger_2023D21+RecoFullGlobal_2023D21+HARVESTFullGlobal_2023D21/step3_TTbar_14TeV+TTbar_14TeV_TuneCUETP8M1_2023D21_GenSimHLBeamSpotFull14+DigiFullTrigger_2023D21+RecoFullGlobal_2023D21+HARVESTFullGlobal_2023D21.log

27434.0 step3

runTheMatrix-results/27434.0_TTbar_14TeV+TTbar_14TeV_TuneCUETP8M1_2023D35_GenSimHLBeamSpotFull14+DigiFullTrigger_2023D35+RecoFullGlobal_2023D35+HARVESTFullGlobal_2023D35/step3_TTbar_14TeV+TTbar_14TeV_TuneCUETP8M1_2023D35_GenSimHLBeamSpotFull14+DigiFullTrigger_2023D35+RecoFullGlobal_2023D35+HARVESTFullGlobal_2023D35.log

29034.0 step3

runTheMatrix-results/29034.0_TTbar_14TeV+TTbar_14TeV_TuneCUETP8M1_2023D41_GenSimHLBeamSpotFull14+DigiFullTrigger_2023D41+RecoFullGlobal_2023D41+HARVESTFullGlobal_2023D41/step3_TTbar_14TeV+TTbar_14TeV_TuneCUETP8M1_2023D41_GenSimHLBeamSpotFull14+DigiFullTrigger_2023D41+RecoFullGlobal_2023D41+HARVESTFullGlobal_2023D41.log

[cmsbuild]

Comparison not run due to runTheMatrix errors (RelVals and Igprof tests were also skipped)

[cmsbuild]

Comparison is ready
https://cmssdt.cern.ch/SDT/jenkins-artifacts/pull-request-integration/PR-ef1fd2/1004/summary.html

Comparison Summary:

• No significant changes to the logs found
• Reco comparison results: 2149 differences found in the comparisons
• DQMHistoTests: Total files compared: 33
• DQMHistoTests: Total histograms compared: 3254096
• DQMHistoTests: Total failures: 4744
• DQMHistoTests: Total nulls: 0
• DQMHistoTests: Total successes: 3249018
• DQMHistoTests: Total skipped: 334
• DQMHistoTests: Total Missing objects: 0
• DQMHistoSizes: Histogram memory added: 0.0 KiB( 32 files compared)
• Checked 137 log files, 14 edm output root files, 33 DQM output files

[jfernan2]

@mmusich apart from the fix on Pixel resolution, the changes in this PR are so deep that they are affecting almost all subsystems in the Run3 wf, even HCAL and ECAL RecHits and CaloTowers...
http://tinyurl.com/y3x626bd
Is this expected?

[jfernan2]

OK, answering myself, I realize now that ALCA conditions are different too, please ignore my previous message, sorry

[mmusich]

OK, answering myself, I realize now that ALCA conditions are different too, please ignore my previous message, sorry.

@jfernan2 the changes in the ALCA conditions are affecting only the Pixel subsystem.
The fact that there are changes in other subdetectors is due to the fact that since the (pixel) digitizer conditions are updated there are fluctuation due to the random number generator service usage in the digitization step.

[christopheralanwest]

@mmusich Your candidate GTs do not contain two tags that the Hypernews thread [MC][GT][Run3] Pixel conditions for Run3 studies indicates are needed. If you confirm that these tags should be used, I can create auto:phase1_2021_realistic (106X_upgrade2021_realistic_v8) and auto:phase1_2021_cosmics (106X_upgrade2021cosmics_realistic_deco_v3) GTs for you that includes all of the 2021 pixel updates.

[mmusich]

@mmusich Your candidate GTs do not contain two tags that the Hypernews thread [MC][GT][Run3] Pixel conditions for Run3 studies indicates are needed.

@christopheralanwest it is done on purpose, please try to read carefully #27183 (comment) and #27183 (comment) ...
(only the CPE conditions are actually needed for this PR, the other two conditions mentioned in the thread would just add spurious differences)

If you confirm that these tags should be used, I can create auto:phase1_2021_realistic (106X_upgrade2021_realistic_v8) and auto:phase1_2021_cosmics (106X_upgrade2021cosmics_realistic_deco_v3) GTs for you that includes all of the 2021 pixel updates.

as far as I can see these two GTs do not exist, but if you want to make my candidates real GTs that would actually be better. You can create another set of GTs with the full package (including the other two tags) at a later time and update autoCond in another PR.

[christopheralanwest]

@mmusich Thanks for pointing me to the relevant section of the thread. I would rather use the candidate GTs for now in case some other development arises and needs to be merged before this PR. Once this PR has a reco signature, I will submit a PR that cherry-picks your GT update and then updates the two GTs to use the full 2021 pixel conditions. That will allow the two PRs to be tested separately or together as needed, without generating merge conflicts.

[slava77]

Here are some highlights from ttbar no-PU workflow 11624 (1K events).

The track uncertainties are down, which is reflected in the PV position uncertainties (by about 20%, if I'm reading off the log scale right)

About 7% fewer initialStep tracks

This is partly due to loss of efficiency

It looks like more things than just what's immediately related to the irradiation bias correction have changed.

At the pixel level (also can be seen with less stats in in the DQM gui posted earlier http://tinyurl.com/y5hjwlbw ):
There is more charge in the on-track clusters (most visible in BPIX1)

The x resolutions for the hits have some bias

The y direction in BPIX1 looks OK for the resolution

There is some small bias in FPIX x:

The y pulls look better, closer to 1 (the most significant effect is in BPIX1: Gaussian width changes from 0.61 to 0.94 for a fit within -2, 2)

a similar improvement is visible in FPIX as well

The cluster prob changed, most visibly in BPIX1.

From the loss of efficiency I'm guessing that this change is for the worse.

Overall, the efficiency loss (driven by cluster prob?) seems a bit of a showstopper.

I ran with IrradiationBiasCorrection = False and it seems like by itself it has a fairly small effect compared to the first order large effects mentioned above.
[In the following several plots the baseline in black is IrradiationBiasCorrection = False with otherwise this PR and red is this PR as is]
x and y resolutions get a little bit worse in the inner layers

Another significant effect is in FPIX, but here it's going in the right direction to reduce the average bias/offset from 0. after a fit in +/-25um range the mean shifts from -0.7 to 1.2 for the pos side. So, this is a bit worse, but this seems somewhat negligible.

So, incrementally, just doing IrradiationBiasCorrection = True on top of the GT seems better now (y res in BPIX1 is only a little worse and an increased bias in x res is probably too small to worry much).
However, relative to the baseline, the GT change seems either incomplete or partly incorrect (see cluster probs and initialStep efficiency changes).

[mmusich]

However, relative to the baseline, the GT change seems either incomplete or partly incorrect (see cluster probs and initialStep efficiency changes).

@slava77 I think we are simply correctly simulating more radiation damage now...

[slava77]

+1

for #27183 9310ec1

• this includes a coherent GT update together with enabling the irradiation correction in the pixel CPE. Only Run3 is affected.
• jenkins tests pass and comparisons with the baseline show roughly expected behavior
• local tests with more events also show roughly expected behavior as described in Activate IrradiationBiasCorrection in Pixel CPE generic for Run3 #27183 (comment)
  • some loss in tracking efficiency is attributed to more correct modeling of expected irradiation effects Activate IrradiationBiasCorrection in Pixel CPE generic for Run3 #27183 (comment)

[mmusich]

It looks like more things than just what's immediately related to the irradiation bias correction have changed.

@slava77 thanks for the sign-off.
Just to give a more detailed explanation on #27183 (comment) (maybe a bit too dry) I detailed what has changed in the GT in the PR description (please have a look at: #27183 (comment)).
The radiation damage simulation which is going through the 2D templates has changed:

SiPixel2DTemplateDBObjectRcd  denominator  SiPixel2DTemplateDBObject_phase1_BoR3_HV350_Tr1300_den             SiPixel2DTemplateDBObject_phase1_38T_mc_v4_den_unirradiated     
SiPixel2DTemplateDBObjectRcd  numerator    SiPixel2DTemplateDBObject_phase1_BoR3_HV350_Tr1300_num             SiPixel2DTemplateDBObject_phase1_38T_2018_ultralegacymc_v1_num  

The whole CPE package (reconstruction and digitization conditions) have been updated. These come as an indivisible block as they are all derived with a target integrated dose (in this case end of 2021).
Also, for completeness, we still expect to update the tracker alignment on top of these changes.
By the way this is what will be used for the Run3Summer19DR campaign.

[christopheralanwest]

+1

[cmsbuild]

This pull request is fully signed and it will be integrated in one of the next master IBs (tests are also fine). This pull request will now be reviewed by the release team before it's merged. @davidlange6, @slava77, @smuzaffar, @fabiocos (and backports should be raised in the release meeting by the corresponding L2)

[fabiocos]

+1