Improved muon track validation (new code + customization)

[cmsbuild]

This PR superseded #16646, which had been automatically ported to 90X from the previous PR #15153 (let's hope this time it will make through swiftly !!)

-- Summary description

This PR contains new code and configuration files, introduced alongside with the current code and cfgs. The new code is not yet used in the standard workflows, i.e. nothing should change anywhere.

A customization function will allow the new and the default code to run (both) for a given prerelease cycle in few selected muon RelVals. This will allow for a smooth transition between the current and the new validation code. The plan foresees to deploy the new validation at the prerelease following the one with duplicate validations. At that point the old code should be possibly removed.

The new code and configuration has been flagged by a prefix "New" in filenames, class/module names, when it is a duplicate of the current production code.

Features of the initial PR:

• much clearer code, easier maintenance and developments, some technical improvements
• new plots for performance vs PU, charge misid ...
• removal of a few unused plots
• improved settings of the histogram binning/ranges for different types of muons. All such settings contained in a cfg file "histoParameters_cff.py".
• better choice of names for histograms, corrected labels etc...
• better use of selectors to clean up the cfgs
• slimmed/clearer cfgs as a consequence of the above changes
• kept legacy configurations for use of TrackAssociatorByPosition in separate files, avoiding to clutter the main cfgs as in the current version
• new default setting of the MuonAssociatorByHits to flag as fakes global muons with tracker and muon hits matched to different particles: this will produce sensible fake rates (much higher than what is obtained currently). Side note: the current configuration has always lived for historical reasons.

Added features to the current PR:

• removed the pointer to DQMStore (as requested)
• synchronized the New validation code to the current code, after parallel developments in the last few months (for upgrades)
• customization function to activate the New validation for the temporary period of coexistence with the old one (thanks to Carlo Battilana)

Automatically ported from CMSSW_9_0_X #17489 (original by @abbiendi).
Please wait for a new IB (12 to 24H) before requesting to test this PR.

[cmsbuild]

-1

Tested at: 30b66fb

You can see the results of the tests here:
https://cmssdt.cern.ch/SDT/jenkins-artifacts/pull-request-integration/PR-17723/18391/summary.html

I found follow errors while testing this PR

Failed tests: RelVals AddOn

• RelVals:

When I ran the RelVals I found an error in the following worklfows:
5.1 step1

runTheMatrix-results/5.1_TTbar+TTbarFS+HARVESTFS/step1_TTbar+TTbarFS+HARVESTFS.log

101.0 step1

runTheMatrix-results/101.0_SingleElectronE120EHCAL+SingleElectronE120EHCAL/step1_SingleElectronE120EHCAL+SingleElectronE120EHCAL.log

1003.0 step2

runTheMatrix-results/1003.0_RunMinBias2012A+RunMinBias2012A+RECODDQM+HARVESTDDQM/step2_RunMinBias2012A+RunMinBias2012A+RECODDQM+HARVESTDDQM.log

1000.0 step2

runTheMatrix-results/1000.0_RunMinBias2011A+RunMinBias2011A+TIER0+SKIMD+HARVESTDfst2+ALCASPLIT/step2_RunMinBias2011A+RunMinBias2011A+TIER0+SKIMD+HARVESTDfst2+ALCASPLIT.log

1001.0 step2

runTheMatrix-results/1001.0_RunMinBias2011A+RunMinBias2011A+TIER0EXP+ALCAEXP+ALCAHARVD1+ALCAHARVD2+ALCAHARVD3+ALCAHARVD4+ALCAHARVD5/step2_RunMinBias2011A+RunMinBias2011A+TIER0EXP+ALCAEXP+ALCAHARVD1+ALCAHARVD2+ALCAHARVD3+ALCAHARVD4+ALCAHARVD5.log

135.4 step1

runTheMatrix-results/135.4_ZEE_13+ZEEFS_13+HARVESTUP15FS+MINIAODMCUP15FS/step1_ZEE_13+ZEEFS_13+HARVESTUP15FS+MINIAODMCUP15FS.log

4.53 step3

runTheMatrix-results/4.53_RunPhoton2012B+RunPhoton2012B+HLTD+RECODR1reHLT+HARVESTDR1reHLT/step3_RunPhoton2012B+RunPhoton2012B+HLTD+RECODR1reHLT+HARVESTDR1reHLT.log

1306.0 step3

runTheMatrix-results/1306.0_SingleMuPt1_UP15+SingleMuPt1_UP15+DIGIUP15+RECOUP15+HARVESTUP15/step3_SingleMuPt1_UP15+SingleMuPt1_UP15+DIGIUP15+RECOUP15+HARVESTUP15.log

1330.0 step3

runTheMatrix-results/1330.0_ZMM_13+ZMM_13+DIGIUP15+RECOUP15+HARVESTUP15/step3_ZMM_13+ZMM_13+DIGIUP15+RECOUP15+HARVESTUP15.log

10042.0 step3

runTheMatrix-results/10042.0_ZMM_13+ZMM_13TeV_TuneCUETP8M1_2017_GenSimFull+DigiFull_2017+RecoFull_2017+ALCAFull_2017+HARVESTFull_2017/step3_ZMM_13+ZMM_13TeV_TuneCUETP8M1_2017_GenSimFull+DigiFull_2017+RecoFull_2017+ALCAFull_2017+HARVESTFull_2017.log

20034.0 step3

runTheMatrix-results/20034.0_TTbar_14TeV+TTbar_14TeV_TuneCUETP8M1_2023D7_GenSimHLBeamSpotFull14+DigiFullTrigger_2023D7+RecoFullGlobal_2023D7+HARVESTFullGlobal_2023D7/step3_TTbar_14TeV+TTbar_14TeV_TuneCUETP8M1_2023D7_GenSimHLBeamSpotFull14+DigiFullTrigger_2023D7+RecoFullGlobal_2023D7+HARVESTFullGlobal_2023D7.log

136.731 step3

runTheMatrix-results/136.731_RunSinglePh2016B+RunSinglePh2016B+HLTDR2_2016+RECODR2_2016reHLT_skimSinglePh_HIPM+HARVESTDR2/step3_RunSinglePh2016B+RunSinglePh2016B+HLTDR2_2016+RECODR2_2016reHLT_skimSinglePh_HIPM+HARVESTDR2.log

9.0 step3

runTheMatrix-results/9.0_Higgs200ChargedTaus+Higgs200ChargedTaus+DIGI+RECO+HARVEST/step3_Higgs200ChargedTaus+Higgs200ChargedTaus+DIGI+RECO+HARVEST.log

25.0 step3

runTheMatrix-results/25.0_TTbar+TTbar+DIGI+RECOAlCaCalo+HARVEST+ALCATT/step3_TTbar+TTbar+DIGI+RECOAlCaCalo+HARVEST+ALCATT.log

10024.0 step3

runTheMatrix-results/10024.0_TTbar_13+TTbar_13TeV_TuneCUETP8M1_2017_GenSimFull+DigiFull_2017+RecoFull_2017+ALCAFull_2017+HARVESTFull_2017/step3_TTbar_13+TTbar_13TeV_TuneCUETP8M1_2017_GenSimFull+DigiFull_2017+RecoFull_2017+ALCAFull_2017+HARVESTFull_2017.log

10824.0 step3

runTheMatrix-results/10824.0_TTbar_13+TTbar_13TeV_TuneCUETP8M1_2018_GenSimFull+DigiFull_2018+RecoFull_2018+ALCAFull_2018+HARVESTFull_2018/step3_TTbar_13+TTbar_13TeV_TuneCUETP8M1_2018_GenSimFull+DigiFull_2018+RecoFull_2018+ALCAFull_2018+HARVESTFull_2018.log

50202.0 step3

runTheMatrix-results/50202.0_TTbar_13+TTbar_13+DIGIUP15_PU50+RECOUP15_PU50+HARVESTUP15_PU50/step3_TTbar_13+TTbar_13+DIGIUP15_PU50+RECOUP15_PU50+HARVESTUP15_PU50.log

25202.0 step3

runTheMatrix-results/25202.0_TTbar_13+TTbar_13+DIGIUP15_PU25+RECOUP15_PU25+HARVESTUP15_PU25/step3_TTbar_13+TTbar_13+DIGIUP15_PU25+RECOUP15_PU25+HARVESTUP15_PU25.log

23234.0 step3

runTheMatrix-results/23234.0_TTbar_14TeV+TTbar_14TeV_TuneCUETP8M1_2023D8_GenSimHLBeamSpotFull14+DigiFullTrigger_2023D8+RecoFullGlobal_2023D8+HARVESTFullGlobal_2023D8/step3_TTbar_14TeV+TTbar_14TeV_TuneCUETP8M1_2023D8_GenSimHLBeamSpotFull14+DigiFullTrigger_2023D8+RecoFullGlobal_2023D8+HARVESTFullGlobal_2023D8.log

21234.0 step3

runTheMatrix-results/21234.0_TTbar_14TeV+TTbar_14TeV_TuneCUETP8M1_2023D4_GenSimHLBeamSpotFull14+DigiFullTrigger_2023D4+RecoFullGlobal_2023D4+HARVESTFullGlobal_2023D4/step3_TTbar_14TeV+TTbar_14TeV_TuneCUETP8M1_2023D4_GenSimHLBeamSpotFull14+DigiFullTrigger_2023D4+RecoFullGlobal_2023D4+HARVESTFullGlobal_2023D4.log

• AddOn:

I found errors in the following addon tests:

cmsDriver.py TTbar_8TeV_TuneCUETP8M1_cfi --conditions auto:run1_mc --fast -n 100 --eventcontent AODSIM,DQM --relval 100000,1000 -s GEN,SIM,RECOBEFMIX,DIGI:pdigi_valid,L1,DIGI2RAW,L1Reco,RECO,EI,VALIDATION --customise=HLTrigger/Configuration/CustomConfigs.L1THLT --datatier GEN-SIM-DIGI-RECO,DQMIO --beamspot Realistic8TeVCollision : FAILED - time: date Tue Mar 14 14:37:14 2017-date Tue Mar 14 14:36:27 2017 s - exit: 21248
cmsDriver.py RelVal -s HLT:Fake2,RAW2DIGI,L1Reco,RECO --data --scenario=pp -n 10 --conditions auto:run2_data_Fake2 --relval 9000,50 --datatier "RAW-HLT-RECO" --eventcontent FEVTDEBUGHLT --customise=HLTrigger/Configuration/CustomConfigs.L1THLT --era Run2_2016 --processName=HLTRECO --filein file:RelVal_Raw_Fake2_DATA.root --fileout file:RelVal_Raw_Fake2_DATA_HLT_RECO.root : FAILED - time: date Tue Mar 14 14:37:57 2017-date Tue Mar 14 14:36:34 2017 s - exit: 21248
cmsDriver.py RelVal -s HLT:PRef,RAW2DIGI,L1Reco,RECO --data --scenario=pp -n 10 --conditions auto:run2_data_PRef --relval 9000,50 --datatier "RAW-HLT-RECO" --eventcontent FEVTDEBUGHLT --customise=HLTrigger/Configuration/CustomConfigs.L1THLT --era Run2_2017 --magField 38T_PostLS1 --processName=HLTRECO --filein file:RelVal_Raw_PRef_DATA.root --fileout file:RelVal_Raw_PRef_DATA_HLT_RECO.root : FAILED - time: date Tue Mar 14 14:39:33 2017-date Tue Mar 14 14:36:35 2017 s - exit: 21248
cmsDriver.py RelVal -s HLT:GRun,RAW2DIGI,L1Reco,RECO --mc --scenario=pp -n 10 --conditions auto:run2_mc_GRun --relval 9000,50 --datatier "RAW-HLT-RECO" --eventcontent FEVTDEBUGHLT --customise=HLTrigger/Configuration/CustomConfigs.L1THLT --era Run2_2017 --magField 38T_PostLS1 --processName=HLTRECO --filein file:RelVal_Raw_GRun_MC.root --fileout file:RelVal_Raw_GRun_MC_HLT_RECO.root : FAILED - time: date Tue Mar 14 14:44:20 2017-date Tue Mar 14 14:36:45 2017 s - exit: 21248
cmsDriver.py TTbar_13TeV_TuneCUETP8M1_cfi --conditions auto:run2_mc --fast -n 100 --eventcontent AODSIM,DQM --relval 100000,1000 -s GEN,SIM,RECOBEFMIX,DIGI:pdigi_valid,L1,DIGI2RAW,L1Reco,RECO,EI,VALIDATION --customise=HLTrigger/Configuration/CustomConfigs.L1THLT --datatier GEN-SIM-DIGI-RECO,DQMIO --beamspot NominalCollision2015 --era Run2_25ns --magField 38T_PostLS1 : FAILED - time: date Tue Mar 14 14:37:33 2017-date Tue Mar 14 14:36:50 2017 s - exit: 21248
cmsDriver.py RelVal -s HLT:PRef,RAW2DIGI,L1Reco,RECO --mc --scenario=pp -n 10 --conditions auto:run2_mc_PRef --relval 9000,50 --datatier "RAW-HLT-RECO" --eventcontent FEVTDEBUGHLT --customise=HLTrigger/Configuration/CustomConfigs.L1THLT --era Run2_2017 --magField 38T_PostLS1 --processName=HLTRECO --filein file:RelVal_Raw_PRef_MC.root --fileout file:RelVal_Raw_PRef_MC_HLT_RECO.root : FAILED - time: date Tue Mar 14 14:43:27 2017-date Tue Mar 14 14:36:53 2017 s - exit: 21248
cmsDriver.py TTbar_13TeV_TuneCUETP8M1_cfi --conditions auto:run2_mc --fast -n 100 --eventcontent AODSIM,DQM --relval 100000,1000 -s GEN,SIM,RECOBEFMIX,DIGI:pdigi_valid,L1,DIGI2RAW,L1Reco,RECO,EI,VALIDATION --customise=HLTrigger/Configuration/CustomConfigs.L1THLT --datatier GEN-SIM-DIGI-RECO,DQMIO --beamspot NominalCollision2015 --era Run2_2016 --magField 38T_PostLS1 : FAILED - time: date Tue Mar 14 14:38:08 2017-date Tue Mar 14 14:36:58 2017 s - exit: 21248
cmsDriver.py RelVal -s HLT:GRun2016,RAW2DIGI,L1Reco,RECO --mc --scenario=pp -n 10 --conditions auto:run2_mc_GRun2016 --relval 9000,50 --datatier "RAW-HLT-RECO" --eventcontent FEVTDEBUGHLT --customise=HLTrigger/Configuration/CustomConfigs.L1THLT --era Run2_2016 --magField 38T_PostLS1 --processName=HLTRECO --filein file:RelVal_Raw_GRun2016_MC.root --fileout file:RelVal_Raw_GRun2016_MC_HLT_RECO.root : FAILED - time: date Tue Mar 14 14:44:28 2017-date Tue Mar 14 14:37:00 2017 s - exit: 21248
cmsDriver.py RelVal -s HLT:PIon,RAW2DIGI,L1Reco,RECO --data --scenario=pp -n 10 --conditions auto:run2_data_PIon --relval 9000,50 --datatier "RAW-HLT-RECO" --eventcontent FEVTDEBUGHLT --customise=HLTrigger/Configuration/CustomConfigs.L1THLT --era Run2_2017 --magField 38T_PostLS1 --processName=HLTRECO --filein file:RelVal_Raw_PIon_DATA.root --fileout file:RelVal_Raw_PIon_DATA_HLT_RECO.root : FAILED - time: date Tue Mar 14 14:39:17 2017-date Tue Mar 14 14:37:01 2017 s - exit: 21248
cmsDriver.py RelVal -s HLT:Fake,RAW2DIGI,L1Reco,RECO --mc --scenario=pp -n 10 --conditions auto:run1_mc_Fake --relval 9000,50 --datatier "RAW-HLT-RECO" --eventcontent FEVTDEBUGHLT --customise=HLTrigger/Configuration/CustomConfigs.L1THLT --processName=HLTRECO --filein file:RelVal_Raw_Fake_MC.root --fileout file:RelVal_Raw_Fake_MC_HLT_RECO.root : FAILED - time: date Tue Mar 14 14:41:56 2017-date Tue Mar 14 14:37:03 2017 s - exit: 21248
cmsDriver.py RelVal -s HLT:Fake2,RAW2DIGI,L1Reco,RECO --mc --scenario=pp -n 10 --conditions auto:run2_mc_Fake2 --relval 9000,50 --datatier "RAW-HLT-RECO" --eventcontent FEVTDEBUGHLT --customise=HLTrigger/Configuration/CustomConfigs.L1THLT --era Run2_2016 --processName=HLTRECO --filein file:RelVal_Raw_Fake2_MC.root --fileout file:RelVal_Raw_Fake2_MC_HLT_RECO.root : FAILED - time: date Tue Mar 14 14:42:17 2017-date Tue Mar 14 14:37:09 2017 s - exit: 21248
cmsDriver.py RelVal -s HLT:Fake1,RAW2DIGI,L1Reco,RECO --mc --scenario=pp -n 10 --conditions auto:run2_mc_Fake1 --relval 9000,50 --datatier "RAW-HLT-RECO" --eventcontent FEVTDEBUGHLT --customise=HLTrigger/Configuration/CustomConfigs.L1THLT --era Run2_25ns --processName=HLTRECO --filein file:RelVal_Raw_Fake1_MC.root --fileout file:RelVal_Raw_Fake1_MC_HLT_RECO.root : FAILED - time: date Tue Mar 14 14:42:15 2017-date Tue Mar 14 14:37:11 2017 s - exit: 21248
cmsDriver.py RelVal -s HLT:Fake,RAW2DIGI,L1Reco,RECO --data --scenario=pp -n 10 --conditions auto:run1_data_Fake --relval 9000,50 --datatier "RAW-HLT-RECO" --eventcontent FEVTDEBUGHLT --customise=HLTrigger/Configuration/CustomConfigs.L1THLT --processName=HLTRECO --filein file:RelVal_Raw_Fake_DATA.root --fileout file:RelVal_Raw_Fake_DATA_HLT_RECO.root : FAILED - time: date Tue Mar 14 14:38:16 2017-date Tue Mar 14 14:37:17 2017 s - exit: 21248
cmsDriver.py RelVal -s HLT:GRun,RAW2DIGI,L1Reco,RECO --data --scenario=pp -n 10 --conditions auto:run2_data_GRun --relval 9000,50 --datatier "RAW-HLT-RECO" --eventcontent FEVTDEBUGHLT --customise=HLTrigger/Configuration/CustomConfigs.L1THLT --era Run2_2017 --magField 38T_PostLS1 --processName=HLTRECO --filein file:RelVal_Raw_GRun_DATA.root --fileout file:RelVal_Raw_GRun_DATA_HLT_RECO.root : FAILED - time: date Tue Mar 14 14:42:21 2017-date Tue Mar 14 14:37:22 2017 s - exit: 21248
cmsDriver.py RelVal -s HLT:Fake1,RAW2DIGI,L1Reco,RECO --data --scenario=pp -n 10 --conditions auto:run2_data_Fake1 --relval 9000,50 --datatier "RAW-HLT-RECO" --eventcontent FEVTDEBUGHLT --customise=HLTrigger/Configuration/CustomConfigs.L1THLT --era Run2_25ns --processName=HLTRECO --filein file:RelVal_Raw_Fake1_DATA.root --fileout file:RelVal_Raw_Fake1_DATA_HLT_RECO.root : FAILED - time: date Tue Mar 14 14:38:25 2017-date Tue Mar 14 14:37:25 2017 s - exit: 21248
cmsDriver.py RelVal -s HLT:PIon,RAW2DIGI,L1Reco,RECO --mc --scenario=pp -n 10 --conditions auto:run2_mc_PIon --relval 9000,50 --datatier "RAW-HLT-RECO" --eventcontent FEVTDEBUGHLT --customise=HLTrigger/Configuration/CustomConfigs.L1THLT --era Run2_2017 --magField 38T_PostLS1 --processName=HLTRECO --filein file:RelVal_Raw_PIon_MC.root --fileout file:RelVal_Raw_PIon_MC_HLT_RECO.root : FAILED - time: date Tue Mar 14 14:43:12 2017-date Tue Mar 14 14:37:27 2017 s - exit: 21248
cmsDriver.py RelVal -s HLT:GRun2016,RAW2DIGI,L1Reco,RECO --data --scenario=pp -n 10 --conditions auto:run2_data_GRun2016 --relval 9000,50 --datatier "RAW-HLT-RECO" --eventcontent FEVTDEBUGHLT --customise=HLTrigger/Configuration/CustomConfigs.L1THLT --era Run2_2016 --magField 38T_PostLS1 --processName=HLTRECO --filein file:RelVal_Raw_GRun2016_DATA.root --fileout file:RelVal_Raw_GRun2016_DATA_HLT_RECO.root : FAILED - time: date Tue Mar 14 14:42:22 2017-date Tue Mar 14 14:37:28 2017 s - exit: 21248

[cmsbuild]

Comparison not run due to runTheMatrix errors (RelVals and Igprof tests were also skipped)

[abbiendi]

Hi. Again the failures have nothing to do with the PR.

[davidlange6]

please test
lets try again

[cmsbuild]

The tests are being triggered in jenkins.
https://cmssdt.cern.ch/jenkins/job/ib-any-integration/18450/console Started: 2017/03/15 17:48

[cmsbuild]

+1
Tested at: 30b66fb
https://cmssdt.cern.ch/SDT/jenkins-artifacts/pull-request-integration/PR-17723/18450/summary.html

[cmsbuild]

Comparison job queued.

[cmsbuild]

Comparison is ready
https://cmssdt.cern.ch/SDT/jenkins-artifacts/pull-request-integration/PR-17723/18450/summary.html

There are some workflows for which there are errors in the baseline:
25.0 step 5
The results for the comparisons for these workflows could be incomplete
This means most likely that the IB is having errors in the relvals.The error does NOT come from this pull request

[davidlange6]

Ok, merged for pre1, will expect it out by pre2 unless problems in pre1 validation

[abbiendi]

Thanks.
Will follow up with our plan to have the reference validation with the current code in 90X (to decouple this validation of the code itself from the release validation). Hence we will need now to have PR #17489 merged in 90X to proceed with the plan.

[davidlange6]

we won't do that immediately if at all - since you have a release with both codes running, there is no coupling to release validation (ok, if the release is completely broken you obviously can't do your comparisons)

…

On Mar 16, 2017, at 10:18 AM, abbiendi ***@***.***> wrote: Thanks. Will follow up with our plan to have the reference validation with the current code in 90X (to decouple this validation of the code itself from the release validation). Hence we will need now to have PR #17489 merged in 90X to proceed with the plan. — You are receiving this because you modified the open/close state. Reply to this email directly, view it on GitHub, or mute the thread.

[abbiendi]

But we will not be able to tell if the release is safe or broken before the standard validation is done. This means we will hardly be able to be done in time unless we do it at our full risk. The problem is having a moving floor as the release is moving. Once the switch to the new code will be completed the old one will disappear and we will not be able anymore to compare with previous validations if any issue would appear. This seems dangerous to us.
Both @battibass and @slava77 expressed their opinions in favour of validating the code on a stable release. This is also the plan agreed with the Muon POG conveners (@HuguesBrun , @calderona ), which ultimately are the responsibles for the muon validation workflow and PPD (@cerminar). Since this discussion involved many people a change of plans should be rediscussed with them and the potential risks have to be considered and accepted.

[davidlange6]

maybe the new validation suite should keep enough of the old plots that once its validated, you'll have confidence in your results and be able to compare. From a naive look at the code, there was a a lot in common, but of course the configuration may not be - i didn't evaluate that.

…

On Mar 16, 2017, at 10:42 AM, abbiendi ***@***.***> wrote: But we will not be able to tell if the release is safe or broken before the standard validation is done. This means we will hardly be able to be done in time unless we do it at our full risk. The problem is having a moving floor as the release is moving. Once the switch to the new code will be completed the old one will disappear and we will not be able anymore to compare with previous validations if any issue would appear. This seems dangerous to us. Both @battibass and @slava77 expressed their opinions in favour of validating the code on a stable release. This is also the plan agreed with the Muon POG conveners ***@***.*** , @calderona ), which ultimately are the responsibles for the muon validation workflow and PPD ***@***.***). Since this discussion involved many people a change of plans should be rediscussed with them and the potential risks have to be considered and accepted. — You are receiving this because you modified the open/close state. Reply to this email directly, view it on GitHub, or mute the thread.

[abbiendi]

The differences are not that large on the histograms themselves (or better this is not the main problem). But it is the configuration of the association by hits that is changed. As an example, the old logic (historical, never changed since before Run1, not the best choice indeed) was to have the maximum efficiency without being too sharp on the quality of the reconstruction. For this reason the current plots of fake rates are basically useless. The new ones will be completely different (the fake rates themselves). The step3 implies a significant cpu time and memory usage (too much, and I am planning to improve in further steps), so we cannot keep both configurations running together for a number of release cycles.
To avoid any breakup in the validation chain we need to have benchmark histograms obtained with the new code in a stable release. By comparing them with corresponding plots obtained from the old code we will expose all the differences and will relate them to the known changes in the code. After this is done and ok we will be able to move on with the release validations based on the new code and at that point the old one will never be used. The transition will be completed by removing the old code and renaming the new one dropping all the "New" prefixes in the filenames.