New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Improved muon track validation (new code + customization) #17723
Improved muon track validation (new code + customization) #17723
Conversation
-1 Tested at: 30b66fb You can see the results of the tests here: I found follow errors while testing this PR Failed tests: RelVals AddOn
When I ran the RelVals I found an error in the following worklfows: runTheMatrix-results/5.1_TTbar+TTbarFS+HARVESTFS/step1_TTbar+TTbarFS+HARVESTFS.log101.0 step1 runTheMatrix-results/101.0_SingleElectronE120EHCAL+SingleElectronE120EHCAL/step1_SingleElectronE120EHCAL+SingleElectronE120EHCAL.log1003.0 step2 runTheMatrix-results/1003.0_RunMinBias2012A+RunMinBias2012A+RECODDQM+HARVESTDDQM/step2_RunMinBias2012A+RunMinBias2012A+RECODDQM+HARVESTDDQM.log1000.0 step2 runTheMatrix-results/1000.0_RunMinBias2011A+RunMinBias2011A+TIER0+SKIMD+HARVESTDfst2+ALCASPLIT/step2_RunMinBias2011A+RunMinBias2011A+TIER0+SKIMD+HARVESTDfst2+ALCASPLIT.log1001.0 step2 runTheMatrix-results/1001.0_RunMinBias2011A+RunMinBias2011A+TIER0EXP+ALCAEXP+ALCAHARVD1+ALCAHARVD2+ALCAHARVD3+ALCAHARVD4+ALCAHARVD5/step2_RunMinBias2011A+RunMinBias2011A+TIER0EXP+ALCAEXP+ALCAHARVD1+ALCAHARVD2+ALCAHARVD3+ALCAHARVD4+ALCAHARVD5.log135.4 step1 runTheMatrix-results/135.4_ZEE_13+ZEEFS_13+HARVESTUP15FS+MINIAODMCUP15FS/step1_ZEE_13+ZEEFS_13+HARVESTUP15FS+MINIAODMCUP15FS.log4.53 step3 runTheMatrix-results/4.53_RunPhoton2012B+RunPhoton2012B+HLTD+RECODR1reHLT+HARVESTDR1reHLT/step3_RunPhoton2012B+RunPhoton2012B+HLTD+RECODR1reHLT+HARVESTDR1reHLT.log1306.0 step3 runTheMatrix-results/1306.0_SingleMuPt1_UP15+SingleMuPt1_UP15+DIGIUP15+RECOUP15+HARVESTUP15/step3_SingleMuPt1_UP15+SingleMuPt1_UP15+DIGIUP15+RECOUP15+HARVESTUP15.log1330.0 step3 runTheMatrix-results/1330.0_ZMM_13+ZMM_13+DIGIUP15+RECOUP15+HARVESTUP15/step3_ZMM_13+ZMM_13+DIGIUP15+RECOUP15+HARVESTUP15.log10042.0 step3 runTheMatrix-results/10042.0_ZMM_13+ZMM_13TeV_TuneCUETP8M1_2017_GenSimFull+DigiFull_2017+RecoFull_2017+ALCAFull_2017+HARVESTFull_2017/step3_ZMM_13+ZMM_13TeV_TuneCUETP8M1_2017_GenSimFull+DigiFull_2017+RecoFull_2017+ALCAFull_2017+HARVESTFull_2017.log20034.0 step3 runTheMatrix-results/20034.0_TTbar_14TeV+TTbar_14TeV_TuneCUETP8M1_2023D7_GenSimHLBeamSpotFull14+DigiFullTrigger_2023D7+RecoFullGlobal_2023D7+HARVESTFullGlobal_2023D7/step3_TTbar_14TeV+TTbar_14TeV_TuneCUETP8M1_2023D7_GenSimHLBeamSpotFull14+DigiFullTrigger_2023D7+RecoFullGlobal_2023D7+HARVESTFullGlobal_2023D7.log136.731 step3 runTheMatrix-results/136.731_RunSinglePh2016B+RunSinglePh2016B+HLTDR2_2016+RECODR2_2016reHLT_skimSinglePh_HIPM+HARVESTDR2/step3_RunSinglePh2016B+RunSinglePh2016B+HLTDR2_2016+RECODR2_2016reHLT_skimSinglePh_HIPM+HARVESTDR2.log9.0 step3 runTheMatrix-results/9.0_Higgs200ChargedTaus+Higgs200ChargedTaus+DIGI+RECO+HARVEST/step3_Higgs200ChargedTaus+Higgs200ChargedTaus+DIGI+RECO+HARVEST.log25.0 step3 runTheMatrix-results/25.0_TTbar+TTbar+DIGI+RECOAlCaCalo+HARVEST+ALCATT/step3_TTbar+TTbar+DIGI+RECOAlCaCalo+HARVEST+ALCATT.log10024.0 step3 runTheMatrix-results/10024.0_TTbar_13+TTbar_13TeV_TuneCUETP8M1_2017_GenSimFull+DigiFull_2017+RecoFull_2017+ALCAFull_2017+HARVESTFull_2017/step3_TTbar_13+TTbar_13TeV_TuneCUETP8M1_2017_GenSimFull+DigiFull_2017+RecoFull_2017+ALCAFull_2017+HARVESTFull_2017.log10824.0 step3 runTheMatrix-results/10824.0_TTbar_13+TTbar_13TeV_TuneCUETP8M1_2018_GenSimFull+DigiFull_2018+RecoFull_2018+ALCAFull_2018+HARVESTFull_2018/step3_TTbar_13+TTbar_13TeV_TuneCUETP8M1_2018_GenSimFull+DigiFull_2018+RecoFull_2018+ALCAFull_2018+HARVESTFull_2018.log50202.0 step3 runTheMatrix-results/50202.0_TTbar_13+TTbar_13+DIGIUP15_PU50+RECOUP15_PU50+HARVESTUP15_PU50/step3_TTbar_13+TTbar_13+DIGIUP15_PU50+RECOUP15_PU50+HARVESTUP15_PU50.log25202.0 step3 runTheMatrix-results/25202.0_TTbar_13+TTbar_13+DIGIUP15_PU25+RECOUP15_PU25+HARVESTUP15_PU25/step3_TTbar_13+TTbar_13+DIGIUP15_PU25+RECOUP15_PU25+HARVESTUP15_PU25.log23234.0 step3 runTheMatrix-results/23234.0_TTbar_14TeV+TTbar_14TeV_TuneCUETP8M1_2023D8_GenSimHLBeamSpotFull14+DigiFullTrigger_2023D8+RecoFullGlobal_2023D8+HARVESTFullGlobal_2023D8/step3_TTbar_14TeV+TTbar_14TeV_TuneCUETP8M1_2023D8_GenSimHLBeamSpotFull14+DigiFullTrigger_2023D8+RecoFullGlobal_2023D8+HARVESTFullGlobal_2023D8.log21234.0 step3 runTheMatrix-results/21234.0_TTbar_14TeV+TTbar_14TeV_TuneCUETP8M1_2023D4_GenSimHLBeamSpotFull14+DigiFullTrigger_2023D4+RecoFullGlobal_2023D4+HARVESTFullGlobal_2023D4/step3_TTbar_14TeV+TTbar_14TeV_TuneCUETP8M1_2023D4_GenSimHLBeamSpotFull14+DigiFullTrigger_2023D4+RecoFullGlobal_2023D4+HARVESTFullGlobal_2023D4.log
I found errors in the following addon tests: cmsDriver.py TTbar_8TeV_TuneCUETP8M1_cfi --conditions auto:run1_mc --fast -n 100 --eventcontent AODSIM,DQM --relval 100000,1000 -s GEN,SIM,RECOBEFMIX,DIGI:pdigi_valid,L1,DIGI2RAW,L1Reco,RECO,EI,VALIDATION --customise=HLTrigger/Configuration/CustomConfigs.L1THLT --datatier GEN-SIM-DIGI-RECO,DQMIO --beamspot Realistic8TeVCollision : FAILED - time: date Tue Mar 14 14:37:14 2017-date Tue Mar 14 14:36:27 2017 s - exit: 21248 |
Comparison not run due to runTheMatrix errors (RelVals and Igprof tests were also skipped) |
Hi. Again the failures have nothing to do with the PR. |
please test |
The tests are being triggered in jenkins. |
Comparison job queued. |
Comparison is ready There are some workflows for which there are errors in the baseline: |
Ok, merged for pre1, will expect it out by pre2 unless problems in pre1 validation |
Thanks. |
we won't do that immediately if at all - since you have a release with both codes running, there is no coupling to release validation (ok, if the release is completely broken you obviously can't do your comparisons)
… On Mar 16, 2017, at 10:18 AM, abbiendi ***@***.***> wrote:
Thanks.
Will follow up with our plan to have the reference validation with the current code in 90X (to decouple this validation of the code itself from the release validation). Hence we will need now to have PR #17489 merged in 90X to proceed with the plan.
—
You are receiving this because you modified the open/close state.
Reply to this email directly, view it on GitHub, or mute the thread.
|
But we will not be able to tell if the release is safe or broken before the standard validation is done. This means we will hardly be able to be done in time unless we do it at our full risk. The problem is having a moving floor as the release is moving. Once the switch to the new code will be completed the old one will disappear and we will not be able anymore to compare with previous validations if any issue would appear. This seems dangerous to us. |
maybe the new validation suite should keep enough of the old plots that once its validated, you'll have confidence in your results and be able to compare. From a naive look at the code, there was a a lot in common, but of course the configuration may not be - i didn't evaluate that.
… On Mar 16, 2017, at 10:42 AM, abbiendi ***@***.***> wrote:
But we will not be able to tell if the release is safe or broken before the standard validation is done. This means we will hardly be able to be done in time unless we do it at our full risk. The problem is having a moving floor as the release is moving. Once the switch to the new code will be completed the old one will disappear and we will not be able anymore to compare with previous validations if any issue would appear. This seems dangerous to us.
Both @battibass and @slava77 expressed their opinions in favour of validating the code on a stable release. This is also the plan agreed with the Muon POG conveners ***@***.*** , @calderona ), which ultimately are the responsibles for the muon validation workflow and PPD ***@***.***). Since this discussion involved many people a change of plans should be rediscussed with them and the potential risks have to be considered and accepted.
—
You are receiving this because you modified the open/close state.
Reply to this email directly, view it on GitHub, or mute the thread.
|
The differences are not that large on the histograms themselves (or better this is not the main problem). But it is the configuration of the association by hits that is changed. As an example, the old logic (historical, never changed since before Run1, not the best choice indeed) was to have the maximum efficiency without being too sharp on the quality of the reconstruction. For this reason the current plots of fake rates are basically useless. The new ones will be completely different (the fake rates themselves). The step3 implies a significant cpu time and memory usage (too much, and I am planning to improve in further steps), so we cannot keep both configurations running together for a number of release cycles. |
This PR superseded #16646, which had been automatically ported to 90X from the previous PR #15153 (let's hope this time it will make through swiftly !!)
-- Summary description
This PR contains new code and configuration files, introduced alongside with the current code and cfgs. The new code is not yet used in the standard workflows, i.e. nothing should change anywhere.
A customization function will allow the new and the default code to run (both) for a given prerelease cycle in few selected muon RelVals. This will allow for a smooth transition between the current and the new validation code. The plan foresees to deploy the new validation at the prerelease following the one with duplicate validations. At that point the old code should be possibly removed.
The new code and configuration has been flagged by a prefix "New" in filenames, class/module names, when it is a duplicate of the current production code.
Features of the initial PR:
Added features to the current PR:
Automatically ported from CMSSW_9_0_X #17489 (original by @abbiendi).
Please wait for a new IB (12 to 24H) before requesting to test this PR.