HCALDQM: bugfix, QIE11DataFrame::size() should be samples() #19588

DryRun · 2017-07-06T14:50:17Z

As @slava77 pointed out in #18841, there was a bug introduced that caused a plot to be filled with random values (Hcal/DigiTask/TimingCut/depth/*, for QIE11 channels; the timing is basically the charge-weighted average time slice). QIE11DataFrame::size() should be QIE11DataFrame::samples(); size() includes the digi header/footer, so the charge lookup was going past the last element of the vector storing the charge per time slice.

…:samples()

cmsbuild · 2017-07-06T14:50:36Z

A new Pull Request was created by @DryRun (David Yu) for master.

It involves the following packages:

DQM/HcalTasks

@vazzolini, @kmaeshima, @dmitrijus, @cmsbuild, @vanbesien, @davidlange6 can you please review it and eventually sign? Thanks.
@deguio this is something you requested to watch as well.
@davidlange6 you are the release manager for this.

cms-bot commands are listed here

kpedro88 · 2017-07-07T13:55:01Z

please test

cmsbuild · 2017-07-07T13:55:18Z

The tests are being triggered in jenkins.
https://cmssdt.cern.ch/jenkins/job/ib-any-integration/21262/console Started: 2017/07/07 18:58

cmsbuild · 2017-07-07T20:13:35Z

-1

Tested at: 98cd3b8

The following merge commits were also included on top of IB + this PR after doing git cms-merge-topic:
7c04e1c
You can see more details here:
https://cmssdt.cern.ch/SDT/jenkins-artifacts/pull-request-integration/PR-19588/21262/git-log-recent-commits
https://cmssdt.cern.ch/SDT/jenkins-artifacts/pull-request-integration/PR-19588/21262/git-merge-result

You can see the results of the tests here:
https://cmssdt.cern.ch/SDT/jenkins-artifacts/pull-request-integration/PR-19588/21262/summary.html

I found follow errors while testing this PR

Failed tests: UnitTests

Unit Tests:

I found errors in the following unit tests:

The following merge commits were also included on top of IB + this PR after doing git cms-merge-topic:
7c04e1c
You can see more details here:
https://cmssdt.cern.ch/SDT/jenkins-artifacts/pull-request-integration/PR-19588/21262/git-log-recent-commits
https://cmssdt.cern.ch/SDT/jenkins-artifacts/pull-request-integration/PR-19588/21262/git-merge-result

cmsbuild · 2017-07-07T20:13:38Z

Comparison job queued.

kpedro88 · 2017-07-07T20:56:24Z

@smuzaffar @davidlange6 do you know why the bot is printing "Failed tests: UnitTests" with no list of failed tests afterwards? Is this a bug in the bot, or did something actually go wrong?

smuzaffar · 2017-07-07T22:41:04Z

I see in https://cmssdt.cern.ch/SDT/jenkins-artifacts/pull-request-integration/PR-19588/21262/unitTests.log

^^^^ End Test runFastHadd ^^^^
config/SCRAM/GMake/Makefile.rules:2042: recipe for target 'unittests_runTheWhiteRabbit' failed
gmake: *** [unittests_runTheWhiteRabbit] Terminated

which normally happens when unit tests hangs and killed by timeout

cmsbuild · 2017-07-07T23:29:31Z

Comparison is ready
https://cmssdt.cern.ch/SDT/jenkins-artifacts/pull-request-integration/PR-19588/21262/summary.html

Comparison Summary:

You potentially added 2 lines to the logs
Reco comparison results: 0 differences found in the comparisons
DQMHistoTests: Total files compared: 23
DQMHistoTests: Total histograms compared: 1790682
DQMHistoTests: Total failures: 15050
DQMHistoTests: Total nulls: 0
DQMHistoTests: Total successes: 1775466
DQMHistoTests: Total skipped: 166
DQMHistoTests: Total Missing objects: 0
Checked 93 log files, 14 edm output root files, 23 DQM output files

DryRun · 2017-07-10T10:18:38Z

It's hard for me to imagine how this change would cause the unit test to fail like that... could this be a transient issue?

dmitrijus · 2017-07-10T11:48:03Z

+1

cmsbuild · 2017-07-10T11:48:21Z

This pull request is fully signed and it will be integrated in one of the next master IBs (but tests are reportedly failing). This pull request requires discussion in the ORP meeting before it's merged. @davidlange6, @smuzaffar

dmitrijus · 2017-07-10T12:03:38Z

please test

cmsbuild · 2017-07-10T12:04:00Z

The tests are being triggered in jenkins.
https://cmssdt.cern.ch/jenkins/job/ib-any-integration/21301/console Started: 2017/07/10 14:05

cmsbuild · 2017-07-10T13:33:59Z

-1

Tested at: 98cd3b8

The following merge commits were also included on top of IB + this PR after doing git cms-merge-topic:
0619998
You can see more details here:
https://cmssdt.cern.ch/SDT/jenkins-artifacts/pull-request-integration/PR-19588/21301/git-log-recent-commits
https://cmssdt.cern.ch/SDT/jenkins-artifacts/pull-request-integration/PR-19588/21301/git-merge-result

You can see the results of the tests here:
https://cmssdt.cern.ch/SDT/jenkins-artifacts/pull-request-integration/PR-19588/21301/summary.html

I found follow errors while testing this PR

Failed tests: AddOn

AddOn:

I found errors in the following addon tests:

cmsDriver.py TTbar_Tauola_13TeV_TuneCUETP8M1_cfi -s GEN,SIM,DIGI,L1,DIGI2RAW --mc --scenario=pp -n 10 --conditions auto:run2_mc_GRun --relval 9000,50 --datatier "GEN-SIM-RAW" --eventcontent RAWSIM --customise=HLTrigger/Configuration/CustomConfigs.L1T --era Run2_2017 --magField 38T_PostLS1 --fileout file:RelVal_Raw_GRun_MC.root : FAILED - time: date Mon Jul 10 15:04:15 2017-date Mon Jul 10 15:03:07 2017 s - exit: 21248
cmsRun /cvmfs/cms-ib.cern.ch/nweek-02480/slc6_amd64_gcc530/cms/cmssw-patch/CMSSW_9_3_X_2017-07-09-2300/src/HLTrigger/Configuration/test/OnLine_HLT_GRun.py realData=False globalTag=@ inputFiles=@ : FAILED - time: date Mon Jul 10 15:04:15 2017-date Mon Jul 10 15:03:07 2017 s - exit: 21504
cmsDriver.py RelVal -s HLT:GRun,RAW2DIGI,L1Reco,RECO --mc --scenario=pp -n 10 --conditions auto:run2_mc_GRun --relval 9000,50 --datatier "RAW-HLT-RECO" --eventcontent FEVTDEBUGHLT --customise=HLTrigger/Configuration/CustomConfigs.L1THLT --era Run2_2017 --magField 38T_PostLS1 --processName=HLTRECO --filein file:RelVal_Raw_GRun_MC.root --fileout file:RelVal_Raw_GRun_MC_HLT_RECO.root : FAILED - time: date Mon Jul 10 15:04:15 2017-date Mon Jul 10 15:03:07 2017 s - exit: 21248
cmsDriver.py TTbar_13TeV_TuneCUETP8M1_cfi --conditions auto:run2_mc --fast -n 100 --eventcontent AODSIM,DQM --relval 100000,1000 -s GEN,SIM,RECOBEFMIX,DIGI:pdigi_valid,L1,DIGI2RAW,L1Reco,RECO,EI,VALIDATION --customise=HLTrigger/Configuration/CustomConfigs.L1THLT --datatier GEN-SIM-DIGI-RECO,DQMIO --beamspot NominalCollision2015 --era Run2_25ns --magField 38T_PostLS1 : FAILED - time: date Mon Jul 10 15:03:47 2017-date Mon Jul 10 15:03:11 2017 s - exit: 21248
cmsDriver.py TTbar_Tauola_13TeV_TuneCUETP8M1_cfi -s GEN,SIM,DIGI,L1,DIGI2RAW --mc --scenario=pp -n 10 --conditions auto:run2_mc_PRef --relval 9000,50 --datatier "GEN-SIM-RAW" --eventcontent RAWSIM --customise=HLTrigger/Configuration/CustomConfigs.L1T --era Run2_2017 --magField 38T_PostLS1 --fileout file:RelVal_Raw_PRef_MC.root : FAILED - time: date Mon Jul 10 15:04:06 2017-date Mon Jul 10 15:03:16 2017 s - exit: 21248
cmsRun /cvmfs/cms-ib.cern.ch/nweek-02480/slc6_amd64_gcc530/cms/cmssw-patch/CMSSW_9_3_X_2017-07-09-2300/src/HLTrigger/Configuration/test/OnLine_HLT_PRef.py realData=False globalTag=@ inputFiles=@ : FAILED - time: date Mon Jul 10 15:04:06 2017-date Mon Jul 10 15:03:16 2017 s - exit: 21504
cmsDriver.py RelVal -s HLT:PRef,RAW2DIGI,L1Reco,RECO --mc --scenario=pp -n 10 --conditions auto:run2_mc_PRef --relval 9000,50 --datatier "RAW-HLT-RECO" --eventcontent FEVTDEBUGHLT --customise=HLTrigger/Configuration/CustomConfigs.L1THLT --era Run2_2017 --magField 38T_PostLS1 --processName=HLTRECO --filein file:RelVal_Raw_PRef_MC.root --fileout file:RelVal_Raw_PRef_MC_HLT_RECO.root : FAILED - time: date Mon Jul 10 15:04:06 2017-date Mon Jul 10 15:03:16 2017 s - exit: 21248
cmsDriver.py TTbar_13TeV_TuneCUETP8M1_cfi --conditions auto:run2_mc --fast -n 100 --eventcontent AODSIM,DQM --relval 100000,1000 -s GEN,SIM,RECOBEFMIX,DIGI:pdigi_valid,L1,DIGI2RAW,L1Reco,RECO,EI,VALIDATION --customise=HLTrigger/Configuration/CustomConfigs.L1THLT --datatier GEN-SIM-DIGI-RECO,DQMIO --beamspot NominalCollision2015 --era Run2_2016 --magField 38T_PostLS1 : FAILED - time: date Mon Jul 10 15:04:30 2017-date Mon Jul 10 15:03:41 2017 s - exit: 21248
cmsDriver.py TTbar_Tauola_13TeV_TuneCUETP8M1_cfi -s GEN,SIM,DIGI,L1,DIGI2RAW --mc --scenario=pp -n 10 --conditions auto:run2_mc_GRun2016 --relval 9000,50 --datatier "GEN-SIM-RAW" --eventcontent RAWSIM --customise=HLTrigger/Configuration/CustomConfigs.L1T --era Run2_2016 --magField 38T_PostLS1 --fileout file:RelVal_Raw_GRun2016_MC.root : FAILED - time: date Mon Jul 10 15:06:22 2017-date Mon Jul 10 15:03:52 2017 s - exit: 21248
cmsRun /cvmfs/cms-ib.cern.ch/nweek-02480/slc6_amd64_gcc530/cms/cmssw-patch/CMSSW_9_3_X_2017-07-09-2300/src/HLTrigger/Configuration/test/OnLine_HLT_GRun2016.py realData=False globalTag=@ inputFiles=@ : FAILED - time: date Mon Jul 10 15:06:22 2017-date Mon Jul 10 15:03:52 2017 s - exit: 21504
cmsDriver.py RelVal -s HLT:GRun2016,RAW2DIGI,L1Reco,RECO --mc --scenario=pp -n 10 --conditions auto:run2_mc_GRun2016 --relval 9000,50 --datatier "RAW-HLT-RECO" --eventcontent FEVTDEBUGHLT --customise=HLTrigger/Configuration/CustomConfigs.L1THLT --era Run2_2016 --magField 38T_PostLS1 --processName=HLTRECO --filein file:RelVal_Raw_GRun2016_MC.root --fileout file:RelVal_Raw_GRun2016_MC_HLT_RECO.root : FAILED - time: date Mon Jul 10 15:06:22 2017-date Mon Jul 10 15:03:52 2017 s - exit: 21248
cmsDriver.py TTbar_Tauola_13TeV_TuneCUETP8M1_cfi -s GEN,SIM,DIGI,L1,DIGI2RAW --mc --scenario=HeavyIons -n 10 --conditions auto:run2_mc_HIon --relval 9000,50 --datatier "GEN-SIM-RAW" --eventcontent RAWSIM --customise=HLTrigger/Configuration/CustomConfigs.L1T --era Run2_2016,Run2_HI --magField 38T_PostLS1 --fileout file:RelVal_Raw_HIon_MC.root : FAILED - time: date Mon Jul 10 15:06:30 2017-date Mon Jul 10 15:04:33 2017 s - exit: 21248
cmsRun /cvmfs/cms-ib.cern.ch/nweek-02480/slc6_amd64_gcc530/cms/cmssw-patch/CMSSW_9_3_X_2017-07-09-2300/src/HLTrigger/Configuration/test/OnLine_HLT_HIon.py realData=False globalTag=@ inputFiles=@ : FAILED - time: date Mon Jul 10 15:06:30 2017-date Mon Jul 10 15:04:33 2017 s - exit: 21504
cmsDriver.py RelVal -s HLT:HIon,RAW2DIGI,L1Reco,RECO --mc --scenario=HeavyIons -n 10 --conditions auto:run2_mc_HIon --relval 9000,50 --datatier "RAW-HLT-RECO" --eventcontent FEVTDEBUGHLT --customise=HLTrigger/Configuration/CustomConfigs.L1THLT --era Run2_2016,Run2_HI --magField 38T_PostLS1 --processName=HLTRECO --filein file:RelVal_Raw_HIon_MC.root --fileout file:RelVal_Raw_HIon_MC_HLT_RECO.root : FAILED - time: date Mon Jul 10 15:06:30 2017-date Mon Jul 10 15:04:33 2017 s - exit: 21248
cmsDriver.py TTbar_Tauola_13TeV_TuneCUETP8M1_cfi -s GEN,SIM,DIGI,L1,DIGI2RAW --mc --scenario=pp -n 10 --conditions auto:run2_mc_PIon --relval 9000,50 --datatier "GEN-SIM-RAW" --eventcontent RAWSIM --customise=HLTrigger/Configuration/CustomConfigs.L1T --era Run2_2017 --magField 38T_PostLS1 --fileout file:RelVal_Raw_PIon_MC.root : FAILED - time: date Mon Jul 10 15:13:29 2017-date Mon Jul 10 15:11:51 2017 s - exit: 21248
cmsRun /cvmfs/cms-ib.cern.ch/nweek-02480/slc6_amd64_gcc530/cms/cmssw-patch/CMSSW_9_3_X_2017-07-09-2300/src/HLTrigger/Configuration/test/OnLine_HLT_PIon.py realData=False globalTag=@ inputFiles=@ : FAILED - time: date Mon Jul 10 15:13:29 2017-date Mon Jul 10 15:11:51 2017 s - exit: 21504
cmsDriver.py RelVal -s HLT:PIon,RAW2DIGI,L1Reco,RECO --mc --scenario=pp -n 10 --conditions auto:run2_mc_PIon --relval 9000,50 --datatier "RAW-HLT-RECO" --eventcontent FEVTDEBUGHLT --customise=HLTrigger/Configuration/CustomConfigs.L1THLT --era Run2_2017 --magField 38T_PostLS1 --processName=HLTRECO --filein file:RelVal_Raw_PIon_MC.root --fileout file:RelVal_Raw_PIon_MC_HLT_RECO.root : FAILED - time: date Mon Jul 10 15:13:29 2017-date Mon Jul 10 15:11:51 2017 s - exit: 21248

The following merge commits were also included on top of IB + this PR after doing git cms-merge-topic:
0619998
You can see more details here:
https://cmssdt.cern.ch/SDT/jenkins-artifacts/pull-request-integration/PR-19588/21301/git-log-recent-commits
https://cmssdt.cern.ch/SDT/jenkins-artifacts/pull-request-integration/PR-19588/21301/git-merge-result

cmsbuild · 2017-07-10T13:34:03Z

Comparison job queued.

abdoulline · 2017-07-10T13:57:52Z

Looks like all AddOn tests failures (all with the same - exit: 21248) and the same error message:
<...>
Exception Message:
edm::FileInPath unable to find file MagneticField/GeomBuilder/data/MagneticFieldVolumes_1_v7_large.xml anywhere in the search path.
<...>

are not related to this PR. E.g. :
https://cmssdt.cern.ch/SDT/jenkins-artifacts/pull-request-integration/PR-19588/21301/addOnTests/logs/cmsDriver-fastsim1_cmsDriver.py_TTbar_13TeV_TuneCUETP8M1_cfi_--conditions_auto:run2_mc_--fast__-n_100_--eventcontent_AODSIM,DQM_--relval_100000,1000_-s_GEN,SIM,RECOBEFMI.log

=>

dropping patTrigger
dropping patTrigger
%MSG-i ThreadStreamSetup: (NoModuleName) 10-Jul-2017 15:03:40 CEST pre-events
setting # threads 4
setting # streams 4
%MSG
----- Begin Fatal Exception 10-Jul-2017 15:03:47 CEST-----------------------
An exception of category 'FileInPathError' occurred while
[0] Constructing the EventProcessor
[1] Constructing ESSource: class=XMLIdealGeometryESSource label='magfield'
Exception Message:
edm::FileInPath unable to find file MagneticField/GeomBuilder/data/MagneticFieldVolumes_1_v7_large.xml anywhere in the search path.
The search path is defined by: CMSSW_SEARCH_PATH
${CMSSW_SEARCH_PATH} is: /build/cmsbld/jenkins-workarea/workspace/ib-any-integration/CMSSW_9_3_X_2017-07-09-2300/src:/build/cmsbld/jenkins-workarea/workspace/ib-any-integration/CMSSW_9_3_X_2017-07-09-2300/external/slc6_amd64_gcc530/data:/cvmfs/cms-ib.cern.ch/nweek-02480/slc6_amd64_gcc530/cms/cmssw-patch/CMSSW_9_3_X_2017-07-09-2300/src:/cvmfs/cms-ib.cern.ch/nweek-02480/slc6_amd64_gcc530/cms/cmssw-patch/CMSSW_9_3_X_2017-07-09-2300/external/slc6_amd64_gcc530/data
Current directory is: /build/cmsbld/jenkins-workarea/workspace/ib-any-integration/CMSSW_9_3_X_2017-07-09-2300/src/addOnTests/fastsim1
----- End Fatal Exception -------------------------------------------------

kpedro88 · 2017-07-10T14:51:40Z

Yes, looks like a broken IB caused by #19469

cmsbuild · 2017-07-10T16:50:46Z

Comparison is ready
https://cmssdt.cern.ch/SDT/jenkins-artifacts/pull-request-integration/PR-19588/21301/summary.html

Comparison Summary:

You potentially added 3 lines to the logs
Reco comparison results: 0 differences found in the comparisons
DQMHistoTests: Total files compared: 23
DQMHistoTests: Total histograms compared: 1884334
DQMHistoTests: Total failures: 29359
DQMHistoTests: Total nulls: 0
DQMHistoTests: Total successes: 1854809
DQMHistoTests: Total skipped: 166
DQMHistoTests: Total Missing objects: 0
Checked 93 log files, 14 edm output root files, 23 DQM output files

davidlange6 · 2017-07-10T17:41:02Z

merge

Fix improper use of QIE11DataFrame::size() instead of QIE11DataFrame:…

98cd3b8

…:samples()

cmsbuild added this to the CMSSW_9_3_X milestone Jul 6, 2017

cmsbuild added comparison-pending dqm-pending orp-pending pending-signatures tests-pending labels Jul 6, 2017

cmsbuild added tests-started and removed tests-pending labels Jul 7, 2017

cmsbuild added tests-rejected and removed tests-started labels Jul 7, 2017

cmsbuild added comparison-available and removed comparison-pending labels Jul 7, 2017

DryRun mentioned this pull request Jul 10, 2017

HCALDQM: more fixes for crashes due to calib channels, improve TP emulation, and minor plot adjustments (92X backport of 19547) #19569

Closed

cmsbuild added dqm-approved fully-signed and removed dqm-pending pending-signatures labels Jul 10, 2017

cmsbuild added comparison-pending tests-pending and removed comparison-available tests-rejected labels Jul 10, 2017

cmsbuild added tests-started and removed tests-pending labels Jul 10, 2017

cmsbuild added tests-rejected and removed tests-started labels Jul 10, 2017

cmsbuild added comparison-available and removed comparison-pending labels Jul 10, 2017

cmsbuild merged commit 0d534b8 into cms-sw:master Jul 10, 2017

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

HCALDQM: bugfix, QIE11DataFrame::size() should be samples() #19588

HCALDQM: bugfix, QIE11DataFrame::size() should be samples() #19588

DryRun commented Jul 6, 2017

cmsbuild commented Jul 6, 2017

kpedro88 commented Jul 7, 2017

cmsbuild commented Jul 7, 2017 •

edited

cmsbuild commented Jul 7, 2017

cmsbuild commented Jul 7, 2017

kpedro88 commented Jul 7, 2017

smuzaffar commented Jul 7, 2017

cmsbuild commented Jul 7, 2017

DryRun commented Jul 10, 2017

dmitrijus commented Jul 10, 2017

cmsbuild commented Jul 10, 2017

dmitrijus commented Jul 10, 2017

cmsbuild commented Jul 10, 2017 •

edited

cmsbuild commented Jul 10, 2017

cmsbuild commented Jul 10, 2017

abdoulline commented Jul 10, 2017 •

edited

kpedro88 commented Jul 10, 2017

cmsbuild commented Jul 10, 2017

davidlange6 commented Jul 10, 2017

HCALDQM: bugfix, QIE11DataFrame::size() should be samples() #19588

HCALDQM: bugfix, QIE11DataFrame::size() should be samples() #19588

Conversation

DryRun commented Jul 6, 2017

cmsbuild commented Jul 6, 2017

kpedro88 commented Jul 7, 2017

cmsbuild commented Jul 7, 2017 • edited

cmsbuild commented Jul 7, 2017

cmsbuild commented Jul 7, 2017

kpedro88 commented Jul 7, 2017

smuzaffar commented Jul 7, 2017

cmsbuild commented Jul 7, 2017

DryRun commented Jul 10, 2017

dmitrijus commented Jul 10, 2017

cmsbuild commented Jul 10, 2017

dmitrijus commented Jul 10, 2017

cmsbuild commented Jul 10, 2017 • edited

cmsbuild commented Jul 10, 2017

cmsbuild commented Jul 10, 2017

abdoulline commented Jul 10, 2017 • edited

kpedro88 commented Jul 10, 2017

cmsbuild commented Jul 10, 2017

davidlange6 commented Jul 10, 2017

cmsbuild commented Jul 7, 2017 •

edited

cmsbuild commented Jul 10, 2017 •

edited

abdoulline commented Jul 10, 2017 •

edited