GEM unpacker: bugfix and more cross checks #31863

jshlee · 2020-10-19T18:38:40Z

PR description:

GEMDigiToRawModule issue with packing large data
- Between 11_2_0pre2 to pre7, in phase2 PU samples, GEMMuons were not present which increased the fake rate in muons (https://indico.cern.ch/event/965774/#5-muo-gem-validation-studies).
- This was found to be due to GEM digis not being produced (https://indico.cern.ch/event/965774/#6-gem-fed-issue).
- GEM simdigis were not being packed correctly because only 12 AMCs can be stored per AMC13 (https://github.com/cms-sw/cmssw/blob/master/DataFormats/GEMDigi/interface/AMC13Event.h#L25), and with multiple BXs, PU events made many more AMCs which couldn't be unpacked.
- added more cross checks
GEMRawToDigiModule
- GEM digi bx is wrong, it should be the relative difference between AMC and BFAT bx.
- crash protection from bad GEM chamber ID from DAQ
No changes are expected for non PU samples.
PU should be back to 11_2_0pre1

PR validation:

tested with wf 11611.0, 23211.0 and run 337972 MWGR data.

if this PR is a backport please specify the original PR and why you need to backport that PR:

backport might be needed for the crash protection

only 12 AMCs can be readout GEMRawToDigiModule - fix gem digi bx

cmsbuild · 2020-10-19T18:38:59Z

The code-checks are being triggered in jenkins.

cmsbuild · 2020-10-19T18:45:11Z

+code-checks

Logs: https://cmssdt.cern.ch/SDT/code-checks/cms-sw-PR-31863/19221

This PR adds an extra 28KB to repository

cmsbuild · 2020-10-19T18:45:33Z

A new Pull Request was created by @jshlee (Jason Lee) for master.

It involves the following packages:

DataFormats/GEMDigi
EventFilter/GEMRawToDigi

@perrotta, @civanch, @kpedro88, @cmsbuild, @mdhildreth, @slava77, @jpata can you please review it and eventually sign? Thanks.
@Martin-Grunewald, @dildick, @rovere, @watson-ij this is something you requested to watch as well.
@silviodonato, @dpiparo, @qliphy you are the release manager for this.

cms-bot commands are listed here

kpedro88 · 2020-10-19T18:50:10Z

EventFilter/GEMRawToDigi/plugins/GEMRawToDigiModule.cc

+    }
+    // compare trailers
+    if (amc13Event->fragmentLength() != trailer.fragmentLength()) {
+      edm::LogWarning("GEMRawToDigiModule") << " fragmentLength miss match";


typo: miss match -> mismatch (also in next few lines)

in all of these messages (also in the next file), it might be useful to print the actual mismatched values as well

thanks @kpedro88!
I'll fix them after the test

The values for amc13Event and trailer would not be that useful, since they should be reading the same data.
This test is to just make sure that they are both reading the last data line.

EventFilter/GEMRawToDigi/src/GEMRawToDigi.cc

kpedro88 · 2020-10-19T18:51:32Z

please test

cmsbuild · 2020-10-19T18:51:48Z

The tests are being triggered in jenkins.

CMSSW_11_2_X_2020-10-19-1100/slc7_amd64_gcc820: https://cmssdt.cern.ch/jenkins/job/ib-run-pr-tests/10103/console Started: 2020/10/19 20:54

cmsbuild · 2020-10-19T19:56:53Z

+1
Tested at: d7cec9e
https://cmssdt.cern.ch/SDT/jenkins-artifacts/pull-request-integration/PR-fa155c/10103/summary.html
CMSSW: CMSSW_11_2_X_2020-10-19-1100
SCRAM_ARCH: slc7_amd64_gcc820

cmsbuild · 2020-10-19T19:56:56Z

Comparison job queued.

cmsbuild · 2020-10-19T21:14:24Z

Comparison is ready
https://cmssdt.cern.ch/SDT/jenkins-artifacts/pull-request-integration/PR-fa155c/10103/summary.html

Comparison Summary:

No significant changes to the logs found
Reco comparison results: 9 differences found in the comparisons
DQMHistoTests: Total files compared: 35
DQMHistoTests: Total histograms compared: 2543752
DQMHistoTests: Total failures: 19
DQMHistoTests: Total nulls: 0
DQMHistoTests: Total successes: 2543711
DQMHistoTests: Total skipped: 22
DQMHistoTests: Total Missing objects: 0
DQMHistoSizes: Histogram memory added: 0.0 KiB( 34 files compared)
Checked 149 log files, 22 edm output root files, 35 DQM output files

cmsbuild · 2020-10-22T16:55:14Z

The code-checks are being triggered in jenkins.

cmsbuild · 2020-10-22T17:01:29Z

+code-checks

Logs: https://cmssdt.cern.ch/SDT/code-checks/cms-sw-PR-31863/19330

This PR adds an extra 32KB to repository

cmsbuild · 2020-10-22T17:01:54Z

Pull request #31863 was updated. @perrotta, @civanch, @kpedro88, @cmsbuild, @jpata, @mdhildreth, @slava77, @ggovi can you please check and sign again.

slava77 · 2020-10-22T17:07:06Z

@cmsbuild please test

cmsbuild · 2020-10-22T17:07:25Z

The tests are being triggered in jenkins.

CMSSW_11_2_X_2020-10-21-2300/slc7_amd64_gcc820: https://cmssdt.cern.ch/jenkins/job/ib-run-pr-tests/10217/console Started: 2020/10/22 19:08

silviodonato · 2020-10-22T17:24:50Z

@cms-sw/simulation-l2 @cms-sw/upgrade-l2 @cms-sw/db-l2
do you have any comments?
I would try to merge this before the next IB to include it in pre8 (this is a long pending issue discussed at the PPD meetings, it would be good to get it fixed in the next round of validation)

kpedro88 · 2020-10-22T17:33:00Z

I have no other comments; will sign once tests pass.

cmsbuild · 2020-10-22T18:17:17Z

-1

Tested at: 78a7536

CMSSW: CMSSW_11_2_X_2020-10-21-2300
SCRAM_ARCH: slc7_amd64_gcc820
You can see the results of the tests here:
https://cmssdt.cern.ch/SDT/jenkins-artifacts/pull-request-integration/PR-fa155c/10217/summary.html

I found follow errors while testing this PR

Failed tests: AddOn

AddOn:

I found errors in the following addon tests:

cmsDriver.py RelVal -s HLT:PIon,RAW2DIGI,L1Reco,RECO --mc --scenario=pp -n 10 --conditions auto:run3_mc_PIon --relval 9000,50 --datatier "RAW-HLT-RECO" --eventcontent FEVTDEBUGHLT --customise=HLTrigger/Configuration/CustomConfigs.L1THLT --era Run3 --processName=HLTRECO --filein file:RelVal_Raw_PIon_MC.root --fileout file:RelVal_Raw_PIon_MC_HLT_RECO.root : FAILED - time: date Thu Oct 22 20:07:34 2020-date Thu Oct 22 19:59:32 2020 s - exit: 34304

cmsbuild · 2020-10-22T18:17:19Z

Comparison job queued.

silviodonato · 2020-10-22T18:56:18Z

-1

Tested at: 78a7536

CMSSW: CMSSW_11_2_X_2020-10-21-2300
SCRAM_ARCH: slc7_amd64_gcc820
You can see the results of the tests here:
https://cmssdt.cern.ch/SDT/jenkins-artifacts/pull-request-integration/PR-fa155c/10217/summary.html

I found follow errors while testing this PR

Failed tests: AddOn
* **AddOn**:
I found errors in the following addon tests:

cmsDriver.py RelVal -s HLT:PIon,RAW2DIGI,L1Reco,RECO --mc --scenario=pp -n 10 --conditions auto:run3_mc_PIon --relval 9000,50 --datatier "RAW-HLT-RECO" --eventcontent FEVTDEBUGHLT --customise=HLTrigger/Configuration/CustomConfigs.L1THLT --era Run3 --processName=HLTRECO --filein file:RelVal_Raw_PIon_MC.root --fileout file:RelVal_Raw_PIon_MC_HLT_RECO.root : FAILED - time: date Thu Oct 22 20:07:34 2020-date Thu Oct 22 19:59:32 2020 s - exit: 34304

The error is in the very last line of
https://cmssdt.cern.ch/SDT/jenkins-artifacts/pull-request-integration/PR-fa155c/10217/addOnTests/hlt_mc_PIon/cmsDriver.py_RelVal_-s_HLT:PIon,RAW2DIGI,L1Reco,RECO_--mc_--scenario=pp_-n_10_--conditions_auto:run3_mc_PIon_--relval_9000,50_--datatier_RAW-HLT-RECO.log

Fatal system signal has occurred during exit

I hope it is just a transient problem

silviodonato · 2020-10-22T19:06:30Z

Yes, we saw that same problem here #31602 (comment) and it went away by running one more time the test.

Same thing here #31792 (comment)

slava77 · 2020-10-22T19:25:03Z

+1

for #31863 78a7536

most recent updates since the last reco signoff were technical https://github.com/cms-sw/cmssw/pull/31863/files/7433f4cbeac77d60188758fca5f1d8fa3d2a2973..78a7536d48642cf02c37db8d6965c47d65d53d53

cmsbuild · 2020-10-22T19:34:48Z

Comparison is ready
https://cmssdt.cern.ch/SDT/jenkins-artifacts/pull-request-integration/PR-fa155c/10217/summary.html

Comparison Summary:

No significant changes to the logs found
Reco comparison results: 6 differences found in the comparisons
DQMHistoTests: Total files compared: 35
DQMHistoTests: Total histograms compared: 2544110
DQMHistoTests: Total failures: 16
DQMHistoTests: Total nulls: 0
DQMHistoTests: Total successes: 2544072
DQMHistoTests: Total skipped: 22
DQMHistoTests: Total Missing objects: 0
DQMHistoSizes: Histogram memory added: 0.0 KiB( 34 files compared)
Checked 149 log files, 22 edm output root files, 35 DQM output files

silviodonato · 2020-10-22T20:14:35Z

merge

jshlee added 3 commits October 19, 2020 03:12

GEMDigiToRawModule - fix AMC size in packing

5e03f73

only 12 AMCs can be readout GEMRawToDigiModule - fix gem digi bx

remove commented code

066c859

added more sanity checks

d7cec9e

cmsbuild added this to the CMSSW_11_2_X milestone Oct 19, 2020

cmsbuild added code-checks-pending comparison-pending orp-pending pending-signatures reconstruction-pending simulation-pending tests-pending upgrade-pending labels Oct 19, 2020

cmsbuild added code-checks-approved and removed code-checks-pending labels Oct 19, 2020

kpedro88 reviewed Oct 19, 2020

View reviewed changes

cmsbuild added tests-started and removed tests-pending labels Oct 19, 2020

cmsbuild added tests-approved and removed tests-started labels Oct 19, 2020

cmsbuild added comparison-available and removed comparison-pending labels Oct 19, 2020

typo

ff40bab

cmsbuild added reconstruction-pending tests-pending labels Oct 22, 2020

cmsbuild added code-checks-approved and removed code-checks-pending labels Oct 22, 2020

cmsbuild added tests-started and removed tests-pending labels Oct 22, 2020

cmsbuild added tests-rejected and removed tests-started labels Oct 22, 2020

cmsbuild added reconstruction-approved and removed reconstruction-pending labels Oct 22, 2020

cmsbuild added comparison-available and removed comparison-pending labels Oct 22, 2020

cmsbuild added orp-approved and removed orp-pending labels Oct 22, 2020

cmsbuild merged commit 3c9bd1b into cms-sw:master Oct 22, 2020

jshlee mentioned this pull request Oct 23, 2020

GEM unpacker: crash protection from bad FEDs #31928

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

GEM unpacker: bugfix and more cross checks #31863

GEM unpacker: bugfix and more cross checks #31863

jshlee commented Oct 19, 2020

cmsbuild commented Oct 19, 2020

cmsbuild commented Oct 19, 2020

cmsbuild commented Oct 19, 2020

kpedro88 Oct 19, 2020

kpedro88 Oct 19, 2020

jshlee Oct 19, 2020

jshlee Oct 20, 2020

kpedro88 commented Oct 19, 2020

cmsbuild commented Oct 19, 2020 •

edited

cmsbuild commented Oct 19, 2020

cmsbuild commented Oct 19, 2020

cmsbuild commented Oct 19, 2020

cmsbuild commented Oct 22, 2020

cmsbuild commented Oct 22, 2020

cmsbuild commented Oct 22, 2020

slava77 commented Oct 22, 2020

cmsbuild commented Oct 22, 2020 •

edited

silviodonato commented Oct 22, 2020

kpedro88 commented Oct 22, 2020

cmsbuild commented Oct 22, 2020

cmsbuild commented Oct 22, 2020

silviodonato commented Oct 22, 2020 •

edited

silviodonato commented Oct 22, 2020

slava77 commented Oct 22, 2020

cmsbuild commented Oct 22, 2020

silviodonato commented Oct 22, 2020

GEM unpacker: bugfix and more cross checks #31863

GEM unpacker: bugfix and more cross checks #31863

Conversation

jshlee commented Oct 19, 2020

PR description:

PR validation:

if this PR is a backport please specify the original PR and why you need to backport that PR:

cmsbuild commented Oct 19, 2020

cmsbuild commented Oct 19, 2020

cmsbuild commented Oct 19, 2020

kpedro88 Oct 19, 2020

Choose a reason for hiding this comment

kpedro88 Oct 19, 2020

Choose a reason for hiding this comment

jshlee Oct 19, 2020

Choose a reason for hiding this comment

jshlee Oct 20, 2020

Choose a reason for hiding this comment

kpedro88 commented Oct 19, 2020

cmsbuild commented Oct 19, 2020 • edited

cmsbuild commented Oct 19, 2020

cmsbuild commented Oct 19, 2020

cmsbuild commented Oct 19, 2020

cmsbuild commented Oct 22, 2020

cmsbuild commented Oct 22, 2020

cmsbuild commented Oct 22, 2020

slava77 commented Oct 22, 2020

cmsbuild commented Oct 22, 2020 • edited

silviodonato commented Oct 22, 2020

kpedro88 commented Oct 22, 2020

cmsbuild commented Oct 22, 2020

cmsbuild commented Oct 22, 2020

silviodonato commented Oct 22, 2020 • edited

silviodonato commented Oct 22, 2020

slava77 commented Oct 22, 2020

cmsbuild commented Oct 22, 2020

silviodonato commented Oct 22, 2020

cmsbuild commented Oct 19, 2020 •

edited

cmsbuild commented Oct 22, 2020 •

edited

silviodonato commented Oct 22, 2020 •

edited