BMTF-KMTF DQM Algo Selector for the DQM/L1TMonitor Subsystem #24557

panoskatsoulis · 2018-09-16T18:38:36Z

This is the introduction of a new DQM plugin that selects the correct output muons of the BMTF depending on the algorithm that is triggering (Kalman BMTF/ Legacy BMTF).
Also, the PR includes the last BMTF Unpacker which is required by the DQM workflow and by the L1TBMTFAlgoSelector to operate correctly

…orithms

cmsbuild · 2018-09-16T18:39:00Z

The code-checks are being triggered in jenkins.

cmsbuild · 2018-09-16T18:43:04Z

-code-checks

Logs: https://cmssdt.cern.ch/SDT/code-checks/cms-sw-PR-24557/6441

Code check has found code style and quality issues which could be resolved by applying a patch in https://cmssdt.cern.ch/SDT/code-checks/cms-sw-PR-24557/6441/git-diff.patch
e.g. curl https://cmssdt.cern.ch/SDT/code-checks/cms-sw-PR-24557/6441/git-diff.patch | patch -p1

You can run scram build code-checks to apply code checks directly

andrius-k · 2018-09-17T11:17:33Z

@panoskatsoulis Could you please make sure that there is a name of a subsystem in the title of the PR?

…e-checks'

cmsbuild · 2018-09-17T19:06:00Z

The code-checks are being triggered in jenkins.

cmsbuild · 2018-09-17T19:08:46Z

+code-checks

Logs: https://cmssdt.cern.ch/SDT/code-checks/cms-sw-PR-24557/6458

cmsbuild · 2018-09-17T19:09:10Z

A new Pull Request was created by @panoskatsoulis for master.

It involves the following packages:

DQM/L1TMonitor
DQM/L1TMonitorClient
EventFilter/L1TRawToDigi
L1Trigger/L1TMuon

@cmsbuild, @andrius-k, @kmaeshima, @schneiml, @nsmith-, @rekovic, @jfernan2, @thomreis can you please review it and eventually sign? Thanks.
@Martin-Grunewald, @thomreis this is something you requested to watch as well.
@davidlange6, @slava77, @fabiocos you are the release manager for this.

cms-bot commands are listed here

panoskatsoulis · 2018-09-17T19:10:41Z

@andrius-k I've changed the title, I hope this is what you were talking about

andrius-k · 2018-09-17T19:24:39Z

please test
@panoskatsoulis Yes, thanks, it's fine now.

cmsbuild · 2018-09-17T19:25:00Z

The tests are being triggered in jenkins.
https://cmssdt.cern.ch/jenkins/job/ib-any-integration/30466/console Started: 2018/09/17 21:25

cmsbuild · 2018-09-18T00:12:44Z

Comparison is ready
https://cmssdt.cern.ch/SDT/jenkins-artifacts/pull-request-integration/PR-24557/30466/summary.html

Comparison Summary:

No significant changes to the logs found
Reco comparison results: 2 differences found in the comparisons
DQMHistoTests: Total files compared: 32
DQMHistoTests: Total histograms compared: 3145953
DQMHistoTests: Total failures: 23
DQMHistoTests: Total nulls: 37
DQMHistoTests: Total successes: 3145696
DQMHistoTests: Total skipped: 197
DQMHistoTests: Total Missing objects: 0
DQMHistoSizes: Histogram memory added: 98.947 KiB( 31 files compared)
DQMHistoSizes: changed ( 10224.0,... ): 4.363 KiB L1TEMU/L1TdeStage2BMTF
DQMHistoSizes: changed ( 10224.0,... ): 2.233 KiB L1T/L1TStage2BMTF
DQMHistoSizes: changed ( 150.0,... ): 4.359 KiB L1TEMU/L1TdeStage2BMTF
Checked 133 log files, 14 edm output root files, 32 DQM output files

rekovic · 2018-09-18T06:33:32Z

+1

andrius-k · 2018-09-18T17:26:55Z

+1

cmsbuild · 2018-09-18T17:27:16Z

This pull request is fully signed and it will be integrated in one of the next master IBs (tests are also fine). This pull request will now be reviewed by the release team before it's merged. @davidlange6, @slava77, @smuzaffar, @fabiocos (and backports should be raised in the release meeting by the corresponding L2)

fabiocos · 2018-09-23T11:00:50Z

+1

Dr15Jones · 2018-09-24T13:03:59Z

This pull request is causing segmentation faults in the nightly release validation for workflow 136.8562. E.g.

https://cmssdt.cern.ch/SDT/cgi-bin/logreader/slc7_amd64_gcc820/CMSSW_10_3_X_2018-09-23-2300/pyRelValMatrixLogs/run/136.8562_RunZeroBias1_hBStarRP+RunZeroBias1_hBStarRP+HLTDR2_2018_hBStar+RECODR2_2018reHLT_Prompt_hBStar+HARVEST2018_hBStar/step3_RunZeroBias1_hBStarRP+RunZeroBias1_hBStarRP+HLTDR2_2018_hBStar+RECODR2_2018reHLT_Prompt_hBStar+HARVEST2018_hBStar.log#/444-444

panoskatsoulis · 2018-09-24T14:00:35Z

Hello, this seems to come from a try to read a bx that is not present (maybe because of Zero Suppression). Can you please send me the workflow to reproduce it? Maybe an "if" check can be used there. Kind Regards, Panos

…

On 24/09/18 15:04, Chris Jones wrote: This pull request is causing segmentation faults in the nightly release validation for workflow 136.8562. E.g. https://cmssdt.cern.ch/SDT/cgi-bin/logreader/slc7_amd64_gcc820/CMSSW_10_3_X_2018-09-23-2300/pyRelValMatrixLogs/run/136.8562_RunZeroBias1_hBStarRP+RunZeroBias1_hBStarRP+HLTDR2_2018_hBStar+RECODR2_2018reHLT_Prompt_hBStar+HARVEST2018_hBStar/step3_RunZeroBias1_hBStarRP+RunZeroBias1_hBStarRP+HLTDR2_2018_hBStar+RECODR2_2018reHLT_Prompt_hBStar+HARVEST2018_hBStar.log#/444-444 — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#24557 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AU_Y_ciPsUCpaSHE_Y8BiyofUmEleZXjks5ueNhMgaJpZM4Wq-0o>.

Dr15Jones · 2018-09-24T14:03:19Z

runTheMatrix -l 136.8562 should be able to reproduce the problem.

panoskatsoulis · 2018-09-24T14:07:02Z

Thanks for this, I will reply back once I have results. Panos

…

On 24/09/18 16:03, Chris Jones wrote: |runTheMatrix -l 136.8562| should be able to reproduce the problem. — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#24557 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AU_Y_ZCZogtwZ4tKCdYaCEguDPV-NlQnks5ueOYvgaJpZM4Wq-0o>.

panoskatsoulis · 2018-09-24T15:03:28Z

Hello, when I run the command "|runTheMatrix -l 136.8562"||| I cannot fetch the files using DAS. The requested files by the workflow 136.8562 are not to accessible place for me. Below are the available sited I get by the das query. T0_CH_CERN_Export T0_CH_CERN_MSS T1_US_FNAL_Buffer T1_US_FNAL_MSS However, the problem has been spotted since I have the run number now. The PR 24557 introduces software that is necessary when the BMTF fw runs with 2 algorithms. The used run by the test, 314276, is an old run that only one algorithm runs on the BMTF fpga. What happened is that the MP7 header is suppressed because all the BXs are also suppressed. The L1TBMTFAlgoSelector plugin tries to fetch information from this header that is not present. This was not expected to happen because on newer runs the ZS for the second algorithm is disabled. The newly added L1TBMTFAlgoSelector is not intended to run under these circumstances. The code must be extended to cover such cases. I will prepare a new PR for the master to fix this. How much urgent is this? Also, could you please send me an other workflow (old enough) that the files will be available? It will be very useful, for debugging this. Kind Regards, Panos

…

On 24/09/18 16:03, Chris Jones wrote: |runTheMatrix -l 136.8562| should be able to reproduce the problem. — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#24557 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AU_Y_ZCZogtwZ4tKCdYaCEguDPV-NlQnks5ueOYvgaJpZM4Wq-0o>.

schneiml · 2018-09-24T16:04:05Z

@panoskatsoulis If you are running your test on CERN infrastructure (lxplus or similar), try running runTheMatrix -l 136.8562 --ibeos. This should get the data exactly the same way the integration tests do.

Martin-Grunewald · 2018-09-25T10:54:08Z

It seems this PR also creates the following problem in a TSG HIon workflow:

----- Begin Fatal Exception 25-Sep-2018 12:33:54 CEST-----------------------
An exception of category 'ProductNotFound' occurred while
   [0] Processing  Event run: 263689 lumi: 5 event: 395501 stream: 2
   [1] Calling method for module L1TStage2RegionalMuonCandComp/'l1tdeStage2Bmtf'
Exception Message:
Principal::getByToken: Found zero products matching all criteria
Looking for type: BXVector<l1t::RegionalMuonCand>
Looking for module label: valBmtfAlgoSel
Looking for productInstanceName: BMTF

----- End Fatal Exception -------------------------------------------------

as verified using CMSSW_10_3_X_2018-09-23-1100 (OK) vs CMSSW_10_3_X_2018-09-23-1100 + this PR (fatal exception).

It may be related to the problem discussed above.

The cmsDriver command is:

RelVal --step=RAW2DIGI,L1Reco,RECO,DQM --conditions=auto:run2_data_HIon --filein=file:RelVal_HLT_HIon_DATA.root --custom_conditions= --fileout=RelVal_RECO_HIon_DATA.root --number=100 --data --no_exec --datatier RECO,MINIAOD,DQMIO --eventcontent=RECO,MINIAOD,DQM --customise=HLTrigger/Configuration/CustomConfigs.Base --era=Run2_2016,Run2_HI --customise= --scenario=HeavyIons --python_filename=RelVal_RECO_HIon_DATA.py --processName=RECO1 --runUnscheduled

panoskatsoulis · 2018-09-25T11:03:11Z

The issue about the SEGFAULT is solved in the PR #24652
I have run the test locally, it works fine for me after the fix.

Martin-Grunewald · 2018-09-25T12:40:56Z

Looks like your bug fix also cures the fatal exception I observed in our TSG tests. Thanks!

panoskatsoulis · 2018-09-25T12:58:33Z

@Martin-Grunewald
Have you tested this?
Because I couldn't perform the last test you mentioned with the cmsDriver command.
I didn't have RAW files from era 2016-Heavy-ions

Martin-Grunewald · 2018-09-25T12:59:11Z

Yes, I added the bug fix PR and reran my tests.

panoskatsoulis · 2018-09-25T13:01:25Z

Great.
then it seems this has been solved.

panoskatsoulis added 2 commits September 16, 2018 20:11

branch to PR the DQM fix to the CMSSW master

aef27ff

removed different handling of the Unpacker on muon-qual for the 2 alg…

bfed7ce

…orithms

cmsbuild added this to the CMSSW_10_3_X milestone Sep 16, 2018

cmsbuild added code-checks-pending comparison-pending dqm-pending l1-pending orp-pending pending-signatures tests-pending labels Sep 16, 2018

panoskatsoulis mentioned this pull request Sep 16, 2018

BMTF-KMTF DQM Algo Selector for the DQM/L1TMonitor Subsystem #24555

Closed

cmsbuild added code-checks-rejected and removed code-checks-pending labels Sep 16, 2018

changed base to 10_3_X version, and made proposed changes by the 'cod…

bc88a05

…e-checks'

cmsbuild added code-checks-pending and removed code-checks-rejected labels Sep 17, 2018

cmsbuild added code-checks-approved and removed code-checks-pending labels Sep 17, 2018

panoskatsoulis changed the title ~~BMTF-KMTF DQM Algo Selector~~ BMTF-KMTF DQM Algo Selector for the DQM/L1TMonitor Subsystem Sep 17, 2018

cmsbuild added tests-started and removed tests-pending labels Sep 17, 2018

cmsbuild added comparison-available and removed comparison-pending labels Sep 18, 2018

cmsbuild added l1-approved and removed l1-pending labels Sep 18, 2018

rekovic mentioned this pull request Sep 18, 2018

Latest BMTF Unpacker (18 Sept 2018) cms-l1t-offline/cmssw#721

Merged

cmsbuild added dqm-approved fully-signed and removed dqm-pending pending-signatures labels Sep 18, 2018

cmsbuild added orp-approved and removed orp-pending labels Sep 23, 2018

cmsbuild merged commit 36633bb into cms-sw:master Sep 23, 2018

fabiocos mentioned this pull request Sep 24, 2018

RelVal failure for BMTF code update #24628

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

BMTF-KMTF DQM Algo Selector for the DQM/L1TMonitor Subsystem #24557

BMTF-KMTF DQM Algo Selector for the DQM/L1TMonitor Subsystem #24557

panoskatsoulis commented Sep 16, 2018

cmsbuild commented Sep 16, 2018

cmsbuild commented Sep 16, 2018

andrius-k commented Sep 17, 2018

cmsbuild commented Sep 17, 2018

cmsbuild commented Sep 17, 2018

cmsbuild commented Sep 17, 2018

panoskatsoulis commented Sep 17, 2018

andrius-k commented Sep 17, 2018

cmsbuild commented Sep 17, 2018 •

edited

cmsbuild commented Sep 18, 2018

rekovic commented Sep 18, 2018

andrius-k commented Sep 18, 2018

cmsbuild commented Sep 18, 2018

fabiocos commented Sep 23, 2018

Dr15Jones commented Sep 24, 2018

panoskatsoulis commented Sep 24, 2018 via email

Dr15Jones commented Sep 24, 2018

panoskatsoulis commented Sep 24, 2018 via email

panoskatsoulis commented Sep 24, 2018 via email

schneiml commented Sep 24, 2018

Martin-Grunewald commented Sep 25, 2018 •

edited

panoskatsoulis commented Sep 25, 2018

Martin-Grunewald commented Sep 25, 2018

panoskatsoulis commented Sep 25, 2018

Martin-Grunewald commented Sep 25, 2018

panoskatsoulis commented Sep 25, 2018

BMTF-KMTF DQM Algo Selector for the DQM/L1TMonitor Subsystem #24557

BMTF-KMTF DQM Algo Selector for the DQM/L1TMonitor Subsystem #24557

Conversation

panoskatsoulis commented Sep 16, 2018

cmsbuild commented Sep 16, 2018

cmsbuild commented Sep 16, 2018

andrius-k commented Sep 17, 2018

cmsbuild commented Sep 17, 2018

cmsbuild commented Sep 17, 2018

cmsbuild commented Sep 17, 2018

panoskatsoulis commented Sep 17, 2018

andrius-k commented Sep 17, 2018

cmsbuild commented Sep 17, 2018 • edited

cmsbuild commented Sep 18, 2018

rekovic commented Sep 18, 2018

andrius-k commented Sep 18, 2018

cmsbuild commented Sep 18, 2018

fabiocos commented Sep 23, 2018

Dr15Jones commented Sep 24, 2018

panoskatsoulis commented Sep 24, 2018 via email

Dr15Jones commented Sep 24, 2018

panoskatsoulis commented Sep 24, 2018 via email

panoskatsoulis commented Sep 24, 2018 via email

schneiml commented Sep 24, 2018

Martin-Grunewald commented Sep 25, 2018 • edited

panoskatsoulis commented Sep 25, 2018

Martin-Grunewald commented Sep 25, 2018

panoskatsoulis commented Sep 25, 2018

Martin-Grunewald commented Sep 25, 2018

panoskatsoulis commented Sep 25, 2018

cmsbuild commented Sep 17, 2018 •

edited

Martin-Grunewald commented Sep 25, 2018 •

edited