Add protection vs Minuit2 Fatal Root Error and update VxErrCorr in Vx3DHLTAnalyzer #38687

francescobrivio · 2022-07-11T19:09:48Z

PR description:

In recent Fills ( >=7920 ) the beampixel DQM client has been crashing with error:

An exception of category 'FatalRootError' occurred while
   [0] Processing global end LuminosityBlock run: 355443 luminosityBlock: 2
   [1] Calling method for module Vx3DHLTAnalyzer/'pixelVertexDQM'
   Additional Info:
      [a] Fatal Root Error: @SUB=Minuit2
VariableMetricBuilder Initial matrix not pos.def.

This PR:

updates the VxErrCorr parameter in Vx3DHLTAnalyzer from 1.2 to 1.0, similarly to the errorScale of the beam client which has been recently fixed in Update errorScale for BeamSpot Legacy DQM client #38632.
adds some try/catch protections are added in Vx3DHLTAnalyzer.cc in order to avoid crashes of the whole client in case of Fatal Errors coming from Minuit2 minimization
changes couts to edm:Log*s in Vx3DHLTAnalyzer

PR validation:

Code compiles.
Run privately on FEVT events and fit now converges, also, in case it fails, the Fatal Error is catched correctly by the LogError avoiding the crash of the beampixel client.

If this PR is a backport please specify the original PR and why you need to backport that PR. If this PR will be backported please specify to which release cycle the backport is meant for:

Not a backport, but 12_4_X and 12_3_X backports will be provided soon

FYI @dinardo

…3DHLTAnalyzer

francescobrivio · 2022-07-11T19:17:22Z

@cmsbuild please test

cmsbuild · 2022-07-11T19:17:51Z

+code-checks

Logs: https://cmssdt.cern.ch/SDT/code-checks/cms-sw-PR-38687/30988

This PR adds an extra 24KB to repository

cmsbuild · 2022-07-11T19:17:55Z

A new Pull Request was created by @francescobrivio for master.

It involves the following packages:

DQM/BeamMonitor (dqm, db)
DQM/Integration (dqm)

@malbouis, @pmandrik, @emanueleusai, @ahmad3213, @tvami, @jfernan2, @ggovi, @francescobrivio, @micsucmed, @rvenditti can you please review it and eventually sign? Thanks.
@mmusich, @threus, @batinkov, @battibass this is something you requested to watch as well.
@perrotta, @dpiparo, @qliphy, @rappoccio you are the release manager for this.

cms-bot commands are listed here

mmusich · 2022-07-11T19:44:37Z

DQM/BeamMonitor/plugins/Vx3DHLTAnalyzer.cc

-      Gauss3D->Minimize();
+      try {
+        Gauss3D->Minimize();
+      } catch (...) {


I think this construct is forbidden, or at least is flagged as such by the S/A

Yes, rule 7.11
https://cms-sw.github.io/cms_coding_rules.html

Following the coding rules I can try something like:

if (!Gauss3D->Minimize()) { throw cms::Exception("FitFailed") << "Vx3DHLTAnalyzer \tInitial matrix not pos. def"); }

Would this be better?

catching cms::Exception or std exceptions is allowed. What is not allowed is using catch(...)

addressed in 76fa9de

dinardo · 2022-07-11T20:14:45Z

Hi guys,
all i know is that if we don't catch the exception the code crashes.
And somehow the exception is generated inside ROOT.

dinardo · 2022-07-11T20:25:36Z

Ok cool.
What should we write then?

cmsbuild · 2022-07-11T23:11:23Z

+1

Summary: https://cmssdt.cern.ch/SDT/jenkins-artifacts/pull-request-integration/PR-bb6b2a/26146/summary.html
COMMIT: 60e5e71
CMSSW: CMSSW_12_5_X_2022-07-11-1100/el8_amd64_gcc10
User test area: For local testing, you can use /cvmfs/cms-ci.cern.ch/week1/cms-sw/cmssw/38687/26146/install.sh to create a dev area with all the needed externals and cmssw changes.

Comparison Summary

Summary:

No significant changes to the logs found
Reco comparison results: 4 differences found in the comparisons
DQMHistoTests: Total files compared: 50
DQMHistoTests: Total histograms compared: 3655970
DQMHistoTests: Total failures: 8
DQMHistoTests: Total nulls: 0
DQMHistoTests: Total successes: 3655940
DQMHistoTests: Total skipped: 22
DQMHistoTests: Total Missing objects: 0
DQMHistoSizes: Histogram memory added: 0.0 KiB( 49 files compared)
Checked 208 log files, 45 edm output root files, 50 DQM output files
TriggerResults: no differences found

francescobrivio · 2022-07-12T14:25:43Z

@cmsbuild please test

cmsbuild · 2022-07-12T14:32:53Z

+code-checks

Logs: https://cmssdt.cern.ch/SDT/code-checks/cms-sw-PR-38687/31001

This PR adds an extra 24KB to repository

cmsbuild · 2022-07-12T14:33:35Z

Pull request #38687 was updated. @malbouis, @pmandrik, @emanueleusai, @ahmad3213, @tvami, @jfernan2, @ggovi, @francescobrivio, @micsucmed, @rvenditti can you please check and sign again.

tvami · 2022-07-12T15:08:26Z

type bugfix

cmsbuild · 2022-07-12T18:24:24Z

+1

Summary: https://cmssdt.cern.ch/SDT/jenkins-artifacts/pull-request-integration/PR-bb6b2a/26171/summary.html
COMMIT: 76fa9de
CMSSW: CMSSW_12_5_X_2022-07-12-1100/el8_amd64_gcc10
User test area: For local testing, you can use /cvmfs/cms-ci.cern.ch/week1/cms-sw/cmssw/38687/26171/install.sh to create a dev area with all the needed externals and cmssw changes.

Comparison Summary

Summary:

No significant changes to the logs found
Reco comparison results: 8 differences found in the comparisons
DQMHistoTests: Total files compared: 50
DQMHistoTests: Total histograms compared: 3653734
DQMHistoTests: Total failures: 19
DQMHistoTests: Total nulls: 1
DQMHistoTests: Total successes: 3653692
DQMHistoTests: Total skipped: 22
DQMHistoTests: Total Missing objects: 0
DQMHistoSizes: Histogram memory added: 0.004 KiB( 49 files compared)
DQMHistoSizes: changed ( 312.0 ): 0.004 KiB MessageLogger/Warnings
Checked 208 log files, 45 edm output root files, 50 DQM output files
TriggerResults: no differences found

tvami · 2022-07-12T19:44:33Z

+db

tests pass
PR is according to the description

emanueleusai · 2022-07-13T04:17:02Z

+1

cmsbuild · 2022-07-13T04:17:23Z

This pull request is fully signed and it will be integrated in one of the next master IBs (tests are also fine). This pull request will now be reviewed by the release team before it's merged. @perrotta, @dpiparo, @qliphy, @rappoccio (and backports should be raised in the release meeting by the corresponding L2)

perrotta · 2022-07-13T05:28:15Z

+1

mmusich · 2022-08-23T10:14:16Z

DQM/BeamMonitor/plugins/Vx3DHLTAnalyzer.cc

+        try {
+          Gauss3D->Minimize();
+        } catch (cms::Exception& er) {
+          edm::LogError("Vx3DHLTAnalyzer") << "\tCaught Minuit2 exception: " << er.what();


given the problems recently observed (again) with the online DQM client (see log), after catching the exception and logging the error, shouldn't we return the function with an error state instead of carrying on?
I am not sure the rest of the computation make sense if the fit fails.

@vicha-w is there a way to reproduce offline the failure reported in the data-taking mattermost channel , e.g. by making available the input streamer files that are giving rise to the issue.

Hi @mmusich. Sorry I didn't reach out. I have given LS files that caused the error in beamspot clients to @francescobrivio this morning.

Hi @vicha-w sorry there is some confusion here: the streamer files you gave me this morning are NOT related to this issue, they are for different studies.

Add protection vs Minuit2 Fatal Root Error and update VxErrCorr in Vx…

60e5e71

…3DHLTAnalyzer

cmsbuild added this to the CMSSW_12_5_X milestone Jul 11, 2022

cmsbuild added code-checks-pending db-pending dqm-pending orp-pending pending-signatures tests-pending labels Jul 11, 2022

This was referenced Jul 11, 2022

[12_4_X] Add protection vs Minuit2 Fatal Root Error and update VxErrCorr in Vx3DHLTAnalyzer #38688

Merged

[12_3_X] Add protection vs Minuit2 Fatal Root Error and update VxErrCorr in Vx3DHLTAnalyzer #38689

Merged

cmsbuild added code-checks-approved tests-started and removed tests-pending code-checks-pending labels Jul 11, 2022

mmusich reviewed Jul 11, 2022

View reviewed changes

cmsbuild added tests-approved and removed tests-started labels Jul 11, 2022

apply review comments on try/catch in Vx3DHLTAnalyzer.cc

76fa9de

cmsbuild added code-checks-pending tests-pending and removed tests-approved code-checks-approved labels Jul 12, 2022

cmsbuild removed the tests-pending label Jul 12, 2022

cmsbuild added the tests-started label Jul 12, 2022

cmsbuild added code-checks-approved and removed code-checks-pending labels Jul 12, 2022

cmsbuild added the bug-fix label Jul 12, 2022

cmsbuild added tests-approved and removed tests-started labels Jul 12, 2022

cmsbuild added db-approved and removed db-pending labels Jul 12, 2022

cmsbuild mentioned this pull request Jul 12, 2022

Update GT for DQM clients unitTests and input file used for unitTest #38712

Merged

cmsbuild added dqm-approved fully-signed and removed dqm-pending pending-signatures labels Jul 13, 2022

cmsbuild added orp-approved and removed orp-pending labels Jul 13, 2022

cmsbuild merged commit 071e67f into cms-sw:master Jul 13, 2022

francescobrivio deleted the alca-fix_beamPixel branch July 19, 2022 13:19

mmusich reviewed Aug 23, 2022

View reviewed changes

francescobrivio mentioned this pull request Sep 1, 2022

Fix try/catch in Vx3DHLTAnalyzer #39285

Merged

mmusich mentioned this pull request Jun 20, 2023

Exception from AlcaBeamMonitor #42029

Open

smuzaffar mentioned this pull request Oct 11, 2023

CMSSW tests fails with Fatal Root Error: @SUB=Minuit2 #42979

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Add protection vs Minuit2 Fatal Root Error and update VxErrCorr in Vx3DHLTAnalyzer #38687

Add protection vs Minuit2 Fatal Root Error and update VxErrCorr in Vx3DHLTAnalyzer #38687

francescobrivio commented Jul 11, 2022

francescobrivio commented Jul 11, 2022

cmsbuild commented Jul 11, 2022

cmsbuild commented Jul 11, 2022

mmusich Jul 11, 2022

tvami Jul 11, 2022

francescobrivio Jul 11, 2022

mmusich Jul 11, 2022 •

edited

francescobrivio Jul 12, 2022

dinardo commented Jul 11, 2022

dinardo commented Jul 11, 2022

cmsbuild commented Jul 11, 2022

francescobrivio commented Jul 12, 2022

cmsbuild commented Jul 12, 2022

cmsbuild commented Jul 12, 2022

tvami commented Jul 12, 2022

cmsbuild commented Jul 12, 2022

tvami commented Jul 12, 2022

emanueleusai commented Jul 13, 2022

cmsbuild commented Jul 13, 2022

perrotta commented Jul 13, 2022

mmusich Aug 23, 2022 •

edited

mmusich Aug 23, 2022

francescobrivio Sep 1, 2022

vicha-w Sep 1, 2022

francescobrivio Sep 1, 2022

Add protection vs Minuit2 Fatal Root Error and update VxErrCorr in Vx3DHLTAnalyzer #38687

Add protection vs Minuit2 Fatal Root Error and update VxErrCorr in Vx3DHLTAnalyzer #38687

Conversation

francescobrivio commented Jul 11, 2022

PR description:

PR validation:

If this PR is a backport please specify the original PR and why you need to backport that PR. If this PR will be backported please specify to which release cycle the backport is meant for:

francescobrivio commented Jul 11, 2022

cmsbuild commented Jul 11, 2022

cmsbuild commented Jul 11, 2022

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

mmusich Jul 11, 2022 • edited

Choose a reason for hiding this comment

Choose a reason for hiding this comment

dinardo commented Jul 11, 2022

dinardo commented Jul 11, 2022

cmsbuild commented Jul 11, 2022

Comparison Summary

francescobrivio commented Jul 12, 2022

cmsbuild commented Jul 12, 2022

cmsbuild commented Jul 12, 2022

tvami commented Jul 12, 2022

cmsbuild commented Jul 12, 2022

Comparison Summary

tvami commented Jul 12, 2022

emanueleusai commented Jul 13, 2022

cmsbuild commented Jul 13, 2022

perrotta commented Jul 13, 2022

mmusich Aug 23, 2022 • edited

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

mmusich Jul 11, 2022 •

edited

mmusich Aug 23, 2022 •

edited