[TsosGaussianStateConversions] Require GSF states to be positive definite [12_6_X] #39873

swagata87 · 2022-10-27T13:38:46Z

PR description:

This PR is to solve a crash in prompt-reco that led to a paused job as reported in https://cms-talk.web.cern.ch/t/logic-error-in-reco-job-for-run-360888-dataset-parkingdoublemuonlowmass2/16641, and discussed in #39570.

It was checked, from 12_4_X, that this patch cures the crash in low-pT electron reconstruction.

PR validation:

runTheMatrix.py -l 12434.0 ran fine.

From 12_4_X, it was checked that [Base] and [Base+thisPR] leads to same number of electrons, photons and low-pT electrons to be reconstructed, with same pT spectra. This check was made by running on 200 raw events on this file: /eos/cms/tier0/store/data/Run2022F/EGamma/RAW/v1/000/361/197/00000/76bd97fa-4ad6-4d85-b941-014e3ed27f9c.root

Tagging @francescobrivio as this week's ORM.

Backport of this PR might be necessary.

cmsbuild · 2022-10-27T13:46:57Z

+code-checks

Logs: https://cmssdt.cern.ch/SDT/code-checks/cms-sw-PR-39873/32784

This PR adds an extra 12KB to repository

cmsbuild · 2022-10-27T13:47:18Z

A new Pull Request was created by @swagata87 (Swagata Mukherjee) for master.

It involves the following packages:

TrackingTools/GsfTracking (reconstruction)

@cmsbuild, @mandrenguyen, @clacaputo can you please review it and eventually sign? Thanks.
@VourMa, @bellan, @felicepantaleo, @GiacomoSguazzoni, @JanFSchulte, @rovere, @VinInn, @missirol, @ebrondol, @lecriste, @gpetruc, @mmusich, @mtosi, @dgulhan this is something you requested to watch as well.
@perrotta, @dpiparo, @rappoccio you are the release manager for this.

cms-bot commands are listed here

francescobrivio · 2022-10-27T13:52:05Z

assign tracking-pog

Since this touches tracking code

cmsbuild · 2022-10-27T13:52:25Z

New categories assigned: tracking-pog

@slava77,@mmusich you have been requested to review this Pull request/Issue and eventually sign? Thanks

mmusich · 2022-10-27T13:57:11Z

please test

TrackingTools/GsfTracking/src/TsosGaussianStateConversions.cc

cmsbuild · 2022-10-27T18:28:58Z

-1

Failed Tests: RelVals-INPUT
Summary: https://cmssdt.cern.ch/SDT/jenkins-artifacts/pull-request-integration/PR-94fd56/28561/summary.html
COMMIT: 57a8f0a
CMSSW: CMSSW_12_6_X_2022-10-27-1100/el8_amd64_gcc10
User test area: For local testing, you can use /cvmfs/cms-ci.cern.ch/week0/cms-sw/cmssw/39873/28561/install.sh to create a dev area with all the needed externals and cmssw changes.

RelVals-INPUT

The relvals timed out after 4 hours.

Comparison Summary

Summary:

No significant changes to the logs found
Reco comparison results: 5 differences found in the comparisons
Reco comparison had 6 failed jobs
DQMHistoTests: Total files compared: 48
DQMHistoTests: Total histograms compared: 3384029
DQMHistoTests: Total failures: 6
DQMHistoTests: Total nulls: 0
DQMHistoTests: Total successes: 3384001
DQMHistoTests: Total skipped: 22
DQMHistoTests: Total Missing objects: 0
DQMHistoSizes: Histogram memory added: 0.0 KiB( 47 files compared)
Checked 201 log files, 48 edm output root files, 48 DQM output files
TriggerResults: no differences found

mmusich · 2022-10-28T06:49:39Z

please test

let's try again

cmsbuild · 2022-10-28T11:23:30Z

+1

Summary: https://cmssdt.cern.ch/SDT/jenkins-artifacts/pull-request-integration/PR-94fd56/28589/summary.html
COMMIT: 57a8f0a
CMSSW: CMSSW_12_6_X_2022-10-27-2300/el8_amd64_gcc10
User test area: For local testing, you can use /cvmfs/cms-ci.cern.ch/week0/cms-sw/cmssw/39873/28589/install.sh to create a dev area with all the needed externals and cmssw changes.

Comparison Summary

Summary:

No significant changes to the logs found
Reco comparison results: 6 differences found in the comparisons
DQMHistoTests: Total files compared: 48
DQMHistoTests: Total histograms compared: 3384029
DQMHistoTests: Total failures: 9
DQMHistoTests: Total nulls: 0
DQMHistoTests: Total successes: 3383998
DQMHistoTests: Total skipped: 22
DQMHistoTests: Total Missing objects: 0
DQMHistoSizes: Histogram memory added: 0.0 KiB( 47 files compared)
Checked 201 log files, 48 edm output root files, 48 DQM output files
TriggerResults: no differences found

mmusich · 2022-10-28T12:01:59Z

+1

tested explicitly it fixes the issue reported in the cmsTalk¹ ;
no differences seen in the tests;
the GSF tracking code could use some revision as the criteria of or positive definiteness are not harmonised across different modules (see [TsosGaussianStateConversions] Require GSF states to be positive definite [12_6_X] #39873 (comment)), perhaps the author could open a gitHub issue with the details to be reminded about it;

using the recipe at https://github.com/cms-sw/cmssw/issues/35929#issuecomment-1288800329 ↩

francescobrivio · 2022-10-28T12:05:15Z

urgent

@cms-sw/reconstruction-l2 please give an high priority to the review of this PR which is then needed to fix crashes in Prompt reconstruction in Tier0

mmusich · 2022-10-28T12:10:49Z

type egamma, tracking

swagata87 · 2022-10-30T11:12:01Z

This fix is useful for HLT also, see #39570 (comment).
I expect the HLT Gsf crashes to go away or happen less frequently after this fix is integrated.

clacaputo · 2022-10-31T12:14:28Z

+reconstruction

PR addresses crash in prompt-reco that led to a paused job
no reco changes observed

cmsbuild · 2022-10-31T12:14:49Z

This pull request is fully signed and it will be integrated in one of the next master IBs (tests are also fine). This pull request will now be reviewed by the release team before it's merged. @perrotta, @dpiparo, @rappoccio (and backports should be raised in the release meeting by the corresponding L2)

rappoccio · 2022-10-31T12:23:07Z

+1

Urgent fix for datataking. Once this clears IBs we will merge the backport and cut a new 12_4.

rappoccio · 2022-10-31T12:24:18Z

Backports are in #39903 and #39904

mmusich · 2022-11-04T09:53:49Z

For the record, when the release containing the backport of this PR (#39903) has been tested at tier0, a job that previously was giving an exception, is now having a segfault https://cms-talk.web.cern.ch/t/replay-testing-for-cmssw-12-4-11/17062/4
FYI: @swagata87

swagata87 · 2022-11-04T10:33:26Z

Hello @mmusich

I copied the folder as indicated in the cmsTalk link you shared, /afs/cern.ch/user/c/cmst0/public/PausedJobs/Replay12_4_11/job_1175

From what is reported there, Run: 361239 Event: 453401426 crashed, right?
So I tried to run on that event, by doing this

import FWCore.ParameterSet.Config as cms
from PSet import process
process.source.eventsToProcess = cms.untracked.VEventRange('361239:453401426-361239:453401426')

But it did not crash.
I am in CMSSW_12_4_11 and slc7_amd64_gcc10.

Am I missing something?
Can someone else try to see if the crash is reproducible in some other machine?

mmusich · 2022-11-04T10:38:40Z

Hi Swagata @swagata87,
I am trying to isolate the event that crashes the process as well. Since the process is run multi-threaded, it's not necessarily the last event printed out in the log that caused the crash. One should look which stream underwent the crash.

slava77 · 2022-11-04T13:13:19Z

The t0 logs say

%MSG-e GsfMultiStateUpdator:  GsfTrackProducer:lowPtGsfEleGsfTracks  04-Nov-2022 08:44:48 CET Run: 361239 Event: 453401426
KF updated state 4 is invalid. skipping.
%MSG

for 0-5. How many are tried? Is it a case of everything failing and the code not expecting that?

swagata87 · 2022-11-04T13:34:40Z

From the file from T0, job_1175/job/WMTaskSpace/cmsRun1/cmsRun1-stdout.log; all of 0-5 failed [1].
So it could be that we need another protection somewhere for a scenario where nothing pass. But we could not locate the event yet.. if we can find the exact event that crashed, we would look for a fix. From the log below it looks like Event 453474850 , but this event runs fine from my area.

[1]

Begin processing the 5984th record. Run 361239, Event 453474850, LumiSection 247 on stream 0 at 04-Nov-2022 08:44:47.720 CET
%MSG-e GsfMultiStateUpdator:  GsfTrackProducer:lowPtGsfEleGsfTracks  04-Nov-2022 08:44:48 CET Run: 361239 Event: 453401426
KF updated state 0 is invalid. skipping.
%MSG
%MSG-e GsfMultiStateUpdator:  GsfTrackProducer:lowPtGsfEleGsfTracks  04-Nov-2022 08:44:48 CET Run: 361239 Event: 453401426
KF updated state 1 is invalid. skipping.
%MSG
%MSG-e GsfMultiStateUpdator:  GsfTrackProducer:lowPtGsfEleGsfTracks  04-Nov-2022 08:44:48 CET Run: 361239 Event: 453401426
KF updated state 2 is invalid. skipping.
%MSG
%MSG-e GsfMultiStateUpdator:  GsfTrackProducer:lowPtGsfEleGsfTracks  04-Nov-2022 08:44:48 CET Run: 361239 Event: 453401426
KF updated state 3 is invalid. skipping.
%MSG
%MSG-e GsfMultiStateUpdator:  GsfTrackProducer:lowPtGsfEleGsfTracks  04-Nov-2022 08:44:48 CET Run: 361239 Event: 453401426
KF updated state 4 is invalid. skipping.
%MSG
%MSG-e GsfMultiStateUpdator:  GsfTrackProducer:lowPtGsfEleGsfTracks  04-Nov-2022 08:44:48 CET Run: 361239 Event: 453401426
KF updated state 5 is invalid. skipping.
%MSG
%MSG-e GsfMultiStateUpdator:  GsfTrackProducer:lowPtGsfEleGsfTracks  04-Nov-2022 08:44:48 CET Run: 361239 Event: 453401426
KF updated state 0 is invalid. skipping.
%MSG
%MSG-e GsfMultiStateUpdator:  GsfTrackProducer:lowPtGsfEleGsfTracks  04-Nov-2022 08:44:48 CET Run: 361239 Event: 453401426
KF updated state 1 is invalid. skipping.
%MSG
%MSG-e GsfMultiStateUpdator:  GsfTrackProducer:lowPtGsfEleGsfTracks  04-Nov-2022 08:44:48 CET Run: 361239 Event: 453401426
KF updated state 2 is invalid. skipping.
%MSG
%MSG-e GsfMultiStateUpdator:  GsfTrackProducer:lowPtGsfEleGsfTracks  04-Nov-2022 08:44:48 CET Run: 361239 Event: 453401426
KF updated state 3 is invalid. skipping.
%MSG
%MSG-e GsfMultiStateUpdator:  GsfTrackProducer:lowPtGsfEleGsfTracks  04-Nov-2022 08:44:48 CET Run: 361239 Event: 453401426
KF updated state 4 is invalid. skipping.
%MSG
%MSG-e GsfMultiStateUpdator:  GsfTrackProducer:lowPtGsfEleGsfTracks  04-Nov-2022 08:44:48 CET Run: 361239 Event: 453401426
KF updated state 5 is invalid. skipping.
%MSG

A fatal system signal has occurred: segmentation violation
The following is the call stack containing the origin of the signal.

mmusich · 2022-11-04T13:43:50Z

follow-up issue at #39987 is perhaps a better place to continue the discussion.

require states to be pos-def

57a8f0a

cmsbuild added this to the CMSSW_12_6_X milestone Oct 27, 2022

cmsbuild added code-checks-pending orp-pending pending-signatures reconstruction-pending tests-pending labels Oct 27, 2022

cmsbuild added code-checks-approved and removed code-checks-pending labels Oct 27, 2022

cmsbuild added the tracking-pog-pending label Oct 27, 2022

cmsbuild added tests-started and removed tests-pending labels Oct 27, 2022

mmusich reviewed Oct 27, 2022

View reviewed changes

TrackingTools/GsfTracking/src/TsosGaussianStateConversions.cc Show resolved Hide resolved

cmsbuild added tests-rejected and removed tests-started labels Oct 27, 2022

cmsbuild added tests-started and removed tests-rejected labels Oct 28, 2022

cmsbuild added tests-approved and removed tests-started labels Oct 28, 2022

cmsbuild added tracking-pog-approved and removed tracking-pog-pending labels Oct 28, 2022

This was referenced Oct 28, 2022

[12.4.X] TsosGaussianStateConversions Require GSF states to be positive definite #39903

Merged

[12.5.X] TsosGaussianStateConversions Require GSF states to be positive definite #39904

Merged

cmsbuild added the urgent label Oct 28, 2022

cmsbuild added egamma tracking labels Oct 28, 2022

swagata87 mentioned this pull request Oct 30, 2022

GsfTracking crash at HLT in run 359686 #39570

Closed

cmsbuild added fully-signed reconstruction-approved and removed reconstruction-pending pending-signatures labels Oct 31, 2022

rappoccio mentioned this pull request Oct 31, 2022

Homogenize positive-definite criteria in GSF code #39933

Open

cmsbuild added orp-approved and removed orp-pending labels Oct 31, 2022

cmsbuild merged commit d624d70 into cms-sw:master Oct 31, 2022

This was referenced Oct 31, 2022

[MkFit] Phase2 geometry & String configurable standard functions #39866

Merged

Introduction of New-All-In-One tool for Offline Validation of TrkAlignment #38304

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[TsosGaussianStateConversions] Require GSF states to be positive definite [12_6_X] #39873

[TsosGaussianStateConversions] Require GSF states to be positive definite [12_6_X] #39873

swagata87 commented Oct 27, 2022

cmsbuild commented Oct 27, 2022

cmsbuild commented Oct 27, 2022

francescobrivio commented Oct 27, 2022

cmsbuild commented Oct 27, 2022

mmusich commented Oct 27, 2022

cmsbuild commented Oct 27, 2022

mmusich commented Oct 28, 2022

cmsbuild commented Oct 28, 2022

mmusich commented Oct 28, 2022 •

edited

francescobrivio commented Oct 28, 2022

mmusich commented Oct 28, 2022

swagata87 commented Oct 30, 2022

clacaputo commented Oct 31, 2022 •

edited

cmsbuild commented Oct 31, 2022

rappoccio commented Oct 31, 2022

rappoccio commented Oct 31, 2022

mmusich commented Nov 4, 2022

swagata87 commented Nov 4, 2022 •

edited

mmusich commented Nov 4, 2022

slava77 commented Nov 4, 2022

swagata87 commented Nov 4, 2022 •

edited

mmusich commented Nov 4, 2022

[TsosGaussianStateConversions] Require GSF states to be positive definite [12_6_X] #39873

[TsosGaussianStateConversions] Require GSF states to be positive definite [12_6_X] #39873

Conversation

swagata87 commented Oct 27, 2022

PR description:

PR validation:

cmsbuild commented Oct 27, 2022

cmsbuild commented Oct 27, 2022

francescobrivio commented Oct 27, 2022

cmsbuild commented Oct 27, 2022

mmusich commented Oct 27, 2022

cmsbuild commented Oct 27, 2022

RelVals-INPUT

Comparison Summary

mmusich commented Oct 28, 2022

cmsbuild commented Oct 28, 2022

Comparison Summary

mmusich commented Oct 28, 2022 • edited

Footnotes

francescobrivio commented Oct 28, 2022

mmusich commented Oct 28, 2022

swagata87 commented Oct 30, 2022

clacaputo commented Oct 31, 2022 • edited

cmsbuild commented Oct 31, 2022

rappoccio commented Oct 31, 2022

rappoccio commented Oct 31, 2022

mmusich commented Nov 4, 2022

swagata87 commented Nov 4, 2022 • edited

mmusich commented Nov 4, 2022

slava77 commented Nov 4, 2022

swagata87 commented Nov 4, 2022 • edited

mmusich commented Nov 4, 2022

mmusich commented Oct 28, 2022 •

edited

clacaputo commented Oct 31, 2022 •

edited

swagata87 commented Nov 4, 2022 •

edited

swagata87 commented Nov 4, 2022 •

edited