treat a predicted state too far as invalid in CosmicMuonSmoother (backport of #33168)

[slava77]

(copy-paste from the 11_3_X)

this is a follow-up to the origins of the issue reported in https://hypernews.cern.ch/HyperNews/CMS/get/recoDevelopment/1706.html
in processing of date in Run: 340323, which has a seg fault.

This is a case of runaway fit leading to running over numerical precision limits with a final result as NaN. Some later code using this can crash (fixed separately in #33157).

I propose to have this backport to avoid possible (rare) crashes in MWGR data (re)processing.
This PR is expected to affect/fix only cases that are going to lead to a crash or are very close to that state; this should be OK in the no-change policy.

[cmsbuild]

A new Pull Request was created by @slava77 (Slava Krutelyov) for CMSSW_11_2_X.

It involves the following packages:

RecoMuon/CosmicMuonProducer

@perrotta, @jpata, @cmsbuild, @slava77 can you please review it and eventually sign? Thanks.
@HuguesBrun, @bellan, @abbiendi, @Fedespring, @calderona, @sscruz, @jhgoh, @trocino, @cericeci, @rociovilar this is something you requested to watch as well.
@silviodonato, @dpiparo, @qliphy you are the release manager for this.

cms-bot commands are listed here

• Backported from treat a predicted state too far as invalid in CosmicMuonSmoother #33168

[slava77]

backport of #33168

[slava77]

@cmsbuild please test

[cmsbuild]

+1

Summary: https://cmssdt.cern.ch/SDT/jenkins-artifacts/pull-request-integration/PR-0d4897/13600/summary.html
COMMIT: 0fedef7
CMSSW: CMSSW_11_2_X_2021-03-18-1100/slc7_amd64_gcc900
User test area: For local testing, you can use /cvmfs/cms-ci.cern.ch/week0/cms-sw/cmssw/33214/13600/install.sh to create a dev area with all the needed externals and cmssw changes.

Comparison Summary

Summary:

• No significant changes to the logs found
• Reco comparison results: 0 differences found in the comparisons
• DQMHistoTests: Total files compared: 36
• DQMHistoTests: Total histograms compared: 2527501
• DQMHistoTests: Total failures: 1
• DQMHistoTests: Total nulls: 0
• DQMHistoTests: Total successes: 2527478
• DQMHistoTests: Total skipped: 22
• DQMHistoTests: Total Missing objects: 0
• DQMHistoSizes: Histogram memory added: 0.0 KiB( 35 files compared)
• Checked 151 log files, 37 edm output root files, 36 DQM output files
• TriggerResults: no differences found

[slava77]

+1

for #33214 0fedef7

• jenkins tests show no differences as expected
• the code update was earlier tested on the problematic events in run 340323 to confirm that the crash goes away

[cmsbuild]

This pull request is fully signed and it will be integrated in one of the next CMSSW_11_2_X IBs (tests are also fine) and once validation in the development release cycle CMSSW_11_3_X is complete. This pull request will now be reviewed by the release team before it's merged. @silviodonato, @dpiparo, @qliphy (and backports should be raised in the release meeting by the corresponding L2)

[silviodonato]

+1