Throw a clear exception if OutputModule SelectEvents refers to a non-existent Path or Process #44767

makortel · 2024-04-17T19:53:05Z

PR description:

This was already the case for a non-existent Path of the present process. For non-existent Path of an earlier Process, or a non-existent Process, lead to a confusing exception message, and in some cases the job got stuck. An example of such stuck job was having an unrelated module in front of the OutputModule in the same EndPath.

Resolves #44743
Resolves #44744
Resolves cms-sw/framework-team#886

PR validation:

Added unit tests pass

cmsbuild · 2024-04-17T19:53:26Z

cms-bot internal usage

cmsbuild · 2024-04-17T20:00:07Z

+code-checks

Logs: https://cmssdt.cern.ch/SDT/code-checks/cms-sw-PR-44767/39985

This PR adds an extra 32KB to repository

cmsbuild · 2024-04-17T20:00:30Z

A new Pull Request was created by @makortel for master.

It involves the following packages:

FWCore/Framework (core)

@makortel, @smuzaffar, @Dr15Jones, @cmsbuild can you please review it and eventually sign? Thanks.
@missirol, @wddgit this is something you requested to watch as well.
@sextonkennedy, @rappoccio, @antoniovilela you are the release manager for this.

cms-bot commands are listed here

makortel · 2024-04-17T20:00:38Z

@cmsbuild, please test

makortel · 2024-04-17T20:00:50Z

@Dr15Jones Please review

cmsbuild · 2024-04-18T00:14:42Z

-1

Failed Tests: RelVals
Summary: https://cmssdt.cern.ch/SDT/jenkins-artifacts/pull-request-integration/PR-b3c1e7/38906/summary.html
COMMIT: 503657b
CMSSW: CMSSW_14_1_X_2024-04-17-1100/el8_amd64_gcc12
User test area: For local testing, you can use /cvmfs/cms-ci.cern.ch/week1/cms-sw/cmssw/44767/38906/install.sh to create a dev area with all the needed externals and cmssw changes.

RelVals

1001.0

----- Begin Fatal Exception 17-Apr-2024 22:44:09 CEST-----------------------
An exception of category 'Configuration' occurred while
   [0] Processing  Event run: 165121 lumi: 62 event: 23609118 stream: 0
   [1] Running path'ALCARECOStreamLumiPixelsMinBiasOutPath'
   [2] Calling OutputModule prePrefetchSelection()
Exception Message:
EventSelector::init, An OutputModule is using SelectEvents
to request a trigger name that does not exist
The unknown trigger name is: pathALCARECOLumiPixelsMinBias
----- End Fatal Exception -------------------------------------------------

cmsbuild · 2024-04-18T21:26:47Z

+code-checks

Logs: https://cmssdt.cern.ch/SDT/code-checks/cms-sw-PR-44767/40013

This PR adds an extra 40KB to repository
There are other open Pull requests which might conflict with changes you have proposed:
- File FWCore/Framework/interface/WorkerInPath.h modified in PR(s): Improve behavior after exception in begin/end stream lumi #44624

…existent Path or Process This was already the case for a non-existent Path of the present process. For non-existent Path of an earlier Process, or a non-existent Process, lead to a confusing exception message, and in some cases the job got stuck. An example of such stuck job was having an unrelated module in front of the OutputModule in the same EndPath.

Path::runNextWorkerAsync() pretty much assumes the runWorkerAsync() doesn't throw. runNextWorkerAsync() needs to schedule all eligible workers for the rest of the system to behave correctly.

cmsbuild · 2024-04-18T21:31:50Z

+code-checks

Logs: https://cmssdt.cern.ch/SDT/code-checks/cms-sw-PR-44767/40014

This PR adds an extra 40KB to repository
There are other open Pull requests which might conflict with changes you have proposed:
- File FWCore/Framework/interface/WorkerInPath.h modified in PR(s): Improve behavior after exception in begin/end stream lumi #44624

makortel · 2024-04-29T13:38:55Z

Is full_cmssw advisable here?

We can certainly do that, thanks for the suggestion. I was in any case waiting for #44787 to be merged before launching next round of tests (not much point testing before)

makortel · 2024-04-29T13:39:08Z

test parameters:

full_cmssw = true

makortel · 2024-04-30T14:30:42Z

@cmsbuild, please test

cmsbuild · 2024-04-30T20:37:51Z

+1

Summary: https://cmssdt.cern.ch/SDT/jenkins-artifacts/pull-request-integration/PR-b3c1e7/39180/summary.html
COMMIT: 27a4db4
CMSSW: CMSSW_14_1_X_2024-04-30-1100/el8_amd64_gcc12
User test area: For local testing, you can use /cvmfs/cms-ci.cern.ch/week1/cms-sw/cmssw/44767/39180/install.sh to create a dev area with all the needed externals and cmssw changes.

Comparison Summary

There are some workflows for which there are errors in the baseline:
29661.402 step 3
29661.403 step 3
29834.402 step 2
29834.403 step 2
The results for the comparisons for these workflows could be incomplete
This means most likely that the IB is having errors in the relvals.The error does NOT come from this pull request

Summary:

You potentially removed 49 lines from the logs
Reco comparison results: 47 differences found in the comparisons
DQMHistoTests: Total files compared: 48
DQMHistoTests: Total histograms compared: 3326929
DQMHistoTests: Total failures: 6
DQMHistoTests: Total nulls: 0
DQMHistoTests: Total successes: 3326903
DQMHistoTests: Total skipped: 20
DQMHistoTests: Total Missing objects: 0
DQMHistoSizes: Histogram memory added: 0.0 KiB( 47 files compared)
Checked 202 log files, 165 edm output root files, 48 DQM output files
TriggerResults: no differences found

makortel · 2024-04-30T21:56:28Z

There are some workflows for which there are errors in the baseline:
29661.402 step 3
29661.403 step 3
29834.402 step 2
29834.403 step 2

@smuzaffar These extra workflows were not run as part of the tests of this PR (or as part of IBs AFAICT). Are the failures reported in the test summary above perhaps because some other PR used the same baseline, and those extra workflows were requested for the tests of that PR?

makortel · 2024-04-30T21:57:38Z

Seems like this PR would be good to go. @Dr15Jones could you take a last look?

smuzaffar · 2024-04-30T22:24:28Z

Are the failures reported in the test summary above perhaps because some other PR used the same baseline, and those extra workflows were requested for the tests of that PR?

yes looks like these extra workflows were requested by #44874 (comment). Bot should have ignore these workflow failures for this PR tests. I will look in to this

FWCore/Framework/interface/WorkerInPath.h

makortel · 2024-05-01T14:33:34Z

+core

cmsbuild · 2024-05-01T14:33:57Z

This pull request is fully signed and it will be integrated in one of the next master IBs (tests are also fine). This pull request will now be reviewed by the release team before it's merged. @sextonkennedy, @rappoccio, @antoniovilela (and backports should be raised in the release meeting by the corresponding L2)

rappoccio · 2024-05-01T19:58:34Z

+1

smuzaffar · 2024-05-02T14:08:36Z

yes looks like these extra workflows were requested by #44874 (comment). Bot should have ignore these workflow failures for this PR tests. I will look in to this

cms-sw/cms-bot#2221 should fix this for future PR tests

cmsbuild added this to the CMSSW_14_1_X milestone Apr 17, 2024

cmsbuild added core-pending pending-signatures tests-pending orp-pending code-checks-pending labels Apr 17, 2024

cmsbuild added code-checks-approved and removed code-checks-pending labels Apr 17, 2024

cmsbuild added tests-started and removed tests-pending labels Apr 17, 2024

cmsbuild added tests-rejected tests-pending code-checks-pending and removed tests-started tests-rejected code-checks-approved labels Apr 18, 2024

makortel mentioned this pull request Apr 18, 2024

Job gets stuck if OutputModuleCore::prePrefetchSelection() throws an exception #44744

Closed

makortel added 2 commits April 18, 2024 16:26

Mark WorkerInPath::runWorkerAsync() as noexcept

27a4db4

Path::runNextWorkerAsync() pretty much assumes the runWorkerAsync() doesn't throw. runNextWorkerAsync() needs to schedule all eligible workers for the rest of the system to behave correctly.

makortel force-pushed the preprefetchException branch from 332c1a4 to 27a4db4 Compare April 18, 2024 21:27

cmsbuild added code-checks-approved and removed code-checks-pending labels Apr 18, 2024

cmsbuild added tests-started and removed tests-pending labels Apr 30, 2024

cmsbuild added tests-approved and removed tests-started labels Apr 30, 2024

Dr15Jones reviewed May 1, 2024

View reviewed changes

FWCore/Framework/interface/WorkerInPath.h Show resolved Hide resolved

cmsbuild added core-approved fully-signed and removed core-pending pending-signatures labels May 1, 2024

cmsbuild added orp-approved and removed orp-pending labels May 1, 2024

cmsbuild merged commit e71321d into cms-sw:master May 1, 2024
12 checks passed

makortel deleted the preprefetchException branch May 1, 2024 20:10

iarspider mentioned this pull request May 2, 2024

Assertion failure in checkResultsOfRunWorker in RelVals 1001.3, 1040.1, 1041.0, 1042.0 #44887

Closed

This was referenced May 2, 2024

Revert "Throw a clear exception if OutputModule SelectEvents refers to a non-existent Path or Process" #44889

Closed

ASAN: heap-use-after-free in edm::Worker::prePrefetchSelectionAsync #44888

Closed

smuzaffar mentioned this pull request May 2, 2024

ignore ref wf failures if Pr tests did not run that wf cms-sw/cms-bot#2221

Merged

makortel mentioned this pull request May 2, 2024

Fix exception handling in Worker::prePrefetchSelectionAsync() #44891

Merged

perrotta mentioned this pull request May 3, 2024

Multiple RelVals: Unknown trigger name exception #44895

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Throw a clear exception if OutputModule SelectEvents refers to a non-existent Path or Process #44767

Throw a clear exception if OutputModule SelectEvents refers to a non-existent Path or Process #44767

makortel commented Apr 17, 2024

cmsbuild commented Apr 17, 2024 •

edited

cmsbuild commented Apr 17, 2024

cmsbuild commented Apr 17, 2024

makortel commented Apr 17, 2024

makortel commented Apr 17, 2024

cmsbuild commented Apr 18, 2024

cmsbuild commented Apr 18, 2024

cmsbuild commented Apr 18, 2024

makortel commented Apr 29, 2024

makortel commented Apr 29, 2024

makortel commented Apr 30, 2024

cmsbuild commented Apr 30, 2024

makortel commented Apr 30, 2024

makortel commented Apr 30, 2024

smuzaffar commented Apr 30, 2024

makortel commented May 1, 2024

cmsbuild commented May 1, 2024

rappoccio commented May 1, 2024

smuzaffar commented May 2, 2024

Throw a clear exception if OutputModule SelectEvents refers to a non-existent Path or Process #44767

Throw a clear exception if OutputModule SelectEvents refers to a non-existent Path or Process #44767

Conversation

makortel commented Apr 17, 2024

PR description:

PR validation:

cmsbuild commented Apr 17, 2024 • edited

cmsbuild commented Apr 17, 2024

cmsbuild commented Apr 17, 2024

makortel commented Apr 17, 2024

makortel commented Apr 17, 2024

cmsbuild commented Apr 18, 2024

RelVals

cmsbuild commented Apr 18, 2024

cmsbuild commented Apr 18, 2024

makortel commented Apr 29, 2024

makortel commented Apr 29, 2024

makortel commented Apr 30, 2024

cmsbuild commented Apr 30, 2024

Comparison Summary

makortel commented Apr 30, 2024

makortel commented Apr 30, 2024

smuzaffar commented Apr 30, 2024

makortel commented May 1, 2024

cmsbuild commented May 1, 2024

rappoccio commented May 1, 2024

smuzaffar commented May 2, 2024

cmsbuild commented Apr 17, 2024 •

edited