Allow intra-Path parallelism

[Dr15Jones]

PR description:

The modules after an EDFilter and before the next EDFilter are allowed to run concurrently. This can be prevented by using cms.wait() on the module in the configuration.

PR validation:

All framework related unit tests pass.

[cmsbuild]

The code-checks are being triggered in jenkins.

[cmsbuild]

+code-checks

Logs: https://cmssdt.cern.ch/SDT/code-checks/cms-sw-PR-29024/13874

• This PR adds an extra 96KB to repository

[cmsbuild]

A new Pull Request was created by @Dr15Jones (Chris Jones) for master.

It involves the following packages:

FWCore/Framework
FWCore/Integration
FWCore/ParameterSet

@makortel, @smuzaffar, @cmsbuild, @Dr15Jones can you please review it and eventually sign? Thanks.
@makortel, @wddgit this is something you requested to watch as well.
@davidlange6, @silviodonato, @fabiocos you are the release manager for this.

cms-bot commands are listed here

[Dr15Jones]

please test

[cmsbuild]

The tests are being triggered in jenkins.
https://cmssdt.cern.ch/jenkins/job/ib-run-pr-tests/4822/console Started: 2020/02/21 23:44

[cmsbuild]

+1
Tested at: 379cb94
https://cmssdt.cern.ch/SDT/jenkins-artifacts/pull-request-integration/PR-f21bd4/4822/summary.html
CMSSW: CMSSW_11_1_X_2020-02-21-1100
SCRAM_ARCH: slc7_amd64_gcc820

[cmsbuild]

Comparison job queued.

[cmsbuild]

Comparison is ready
https://cmssdt.cern.ch/SDT/jenkins-artifacts/pull-request-integration/PR-f21bd4/4822/summary.html

Comparison Summary:

• No significant changes to the logs found
• Reco comparison results: 4 differences found in the comparisons
• DQMHistoTests: Total files compared: 34
• DQMHistoTests: Total histograms compared: 2694086
• DQMHistoTests: Total failures: 2
• DQMHistoTests: Total nulls: 0
• DQMHistoTests: Total successes: 2693765
• DQMHistoTests: Total skipped: 319
• DQMHistoTests: Total Missing objects: 0
• DQMHistoSizes: Histogram memory added: 0.0 KiB( 33 files compared)
• Checked 147 log files, 16 edm output root files, 34 DQM output files

[silviodonato]

Do you have any comments? @makortel @smuzaffar

[makortel]

@fwyzard, can you comment if FastTimerService makes assumptions that modules in Paths are actually run in that specific order, or that at most one module per Path could be running concurrently? By quick look on the code it doesn't look like that, but I could have easily missed something.

[fwyzard]

IIRC, the design should assume that any number of modules can run at the same time.

The only interaction with Paths is through the pre|postPathEvent, so the order and number of running modules should not matter.

[fwyzard]

By the way, can different Paths already run in parallel ?

[cmsbuild]

+code-checks

Logs: https://cmssdt.cern.ch/SDT/code-checks/cms-sw-PR-29024/13917

• This PR adds an extra 96KB to repository

[cmsbuild]

Pull request #29024 was updated. @makortel, @smuzaffar, @cmsbuild, @Dr15Jones can you please check and sign again.

[Dr15Jones]

please test

[cmsbuild]

The tests are being triggered in jenkins.
https://cmssdt.cern.ch/jenkins/job/ib-run-pr-tests/4901/console Started: 2020/02/26 20:03

[cmsbuild]

+1
Tested at: 14f9ff5
https://cmssdt.cern.ch/SDT/jenkins-artifacts/pull-request-integration/PR-f21bd4/4901/summary.html
CMSSW: CMSSW_11_1_X_2020-02-26-1100
SCRAM_ARCH: slc7_amd64_gcc820

[cmsbuild]

Comparison job queued.

[cmsbuild]

Comparison is ready
https://cmssdt.cern.ch/SDT/jenkins-artifacts/pull-request-integration/PR-f21bd4/4901/summary.html

@slava77 comparisons for the following workflows were not done due to missing matrix map:

• /data/cmsbld/jenkins/workspace/compare-root-files-short-matrix/data/PR-f21bd4/11634.0_TTbar_14TeV+TTbar_14TeV_TuneCP5_2021_GenSimFull+DigiFull_2021+RecoFull_2021+HARVESTFull_2021+ALCAFull_2021
• /data/cmsbld/jenkins/workspace/compare-root-files-short-matrix/data/PR-f21bd4/12434.0_TTbar_14TeV+TTbar_14TeV_TuneCP5_2023_GenSimFull+DigiFull_2023+RecoFull_2023+HARVESTFull_2023+ALCAFull_2023
• /data/cmsbld/jenkins/workspace/compare-root-files-short-matrix/data/PR-f21bd4/20034.0_TTbar_14TeV+TTbar_14TeV_TuneCP5_2026D35_GenSimHLBeamSpotFull14+DigiFullTrigger_2026D35+RecoFullGlobal_2026D35+HARVESTFullGlobal_2026D35
• /data/cmsbld/jenkins/workspace/compare-root-files-short-matrix/data/PR-f21bd4/20434.0_TTbar_14TeV+TTbar_14TeV_TuneCP5_2026D41_GenSimHLBeamSpotFull14+DigiFullTrigger_2026D41+RecoFullGlobal_2026D41+HARVESTFullGlobal_2026D41
• /data/cmsbld/jenkins/workspace/compare-root-files-short-matrix/data/PR-f21bd4/21234.0_TTbar_14TeV+TTbar_14TeV_TuneCP5_2026D44_GenSimHLBeamSpotFull14+DigiFullTrigger_2026D44+RecoFullGlobal_2026D44+HARVESTFullGlobal_2026D44
• /data/cmsbld/jenkins/workspace/compare-root-files-short-matrix/data/PR-f21bd4/23234.0_TTbar_14TeV+TTbar_14TeV_TuneCP5_2026D49_GenSimHLBeamSpotFull14+DigiFullTrigger_2026D49+RecoFullGlobal_2026D49+HARVESTFullGlobal_2026D49

Comparison Summary:

• No significant changes to the logs found
• Reco comparison results: 4 differences found in the comparisons
• DQMHistoTests: Total files compared: 34
• DQMHistoTests: Total histograms compared: 2679706
• DQMHistoTests: Total failures: 2
• DQMHistoTests: Total nulls: 0
• DQMHistoTests: Total successes: 2679385
• DQMHistoTests: Total skipped: 319
• DQMHistoTests: Total Missing objects: 0
• DQMHistoSizes: Histogram memory added: 0.0 KiB( 33 files compared)
• Checked 147 log files, 16 edm output root files, 34 DQM output files

[makortel]

+1

[makortel]

comparisons for the following workflows were not done due to missing matrix map:

Are these workflows (11634.0, 12434.0, 20034.0, 20434.0, 21234.0, 23234.0) recent additions to the PR test? @slava77 should these be added to the "matrix map"?

(just wondering out loud)

[cmsbuild]

This pull request is fully signed and it will be integrated in one of the next master IBs (tests are also fine). This pull request will now be reviewed by the release team before it's merged. @davidlange6, @silviodonato, @fabiocos (and backports should be raised in the release meeting by the corresponding L2)

[silviodonato]

+1