Ape parallel features: Backport of #24192

[mteroerd]

Backport of #24192

This branch is based on a 10_2_X CMSSW version, but still incorporates the python print migration for changed and added files.

This PR consists of three commits:

b3856d6 fixes a small typo in one of the rarely used macros for validation plots.

33f7b47 consists of an update to the SkimProducer and AlignmentTrackSelector on one hand to ease the configuration of skims. On the other hand, a new python script startSkim.py was created to use alternatively to skimProducer.bash. This new script enables parallel execution of skims as well as automatic renaming of the output files (from *00N.root to *_(N+1).root which is the input name format needed for the APE measurements) and moving of the skimmed files to a target directory.

e43f208 adds additional optional input parameters to the central _cfg.py files used for a measurement. When using the standard workflow, these parameters are unused and nothing changes. However, the commit also includes a new folder autoSubmitter which includes a script to configure and submit multiple measurements at the same time in order to be able to do multiple measurements at once using only one CMSSW-environment.

[cmsbuild]

A new Pull Request was created by @mteroerd for CMSSW_10_2_X.

It involves the following packages:

Alignment/APEEstimation

@arunhep, @tocheng, @cmsbuild, @franzoni, @pohsun, @lpernie can you please review it and eventually sign? Thanks.
@mschrode, @mmusich, @tocheng this is something you requested to watch as well.
@davidlange6, @slava77, @fabiocos you are the release manager for this.

cms-bot commands are listed here

• Backported from Ape parallel features #24192

[pohsun]

please test

[cmsbuild]

The tests are being triggered in jenkins.
https://cmssdt.cern.ch/jenkins/job/ib-any-integration/29785/console Started: 2018/08/12 19:41

[cmsbuild]

-1

Tested at: e43f208

You can see the results of the tests here:
https://cmssdt.cern.ch/SDT/jenkins-artifacts/pull-request-integration/PR-24233/29785/summary.html

I found follow errors while testing this PR

Failed tests: RelVals

• RelVals:

When I ran the RelVals I found an error in the following worklfows:
4.22 step1

DAS Error

[cmsbuild]

Comparison not run due to runTheMatrix errors (RelVals and Igprof tests were also skipped)

[fabiocos]

please test

[cmsbuild]

The tests are being triggered in jenkins.
https://cmssdt.cern.ch/jenkins/job/ib-any-integration/29790/console Started: 2018/08/13 10:04

[cmsbuild]

+1
Tested at: e43f208
https://cmssdt.cern.ch/SDT/jenkins-artifacts/pull-request-integration/PR-24233/29790/summary.html

[cmsbuild]

Comparison job queued.

[cmsbuild]

Comparison is ready
https://cmssdt.cern.ch/SDT/jenkins-artifacts/pull-request-integration/PR-24233/29790/summary.html

Comparison Summary:

• No significant changes to the logs found
• Reco comparison results: 0 differences found in the comparisons
• DQMHistoTests: Total files compared: 31
• DQMHistoTests: Total histograms compared: 2892884
• DQMHistoTests: Total failures: 2
• DQMHistoTests: Total nulls: 0
• DQMHistoTests: Total successes: 2892692
• DQMHistoTests: Total skipped: 190
• DQMHistoTests: Total Missing objects: 0
• DQMHistoSizes: Histogram memory added: 0.0 KiB( 30 files compared)
• Checked 129 log files, 14 edm output root files, 31 DQM output files

[mteroerd]

Can you please review this? The rereco campaign apparently got delayed, but still it would be good to have the code in place when it is time.

[mmusich]

@arunhep, @tocheng, @pohsun, @lpernie
what is preventing from signing this PR?
It's a backport of #24192 which is merged since 20 days...

[lpernie]

+1

[cmsbuild]

This pull request is fully signed and it will be integrated in one of the next CMSSW_10_2_X IBs (tests are also fine) and once validation in the development release cycle CMSSW_10_3_X is complete. This pull request will now be reviewed by the release team before it's merged. @davidlange6, @slava77, @smuzaffar, @fabiocos (and backports should be raised in the release meeting by the corresponding L2)

[fabiocos]

+1