Added support for condor instead of LSF for APE tool

[mteroerd]

As LSF will soon be decommissioned, an update was made to the APE tool to use condor jobs instead.

Additionally, automated generation of result plots was added, along with some basic support for including systematic uncertainties in those plots. In the long run, these python scripts will probably replace the existing files in macros/.

The batch scripts used to submit files to LSF were removed.

Some additional minor tweaks were done to test/autoSubmitter/autoSubmitter.py (adding the possibility to use shortcuts for mp, hp, sm, and prod when loading conditions) and test/batch/startSkim.py (adding option to limit the number of parallel skims).

[cmsbuild]

The code-checks are being triggered in jenkins.

[cmsbuild]

+code-checks

Logs: https://cmssdt.cern.ch/SDT/code-checks/cms-sw-PR-25287/7307

[cmsbuild]

A new Pull Request was created by @mteroerd (Marius Teroerde) for master.

It involves the following packages:

Alignment/APEEstimation

@cmsbuild, @franzoni, @pohsun, @tocheng, @lpernie can you please review it and eventually sign? Thanks.
@mschrode, @mmusich, @tocheng, @adewit this is something you requested to watch as well.
@davidlange6, @slava77, @fabiocos you are the release manager for this.

cms-bot commands are listed here

[hroskes]

Thanks Marius! Does anyone know if we will need backports for the 2017 and 2016 ultra legacy rereco?

[boudoul]

Hi @hroskes the UL will be performed with >=10_4_X (independently on the year : 2018, 2017, 2016) , so you are safe .
On the other hand are you sure you won't need this APE tool to run directly on HI data/MC done with 10_3 in the near future?

[lpernie]

please test

[cmsbuild]

The tests are being triggered in jenkins.
https://cmssdt.cern.ch/jenkins/job/ib-any-integration/31763/console Started: 2018/11/20 15:30

[cmsbuild]

-1

Tested at: 927e11d

You can see the results of the tests here:
https://cmssdt.cern.ch/SDT/jenkins-artifacts/pull-request-integration/PR-25287/31763/summary.html

I found follow errors while testing this PR

Failed tests: RelVals

• RelVals:

The relvals timed out after 2 hours.

[cmsbuild]

Comparison not run due to runTheMatrix errors (RelVals and Igprof tests were also skipped)

[mmusich]

@boudoul @hroskes

On the other hand are you sure you won't need this APE tool to run directly on HI data/MC done with 10_3 in the near future?

While it is generally desirable to start moving the job submission infrastructure to condor, based on this message: https://hypernews.cern.ch/HyperNews/CMS/get/calibrations/3989/1/1/1.html
it seems this development won't be needed (urgently) before Summer 2019 (if Marius is using caf queues, as I hope he does), hence not needed for 10.3

[mteroerd]

Indeed the current version of the ape tool uses the caf queue.

[fabiocos]

please test

[cmsbuild]

The tests are being triggered in jenkins.
https://cmssdt.cern.ch/jenkins/job/ib-any-integration/31830/console Started: 2018/11/23 19:21

[cmsbuild]

+1
Tested at: 927e11d
https://cmssdt.cern.ch/SDT/jenkins-artifacts/pull-request-integration/PR-25287/31830/summary.html

[cmsbuild]

Comparison job queued.

[cmsbuild]

Comparison is ready
https://cmssdt.cern.ch/SDT/jenkins-artifacts/pull-request-integration/PR-25287/31830/summary.html

Comparison Summary:

• No significant changes to the logs found
• Reco comparison results: 0 differences found in the comparisons
• DQMHistoTests: Total files compared: 33
• DQMHistoTests: Total histograms compared: 3131839
• DQMHistoTests: Total failures: 4
• DQMHistoTests: Total nulls: 0
• DQMHistoTests: Total successes: 3131631
• DQMHistoTests: Total skipped: 204
• DQMHistoTests: Total Missing objects: 0
• DQMHistoSizes: Histogram memory added: 0.0 KiB( 32 files compared)
• Checked 137 log files, 14 edm output root files, 33 DQM output files

[pohsun]

+1

[cmsbuild]

This pull request is fully signed and it will be integrated in one of the next master IBs (tests are also fine). This pull request will now be reviewed by the release team before it's merged. @davidlange6, @slava77, @smuzaffar, @fabiocos (and backports should be raised in the release meeting by the corresponding L2)

[fabiocos]

@mmusich I understand that no backport is needed of this PR, but you do not object merging it either, am I correct?

[mmusich]

@mmusich I understand that no backport is needed of this PR, but you do not object merging it either, am I correct?

yes, I am all in favor of this development

[fabiocos]

+1