[10.0.X] HTCondor migration, fixes, improvements and preparation for AFS migration in MillePede workflow #21471

ghellwig · 2017-11-28T08:47:13Z

Summary of changes:

File list creation script
- adapted script to dasgoclient behaviour
- added option to better handle rereco datasets
- automatically create ini-file snippets to be included in the MPS master config
- added unit test for this script
Migrated the workflow to use the new big-memory machines managed within HTCondor
- adapted scripts to the new infrastructure
- translated old perl script to python
- refactoring of mps_update.py
New options in master config file
- specify maximum number of events per datasets (as opposed to reducing the number of input files)
- added option externalDatasets to include, e.g., automatically created dataset snippets (see above)
  - including a refactoring of the alignment setup script
Miscellaneous fixes and changes
- changed default MC global tag to automatic phase-1 global tag
- option to drop a default pede option
- command line option to submit a specific job instead of only the first in SETUP state
- remove dead code (never executed or the computed variables are never used)
- fix bug in GT-customization logic
  - caused an actually valid configuration to fail
- update of documentation in default templates
- preparation of migration from AFS to EOS

- option for user defined output base directory - option to force overwriting old results in output base directory

- also remove (annoying) debug printout

`argparse` creates automatically a help message which is by construction up-to-date while the hard-coded help text could easily become wrong if an option is added or removed.

…ined.

…a longer time.

cmsbuild · 2017-11-30T14:42:15Z

The code-checks are being triggered in jenkins.

ghellwig · 2017-11-30T14:42:24Z

please test

cmsbuild · 2017-11-30T14:45:02Z

+code-checks

Logs: https://cmssdt.cern.ch/SDT/code-checks/PR-21471/2384

cmsbuild · 2017-11-30T14:45:23Z

The tests are being triggered in jenkins.
https://cmssdt.cern.ch/jenkins/job/ib-any-integration/24846/console Started: 2017/12/01 01:46

cmsbuild · 2017-11-30T14:45:24Z

Pull request #21471 was updated. @ghellwig, @arunhep, @cerminar, @cmsbuild, @franzoni, @lpernie can you please check and sign again.

cmsbuild · 2017-12-01T02:47:03Z

+1
Tested at: e511e29
https://cmssdt.cern.ch/SDT/jenkins-artifacts/pull-request-integration/PR-21471/24846/summary.html

The following merge commits were also included on top of IB + this PR after doing git cms-merge-topic:
a2fb258
You can see more details here:
https://cmssdt.cern.ch/SDT/jenkins-artifacts/pull-request-integration/PR-21471/24846/git-log-recent-commits
https://cmssdt.cern.ch/SDT/jenkins-artifacts/pull-request-integration/PR-21471/24846/git-merge-result

cmsbuild · 2017-12-01T02:47:06Z

Comparison job queued.

cmsbuild · 2017-12-01T13:30:13Z

Comparison is ready
https://cmssdt.cern.ch/SDT/jenkins-artifacts/pull-request-integration/PR-21471/24846/summary.html

Comparison Summary:

No significant changes to the logs found
Reco comparison results: 0 differences found in the comparisons
DQMHistoTests: Total files compared: 27
DQMHistoTests: Total histograms compared: 2835085
DQMHistoTests: Total failures: 1
DQMHistoTests: Total nulls: 0
DQMHistoTests: Total successes: 2834906
DQMHistoTests: Total skipped: 178
DQMHistoTests: Total Missing objects: 0
DQMHistoSizes: Histogram memory added: 1.01999999995 KiB( 23 files compared)
Checked 111 log files, 8 edm output root files, 27 DQM output files

arunhep · 2017-12-06T23:46:24Z

+1

cmsbuild · 2017-12-06T23:47:05Z

This pull request is fully signed and it will be integrated in one of the next master IBs (tests are also fine). This pull request will now be reviewed by the release team before it's merged. @davidlange6, @slava77, @smuzaffar (and backports should be raised in the release meeting by the corresponding L2)

ghellwig · 2017-12-07T08:13:27Z

@arunhep thank you!

davidlange6 · 2017-12-07T12:44:11Z

+1

gregor-mittag added 30 commits November 28, 2017 09:16

Add new options to file list creation script.

da5a5a7

- option for user defined output base directory - option to force overwriting old results in output base directory

Add unit test for file list creation script.

5988188

Streamline property extraction method.

95b00d1

Use local /tmp directory in unit tests.

c4acb6c

Fully migrate default MPS templates to phase-1.

cd93c8c

Add separate folders for binaries, tree files and monitoring files.

dea2c02

- also remove (annoying) debug printout

Encapsulate separate tasks in functions.

3a4e8b9

Encapsulate alignment setup procedure in a class.

866e610

Remove redundant help text in favor of automatically created help.

020c213

`argparse` creates automatically a help message which is by construction up-to-date while the hard-coded help text could easily become wrong if an option is added or removed.

Update documentation in MPS templates.

5214bc1

Allow also to drop pede option with helper method.

a776ef6

Add option to automatically create dataset ini file with all IOVs def…

2e171a7

…ined.

Add option to create file lists and ini files for ReReco datasets.

162f947

Migrate 'mps_setup.pl' to python (-> 'mps_setup.py').

ae97c60

Clean up 'mps_update.py'.

c3e4bea

Remove dead (and unsupported) code.

5113588

Support HTCondor pede jobs and fix CPU-time regex.

b5203e1

Handle case where binaries folder does not yet exist.

8faaab6

Fix disk usage calculation in 'mps_fire.py'.

7de071d

Adapt perl legacy code to handle htcondor job IDs properly.

3cbd2e9

Add job-id option to fire only specific jobs.

9d1c3d9

Fix previously redundant and now ill-defined 'cmsenv' statement.

70be6ef

Fix tree-file-list creation.

ade5195

Clean up 'mps_splice.py'.

8e2a18f

Add option to specify maximum number of events per dataset.

13b076f

Minor clean-up in universal config template.

ff6deaf

Raise the MPS-limit on the CPUs to 16 cores.

67ba601

Fix 'mps_update.py' for HTCondor jobs that have not been checked for …

cd7220c

…a longer time.

Correctly handle prematurely exited jobs.

abda4bf

Fix 'mps_update.py' for old LSF jobs.

7af964e

cmsbuild added comparison-pending tests-pending labels Nov 30, 2017

cmsbuild added code-checks-approved and removed code-checks-pending labels Nov 30, 2017

cmsbuild added tests-started and removed tests-pending labels Nov 30, 2017

cmsbuild added tests-approved and removed tests-started labels Dec 1, 2017

cmsbuild added comparison-available and removed comparison-pending labels Dec 1, 2017

cmsbuild added alca-approved fully-signed and removed alca-pending pending-signatures labels Dec 6, 2017

cmsbuild added orp-approved and removed orp-pending labels Dec 7, 2017

cmsbuild merged commit df94eeb into cms-sw:master Dec 7, 2017

ghellwig deleted the mps-fixes_100X branch December 19, 2017 16:49

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[10.0.X] HTCondor migration, fixes, improvements and preparation for AFS migration in MillePede workflow #21471

[10.0.X] HTCondor migration, fixes, improvements and preparation for AFS migration in MillePede workflow #21471

ghellwig commented Nov 28, 2017

cmsbuild commented Nov 30, 2017

ghellwig commented Nov 30, 2017

cmsbuild commented Nov 30, 2017

cmsbuild commented Nov 30, 2017 •

edited

cmsbuild commented Nov 30, 2017

cmsbuild commented Dec 1, 2017

cmsbuild commented Dec 1, 2017

cmsbuild commented Dec 1, 2017

arunhep commented Dec 6, 2017

cmsbuild commented Dec 6, 2017

ghellwig commented Dec 7, 2017

davidlange6 commented Dec 7, 2017

[10.0.X] HTCondor migration, fixes, improvements and preparation for AFS migration in MillePede workflow #21471

[10.0.X] HTCondor migration, fixes, improvements and preparation for AFS migration in MillePede workflow #21471

Conversation

ghellwig commented Nov 28, 2017

cmsbuild commented Nov 30, 2017

ghellwig commented Nov 30, 2017

cmsbuild commented Nov 30, 2017

cmsbuild commented Nov 30, 2017 • edited

cmsbuild commented Nov 30, 2017

cmsbuild commented Dec 1, 2017

cmsbuild commented Dec 1, 2017

cmsbuild commented Dec 1, 2017

arunhep commented Dec 6, 2017

cmsbuild commented Dec 6, 2017

ghellwig commented Dec 7, 2017

davidlange6 commented Dec 7, 2017

cmsbuild commented Nov 30, 2017 •

edited