Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[9.4.X] HTCondor migration, fixes, improvements and preparation for AFS migration in MillePede workflow #21472

Merged
merged 43 commits into from Dec 14, 2017

Conversation

ghellwig
Copy link

backport of #21471

Summary of changes:

  • File list creation script
    • adapted script to dasgoclient behaviour
    • added option to better handle rereco datasets
    • automatically create ini-file snippets to be included in the MPS master config
    • added unit test for this script
  • Migrated the workflow to use the new big-memory machines managed within HTCondor
    • adapted scripts to the new infrastructure
    • translated old perl script to python
    • refactoring of mps_update.py
  • New options in master config file
    • specify maximum number of events per datasets (as opposed to reducing the number of input files)
    • added option externalDatasets to include, e.g., automatically created dataset snippets (see above)
      • including a refactoring of the alignment setup script
  • Miscellaneous fixes and changes
    • changed default MC global tag to automatic phase-1 global tag
    • option to drop a default pede option
    • command line option to submit a specific job instead of only the first in SETUP state
    • remove dead code (never executed or the computed variables are never used)
    • fix bug in GT-customization logic
      • caused an actually valid configuration to fail
    • update of documentation in default templates
    • preparation of migration from AFS to EOS

 - option for user defined output base directory
 - option to force overwriting old results in output base directory
`argparse` creates automatically a help message which is by
construction up-to-date while the hard-coded help text could easily
become wrong if an option is added or removed.
@cmsbuild
Copy link
Contributor

Pull request #21472 was updated. @ghellwig, @arunhep, @cerminar, @cmsbuild, @franzoni, @lpernie can you please check and sign again.

@ghellwig
Copy link
Author

please test

@cmsbuild
Copy link
Contributor

cmsbuild commented Nov 30, 2017

The tests are being triggered in jenkins.
https://cmssdt.cern.ch/jenkins/job/ib-any-integration/24844/console Started: 2017/12/01 01:27

@cmsbuild
Copy link
Contributor

cmsbuild commented Dec 1, 2017

@cmsbuild
Copy link
Contributor

cmsbuild commented Dec 1, 2017

Comparison job queued.

@cmsbuild
Copy link
Contributor

cmsbuild commented Dec 1, 2017

Comparison is ready
https://cmssdt.cern.ch/SDT/jenkins-artifacts/pull-request-integration/PR-21472/24844/summary.html

There are some workflows for which there are errors in the baseline:
10042.0 step 5
The results for the comparisons for these workflows could be incomplete
This means most likely that the IB is having errors in the relvals.The error does NOT come from this pull request

Comparison Summary:

  • No significant changes to the logs found
  • Reco comparison results: 0 differences found in the comparisons
  • DQMHistoTests: Total files compared: 26
  • DQMHistoTests: Total histograms compared: 2655767
  • DQMHistoTests: Total failures: 87
  • DQMHistoTests: Total nulls: 0
  • DQMHistoTests: Total successes: 2655511
  • DQMHistoTests: Total skipped: 169
  • DQMHistoTests: Total Missing objects: 0
  • DQMHistoSizes: Histogram memory added: 0 KiB( 0 files compared)
  • Checked 107 log files, 8 edm output root files, 26 DQM output files

@arunhep
Copy link
Contributor

arunhep commented Dec 6, 2017

+1

@cmsbuild
Copy link
Contributor

cmsbuild commented Dec 6, 2017

This pull request is fully signed and it will be integrated in one of the next CMSSW_9_4_X IBs (tests are also fine) and once validation in the development release cycle CMSSW_10_0_X is complete. This pull request will now be reviewed by the release team before it's merged. @davidlange6, @slava77, @smuzaffar (and backports should be raised in the release meeting by the corresponding L2)

@davidlange6
Copy link
Contributor

merge

@cmsbuild cmsbuild merged commit c294d47 into cms-sw:CMSSW_9_4_X Dec 14, 2017
@ghellwig ghellwig deleted the mps-fixes_94X branch December 19, 2017 16:49
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging this pull request may close these issues.

None yet

5 participants