Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[9.4.X] HTCondor migration, fixes, improvements and preparation for AFS migration in MillePede workflow #21472

Merged
merged 43 commits into from Dec 14, 2017
Merged

[9.4.X] HTCondor migration, fixes, improvements and preparation for AFS migration in MillePede workflow #21472

merged 43 commits into from Dec 14, 2017

Commits on Nov 28, 2017

  1. Add new options to file list creation script. 

     - option for user defined output base directory
 - option to force overwriting old results in output base directory
    @gregor-mittag
    gregor-mittag committed Nov 28, 2017
    Copy the full SHA
    1d51304 View commit details
    Browse the repository at this point in the history

  2. Add unit test for file list creation script.

    @gregor-mittag
    gregor-mittag committed Nov 28, 2017
    Copy the full SHA
    1d57fd0 View commit details
    Browse the repository at this point in the history

  3. Streamline property extraction method.

    @gregor-mittag
    gregor-mittag committed Nov 28, 2017
    Copy the full SHA
    d1de05c View commit details
    Browse the repository at this point in the history

  4. Use local /tmp directory in unit tests.

    @gregor-mittag
    gregor-mittag committed Nov 28, 2017
    Copy the full SHA
    19a6a9e View commit details
    Browse the repository at this point in the history

  5. Fully migrate default MPS templates to phase-1.

    @gregor-mittag
    gregor-mittag committed Nov 28, 2017
    Copy the full SHA
    c69cee8 View commit details
    Browse the repository at this point in the history

  6. Add separate folders for binaries, tree files and monitoring files. 

     - also remove (annoying) debug printout
    @gregor-mittag
    gregor-mittag committed Nov 28, 2017
    Copy the full SHA
    b34e788 View commit details
    Browse the repository at this point in the history

  7. Encapsulate separate tasks in functions.

    @gregor-mittag
    gregor-mittag committed Nov 28, 2017
    Copy the full SHA
    3daf2cb View commit details
    Browse the repository at this point in the history

  8. Encapsulate alignment setup procedure in a class.

    @gregor-mittag
    gregor-mittag committed Nov 28, 2017
    Copy the full SHA
    0a8da08 View commit details
    Browse the repository at this point in the history

  9. Remove redundant help text in favor of automatically created help. 

    `argparse` creates automatically a help message which is by
construction up-to-date while the hard-coded help text could easily
become wrong if an option is added or removed.
    @gregor-mittag
    gregor-mittag committed Nov 28, 2017
    Copy the full SHA
    39a703d View commit details
    Browse the repository at this point in the history

  10. Update documentation in MPS templates.

    @gregor-mittag
    gregor-mittag committed Nov 28, 2017
    Copy the full SHA
    2f93463 View commit details
    Browse the repository at this point in the history

  11. Allow also to drop pede option with helper method.

    @gregor-mittag
    gregor-mittag committed Nov 28, 2017
    Copy the full SHA
    b963a5c View commit details
    Browse the repository at this point in the history

  12. Add option to automatically create dataset ini file with all IOVs def… 

    …ined.
    @gregor-mittag
    gregor-mittag committed Nov 28, 2017
    Copy the full SHA
    6472dcc View commit details
    Browse the repository at this point in the history

  13. Add option to create file lists and ini files for ReReco datasets.

    @gregor-mittag
    gregor-mittag committed Nov 28, 2017
    Copy the full SHA
    84afe49 View commit details
    Browse the repository at this point in the history

  14. Migrate 'mps_setup.pl' to python (-> 'mps_setup.py').

    @gregor-mittag
    gregor-mittag committed Nov 28, 2017
    Copy the full SHA
    4734c43 View commit details
    Browse the repository at this point in the history

  15. Clean up 'mps_update.py'.

    @gregor-mittag
    gregor-mittag committed Nov 28, 2017
    Copy the full SHA
    9a71a1a View commit details
    Browse the repository at this point in the history

  16. Remove dead (and unsupported) code.

    @gregor-mittag
    gregor-mittag committed Nov 28, 2017
    Copy the full SHA
    e8c0693 View commit details
    Browse the repository at this point in the history

  17. Support HTCondor pede jobs and fix CPU-time regex.

    @gregor-mittag
    gregor-mittag committed Nov 28, 2017
    Copy the full SHA
    42f04bf View commit details
    Browse the repository at this point in the history

  18. Handle case where binaries folder does not yet exist.

    @gregor-mittag
    gregor-mittag committed Nov 28, 2017
    Copy the full SHA
    2a1b307 View commit details
    Browse the repository at this point in the history

  19. Fix disk usage calculation in 'mps_fire.py'.

    @gregor-mittag
    gregor-mittag committed Nov 28, 2017
    Copy the full SHA
    368c52e View commit details
    Browse the repository at this point in the history

  20. Adapt perl legacy code to handle htcondor job IDs properly.

    @gregor-mittag
    gregor-mittag committed Nov 28, 2017
    Copy the full SHA
    407a3c6 View commit details
    Browse the repository at this point in the history

  21. Add job-id option to fire only specific jobs.

    @gregor-mittag
    gregor-mittag committed Nov 28, 2017
    Copy the full SHA
    1e77435 View commit details
    Browse the repository at this point in the history

  22. Fix previously redundant and now ill-defined 'cmsenv' statement.

    @gregor-mittag
    gregor-mittag committed Nov 28, 2017
    Copy the full SHA
    fc7982b View commit details
    Browse the repository at this point in the history

  23. Fix tree-file-list creation.

    @gregor-mittag
    gregor-mittag committed Nov 28, 2017
    Copy the full SHA
    feaeab8 View commit details
    Browse the repository at this point in the history

  24. Clean up 'mps_splice.py'.

    @gregor-mittag
    gregor-mittag committed Nov 28, 2017
    Copy the full SHA
    c9984d6 View commit details
    Browse the repository at this point in the history

  25. Add option to specify maximum number of events per dataset.

    @gregor-mittag
    gregor-mittag committed Nov 28, 2017
    Copy the full SHA
    e66d3fe View commit details
    Browse the repository at this point in the history

  26. Minor clean-up in universal config template.

    @gregor-mittag
    gregor-mittag committed Nov 28, 2017
    Copy the full SHA
    f01d2bc View commit details
    Browse the repository at this point in the history

  27. Raise the MPS-limit on the CPUs to 16 cores.

    @gregor-mittag
    gregor-mittag committed Nov 28, 2017
    Copy the full SHA
    bf00584 View commit details
    Browse the repository at this point in the history

  28. Fix 'mps_update.py' for HTCondor jobs that have not been checked for … 

    …a longer time.
    @gregor-mittag
    gregor-mittag committed Nov 28, 2017
    Copy the full SHA
    e136460 View commit details
    Browse the repository at this point in the history

  29. Correctly handle prematurely exited jobs.

    @gregor-mittag
    gregor-mittag committed Nov 28, 2017
    Copy the full SHA
    163e1b6 View commit details
    Browse the repository at this point in the history

  30. Fix 'mps_update.py' for old LSF jobs.

    @gregor-mittag
    gregor-mittag committed Nov 28, 2017
    Copy the full SHA
    cf31e0f View commit details
    Browse the repository at this point in the history

  31. Add possibility to assign a global weight to an external dataset.

    @gregor-mittag
    gregor-mittag committed Nov 28, 2017
    Copy the full SHA
    5dfd5b8 View commit details
    Browse the repository at this point in the history

  32. Optimize usage of regular expressions.

    @gregor-mittag
    gregor-mittag committed Nov 28, 2017
    Copy the full SHA
    e4f12ab View commit details
    Browse the repository at this point in the history

  33. More useful error message if proxy is not initialized. 

    This commit also makes the proxy check a tool.
    @gregor-mittag
    gregor-mittag committed Nov 28, 2017
    Copy the full SHA
    1f6399f View commit details
    Browse the repository at this point in the history

  34. Fix behaviour when specifying maximum number of events.

    @gregor-mittag
    gregor-mittag committed Nov 28, 2017
    Copy the full SHA
    767a671 View commit details
    Browse the repository at this point in the history

  35. Make 'datasetdir' available for 'externalDatasets'.

    @gregor-mittag
    gregor-mittag committed Nov 28, 2017
    Copy the full SHA
    09a6371 View commit details
    Browse the repository at this point in the history

  36. Fix behaviour of 'mps_update.py' in case the user has been moved to a… 

    …nother HTCondor Schedd.
    @gregor-mittag
    gregor-mittag committed Nov 28, 2017
    Copy the full SHA
    de3a7bb View commit details
    Browse the repository at this point in the history

  37. Kill jobs if the submitter has no permissions to run a 'BigMemJob'.

    @gregor-mittag
    gregor-mittag committed Nov 28, 2017
    Copy the full SHA
    e0d5b44 View commit details
    Browse the repository at this point in the history

  38. Use HTCondor machines by default for pede jobs.

    @gregor-mittag
    gregor-mittag committed Nov 28, 2017
    Copy the full SHA
    f69842e View commit details
    Browse the repository at this point in the history

  39. Update campaign name when copying configuration files.

    @gregor-mittag
    gregor-mittag committed Nov 28, 2017
    Copy the full SHA
    985661a View commit details
    Browse the repository at this point in the history

  40. Drop possible previous customizations of GT.

    @gregor-mittag
    gregor-mittag committed Nov 28, 2017
    Copy the full SHA
    68821f8 View commit details
    Browse the repository at this point in the history

  41. Circumvent potential delete-permission issues.

    @gregor-mittag
    gregor-mittag committed Nov 28, 2017
    Copy the full SHA
    61f72be View commit details
    Browse the repository at this point in the history

  42. Use python's tar interface to circumvent EOS timestamp issues.

    @gregor-mittag
    gregor-mittag committed Nov 28, 2017
    Copy the full SHA
    ed96f84 View commit details
    Browse the repository at this point in the history

Commits on Nov 30, 2017

  1. Fix JSON file name if used with a non-default output path.

    @gregor-mittag
    gregor-mittag committed Nov 30, 2017
    Copy the full SHA
    891237f View commit details
    Browse the repository at this point in the history