Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

DM-27492: Improvements to pipetask execution options #133

Merged
merged 7 commits into from Aug 12, 2021

Conversation

andy-slac
Copy link
Collaborator

Several improvements in pipetask execution options:

  • New option --skip-existing-in which takes collection names(s), if output
    datasets already exist in those collections corresponding quanta is skipped.
  • A --skip-existing option is now equivalent to appending output run
    collection to the --skip-existing-in list.
  • An --extend-run option implicitly enables --skip-existing option.
  • A --prune-replaced=unstore option only removes regular output datasets;
    InitOutputs, task configs, and package versions are not removed.

Checklist

  • ran Jenkins
  • added a release note for user-visible changes to doc/changes

Copy link
Member

@timj timj left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looks okay. I wasn't really able to get my head around all the test changes in a reasonable amount of time.

python/lsst/ctrl/mpexec/cli/opt/options.py Outdated Show resolved Hide resolved
python/lsst/ctrl/mpexec/cli/opt/options.py Outdated Show resolved Hide resolved
python/lsst/ctrl/mpexec/cli/script/qgraph.py Outdated Show resolved Hide resolved
python/lsst/ctrl/mpexec/cli/script/run.py Outdated Show resolved Hide resolved
python/lsst/ctrl/mpexec/cli/script/run.py Outdated Show resolved Hide resolved
python/lsst/ctrl/mpexec/preExecInit.py Outdated Show resolved Hide resolved
python/lsst/ctrl/mpexec/preExecInit.py Outdated Show resolved Hide resolved
python/lsst/ctrl/mpexec/preExecInit.py Outdated Show resolved Hide resolved
python/lsst/ctrl/mpexec/singleQuantumExecutor.py Outdated Show resolved Hide resolved
tests/test_preExecInit.py Outdated Show resolved Hide resolved
@andy-slac andy-slac merged commit 3316c66 into master Aug 12, 2021
@andy-slac andy-slac deleted the tickets/DM-27492 branch August 12, 2021 06:15
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

None yet

2 participants