-
Notifications
You must be signed in to change notification settings - Fork 4
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
DM-25385 make a pipetask command based on click #58
Merged
Merged
Changes from all commits
Commits
Show all changes
6 commits
Select commit
Hold shift + click to select a range
1f16c81
add a pipetask command
n8pease e20ebc4
add obs_base dependency
n8pease 329f824
add command, script, and optons
n8pease fd73c17
add pipetask2 command
n8pease a8172c3
add package-docs for the pipetask command
n8pease a8152fe
change to LogCliRunner for CLI unit tests
n8pease File filter
Filter by extension
Conversations
Failed to load comments.
Jump to
Jump to file
Failed to load files.
Diff view
Diff view
There are no files selected for viewing
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,29 @@ | ||
#!/usr/bin/env python | ||
|
||
# This file is part of ctrl_mpexec. | ||
# | ||
# Developed for the LSST Data Management System. | ||
# This product includes software developed by the LSST Project | ||
# (https://www.lsst.org). | ||
# See the COPYRIGHT file at the top-level directory of this distribution | ||
# for details of code ownership. | ||
# | ||
# This program is free software: you can redistribute it and/or modify | ||
# it under the terms of the GNU General Public License as published by | ||
# the Free Software Foundation, either version 3 of the License, or | ||
# (at your option) any later version. | ||
# | ||
# This program is distributed in the hope that it will be useful, | ||
# but WITHOUT ANY WARRANTY; without even the implied warranty of | ||
# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the | ||
# GNU General Public License for more details. | ||
# | ||
# You should have received a copy of the GNU General Public License | ||
# along with this program. If not, see <https://www.gnu.org/licenses/>. | ||
|
||
import sys | ||
|
||
from lsst.ctrl.mpexec.cli.pipetask import main | ||
|
||
if __name__ == "__main__": | ||
sys.exit(main()) |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,20 @@ | ||
# This file is part of ctrl_mpexec. | ||
# | ||
# Developed for the LSST Data Management System. | ||
# This product includes software developed by the LSST Project | ||
# (https://www.lsst.org). | ||
# See the COPYRIGHT file at the top-level directory of this distribution | ||
# for details of code ownership. | ||
# | ||
# This program is free software: you can redistribute it and/or modify | ||
# it under the terms of the GNU General Public License as published by | ||
# the Free Software Foundation, either version 3 of the License, or | ||
# (at your option) any later version. | ||
# | ||
# This program is distributed in the hope that it will be useful, | ||
# but WITHOUT ANY WARRANTY; without even the implied warranty of | ||
# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the | ||
# GNU General Public License for more details. | ||
# | ||
# You should have received a copy of the GNU General Public License | ||
# along with this program. If not, see <https://www.gnu.org/licenses/>. |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,25 @@ | ||
# This file is part of ctrl_mpexec. | ||
# | ||
# Developed for the LSST Data Management System. | ||
# This product includes software developed by the LSST Project | ||
# (https://www.lsst.org). | ||
# See the COPYRIGHT file at the top-level directory of this distribution | ||
# for details of code ownership. | ||
# | ||
# This program is free software: you can redistribute it and/or modify | ||
# it under the terms of the GNU General Public License as published by | ||
# the Free Software Foundation, either version 3 of the License, or | ||
# (at your option) any later version. | ||
# | ||
# This program is distributed in the hope that it will be useful, | ||
# but WITHOUT ANY WARRANTY; without even the implied warranty of | ||
# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the | ||
# GNU General Public License for more details. | ||
# | ||
# You should have received a copy of the GNU General Public License | ||
# along with this program. If not, see <http://www.gnu.org/licenses/>. | ||
|
||
__all__ = ["build", "qgraph", "run"] | ||
|
||
|
||
from .commands import build, qgraph, run |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,103 @@ | ||
# This file is part of ctrl_mpexec. | ||
# | ||
# Developed for the LSST Data Management System. | ||
# This product includes software developed by the LSST Project | ||
# (http://www.lsst.org). | ||
# See the COPYRIGHT file at the top-level directory of this distribution | ||
# for details of code ownership. | ||
# | ||
# This program is free software: you can redistribute it and/or modify | ||
# it under the terms of the GNU General Public License as published by | ||
# the Free Software Foundation, either version 3 of the License, or | ||
# (at your option) any later version. | ||
# | ||
# This program is distributed in the hope that it will be useful, | ||
# but WITHOUT ANY WARRANTY; without even the implied warranty of | ||
# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the | ||
# GNU General Public License for more details. | ||
# | ||
# You should have received a copy of the GNU General Public License | ||
# along with this program. If not, see <http://www.gnu.org/licenses/>. | ||
|
||
import click | ||
import copy | ||
|
||
from lsst.daf.butler.cli.opt import (config_file_option, | ||
config_option, | ||
log_level_option) | ||
from lsst.daf.butler.cli.utils import cli_handle_exception | ||
from lsst.obs.base.cli.opt import instrument_parameter | ||
from ..opt import (delete_option, | ||
order_pipeline_option, | ||
pipeline_dot_option, | ||
pipeline_option, | ||
save_pipeline_option, | ||
show_option, | ||
task_option) | ||
from .. import script | ||
from ..utils import makePipelineActions | ||
|
||
|
||
instrumentOptionHelp = ("Add an instrument which will be used to load config overrides when defining a " | ||
"pipeline. This must be the fully qualified class name.") | ||
|
||
|
||
class PipetaskCommand(click.Command): | ||
def parse_args(self, ctx, args): | ||
ctx.obj = copy.copy(args) | ||
super().parse_args(ctx, args) | ||
|
||
|
||
@click.command(cls=PipetaskCommand, short_help="Build pipeline definition.") | ||
@click.pass_context | ||
@pipeline_option() | ||
@task_option(multiple=True) | ||
@delete_option(metavar="LABEL", multiple=True) | ||
@config_option(metavar="LABEL:NAME=VALUE", multiple=True) | ||
@config_file_option(help="Configuration override file(s), applies to a task with a given label.", | ||
metavar="LABEL:FILE", | ||
multiple=True) | ||
@order_pipeline_option() | ||
@save_pipeline_option() | ||
@pipeline_dot_option() | ||
@instrument_parameter(help=instrumentOptionHelp, metavar="instrument", multiple=True) | ||
@show_option(multiple=True) | ||
@log_level_option(defaultValue=None) | ||
def build(ctx, *args, **kwargs): | ||
"""Build and optionally save pipeline definition. | ||
|
||
This does not require input data to be specified. | ||
""" | ||
# The pipeline actions (task, delete, config, config_file, and instrument) | ||
# must be handled in the order they appear on the command line, but the CLI | ||
# specification gives them all different option names. So, instead of using | ||
# the individual action options as they appear in kwargs (because | ||
# invocation order can't be known), we capture the CLI arguments by | ||
# overriding `click.Command.parse_args` and save them in the Context's | ||
# `obj` parameter. We use `makePipelineActions` to create a list of | ||
# pipeline actions from the CLI arguments and pass that list to the script | ||
# function using the `pipeline_actions` kwarg name, and remove the action | ||
# options from kwargs. | ||
for pipelineAction in (task_option.optionKey, delete_option.optionKey, config_option.optionKey, | ||
config_file_option.optionKey, instrument_parameter.optionKey): | ||
kwargs.pop(pipelineAction) | ||
kwargs['pipeline_actions'] = makePipelineActions(ctx.obj) | ||
cli_handle_exception(script.build, *args, **kwargs) | ||
|
||
|
||
@click.command(cls=PipetaskCommand) | ||
def qgraph(*args, **kwargs): | ||
"""Not implemented. | ||
|
||
Build and optionally save pipeline and quantum graph. | ||
""" | ||
print("Not implemented.") | ||
|
||
|
||
@click.command(cls=PipetaskCommand) | ||
def run(*args, **kwargs): | ||
"""Not implemented. | ||
|
||
Build and execute pipeline and quantum graph. | ||
""" | ||
print("Not implemented.") |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,29 @@ | ||
# This file is part of ctrl_mpexec. | ||
# | ||
# Developed for the LSST Data Management System. | ||
# This product includes software developed by the LSST Project | ||
# (https://www.lsst.org). | ||
# See the COPYRIGHT file at the top-level directory of this distribution | ||
# for details of code ownership. | ||
# | ||
# This program is free software: you can redistribute it and/or modify | ||
# it under the terms of the GNU General Public License as published by | ||
# the Free Software Foundation, either version 3 of the License, or | ||
# (at your option) any later version. | ||
# | ||
# This program is distributed in the hope that it will be useful, | ||
# but WITHOUT ANY WARRANTY; without even the implied warranty of | ||
# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the | ||
# GNU General Public License for more details. | ||
# | ||
# You should have received a copy of the GNU General Public License | ||
# along with this program. If not, see <http://www.gnu.org/licenses/>. | ||
|
||
|
||
from .delete import delete_option | ||
from .order_pipeline import order_pipeline_option | ||
from .pipeline import pipeline_option | ||
from .pipeline_dot import pipeline_dot_option | ||
from .save_pipeline import save_pipeline_option | ||
from .show import show_option | ||
from .task import task_option |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,47 @@ | ||
# This file is part of ctrl_mpexec. | ||
# | ||
# Developed for the LSST Data Management System. | ||
# This product includes software developed by the LSST Project | ||
# (https://www.lsst.org). | ||
# See the COPYRIGHT file at the top-level directory of this distribution | ||
# for details of code ownership. | ||
# | ||
# This program is free software: you can redistribute it and/or modify | ||
# it under the terms of the GNU General Public License as published by | ||
# the Free Software Foundation, either version 3 of the License, or | ||
# (at your option) any later version. | ||
# | ||
# This program is distributed in the hope that it will be useful, | ||
# but WITHOUT ANY WARRANTY; without even the implied warranty of | ||
# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the | ||
# GNU General Public License for more details. | ||
# | ||
# You should have received a copy of the GNU General Public License | ||
# along with this program. If not, see <http://www.gnu.org/licenses/>. | ||
|
||
import click | ||
|
||
from lsst.daf.butler.core.utils import iterable | ||
from lsst.daf.butler.cli.utils import MWOption, split_commas | ||
|
||
|
||
class delete_option: # noqa: N801 | ||
|
||
defaultHelp = "Delete task with given label from pipeline." | ||
optionFlags = ("--delete") | ||
optionKey = "delete" | ||
|
||
def __init__(self, help=defaultHelp, metavar=None, multiple=False, required=False): | ||
self.callback = split_commas if multiple else None | ||
self.help = help | ||
self.metavar = metavar | ||
self.multiple = multiple | ||
self.required = required | ||
|
||
def __call__(self, f): | ||
return click.option(*iterable(self.optionFlags), cls=MWOption, | ||
callback=self.callback, | ||
help=self.help, | ||
metavar=self.metavar, | ||
multiple=self.multiple, | ||
required=self.required)(f) |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,41 @@ | ||
# This file is part of ctrl_mpexec. | ||
# | ||
# Developed for the LSST Data Management System. | ||
# This product includes software developed by the LSST Project | ||
# (https://www.lsst.org). | ||
# See the COPYRIGHT file at the top-level directory of this distribution | ||
# for details of code ownership. | ||
# | ||
# This program is free software: you can redistribute it and/or modify | ||
# it under the terms of the GNU General Public License as published by | ||
# the Free Software Foundation, either version 3 of the License, or | ||
# (at your option) any later version. | ||
# | ||
# This program is distributed in the hope that it will be useful, | ||
# but WITHOUT ANY WARRANTY; without even the implied warranty of | ||
# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the | ||
# GNU General Public License for more details. | ||
# | ||
# You should have received a copy of the GNU General Public License | ||
# along with this program. If not, see <http://www.gnu.org/licenses/>. | ||
|
||
import click | ||
|
||
from lsst.daf.butler.cli.utils import MWOption | ||
|
||
|
||
class order_pipeline_option: # noqa: N801 | ||
|
||
defaultHelp = ("Order tasks in pipeline based on their data dependencies, ordering is performed as last " | ||
"step before saving or executing pipeline.") | ||
|
||
def __init__(self, required=False, help=defaultHelp): | ||
self.help = help | ||
self.isFlag = True | ||
self.required = required | ||
|
||
def __call__(self, f): | ||
return click.option("--order-pipeline", cls=MWOption, | ||
help=self.help, | ||
is_flag=self.isFlag, | ||
required=self.required)(f) |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,40 @@ | ||
# This file is part of ctrl_mpexec. | ||
# | ||
# Developed for the LSST Data Management System. | ||
# This product includes software developed by the LSST Project | ||
# (https://www.lsst.org). | ||
# See the COPYRIGHT file at the top-level directory of this distribution | ||
# for details of code ownership. | ||
# | ||
# This program is free software: you can redistribute it and/or modify | ||
# it under the terms of the GNU General Public License as published by | ||
# the Free Software Foundation, either version 3 of the License, or | ||
# (at your option) any later version. | ||
# | ||
# This program is distributed in the hope that it will be useful, | ||
# but WITHOUT ANY WARRANTY; without even the implied warranty of | ||
# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the | ||
# GNU General Public License for more details. | ||
# | ||
# You should have received a copy of the GNU General Public License | ||
# along with this program. If not, see <http://www.gnu.org/licenses/>. | ||
|
||
import click | ||
|
||
from lsst.daf.butler.cli.utils import MWOption | ||
|
||
|
||
class pipeline_option: # noqa: N801 | ||
|
||
defaultHelp = "Location of a pipeline definition file in YAML format." | ||
|
||
def __init__(self, required=False, help=defaultHelp): | ||
self.required = required | ||
self.help = help | ||
self.type = click.Path(exists=True, file_okay=True, dir_okay=False, readable=True) | ||
|
||
def __call__(self, f): | ||
return click.option("-p", "--pipeline", cls=MWOption, | ||
help=self.help, | ||
required=self.required, | ||
type=self.type)(f) |
Oops, something went wrong.
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
This looks a bit hackish. Could you instead just define a callback for each option that updates a common list? Or maybe click already has a way to append values from multiple options to the same parameter? Then you could use the same approach as I did in CmdLineParser - use those
_ACTION...
as type converters for individual options?There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
The click api really doesn't support any of these approaches. I think the best way would be to pivot to chaining subcommands, and making each of the action options a type of subcommand. Discussing that in Slack now.