Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Add parameter info to argo workflow template annotations #1549

Merged
merged 6 commits into from
Oct 12, 2023
Merged

Add parameter info to argo workflow template annotations #1549

Show file tree
Hide file tree
Changes from 3 commits
Commits
Show all changes
6 commits
Select commit Hold shift + click to select a range
b875b12
Add parameter-required info to template annotations
rohanrebello Sep 26, 2023
76723bc
Use a single annotation instead
rohanrebello Sep 27, 2023
581cc22
Add type information as well and change serialization method
rohanrebello Oct 7, 2023
a46965a
Avoid touching core by special-casing JSONType
rohanrebello Oct 10, 2023
fae29ad
Formatting
rohanrebello Oct 10, 2023
79d8635
Handle IncludeFile parameter types
rohanrebello Oct 11, 2023
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Jump to
Jump to file
Failed to load files.
Diff view
Diff view
2 changes: 1 addition & 1 deletion metaflow/parameters.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -40,7 +40,7 @@


class JSONTypeClass(click.ParamType):
name = "JSON"
name = __name__ = "JSON"
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Needed to supply the type information in a standardized way across all types. The rest of the types are builtins which already have this field.


def convert(self, value, param, ctx):
if not isinstance(value, strtype):
Expand Down
19 changes: 15 additions & 4 deletions metaflow/plugins/argo/argo_workflows.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -401,6 +401,8 @@ def _process_parameters(self):
)
seen.add(norm)

param_type = str(param.kwargs.get("type").__name__)
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Flatten the type information to a string instead of the type Object (which isn't serializable either). Use just the name of the type rather than the <class '...'> stuff.


is_required = param.kwargs.get("required", False)
# Throw an exception if a schedule is set for a flow with required
# parameters with no defaults. We currently don't have any notion
Expand All @@ -412,16 +414,17 @@ def _process_parameters(self):
"Scheduling such parameters via Argo CronWorkflows is not "
"currently supported." % param.name
)
value = deploy_time_eval(param.kwargs.get("default"))
default_value = deploy_time_eval(param.kwargs.get("default"))
# If the value is not required and the value is None, we set the value to
# the JSON equivalent of None to please argo-workflows. Unfortunately it
# has the side effect of casting the parameter value to string null during
# execution - which needs to be fixed imminently.
if not is_required or value is not None:
value = json.dumps(value)
if not is_required or default_value is not None:
default_value = json.dumps(default_value)
parameters[param.name] = dict(
name=param.name,
value=value,
value=default_value,
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It would be nice to rename this in the parameter map too but I didn't want to get too ambitious, especially since it's serialized in the argo workflow template as value again. #ifItAintBroke

type=param_type,
description=param.kwargs.get("help"),
is_required=is_required,
)
Expand Down Expand Up @@ -574,6 +577,14 @@ def _compile_workflow_template(self):
"metaflow/user": "argo-workflows",
"metaflow/flow_name": self.flow.name,
}

if self.parameters:
annotations.update(
{
"metaflow/parameters": json.dumps(self.parameters)
}
)

if current.get("project_name"):
annotations.update(
{
Expand Down