Support list as pipeline parameter expression #147
Conversation
|
Adopting this branch and PR with a slightly different approach |
hylje
force-pushed
the
memona/support-list-as-pipeline-parameter
branch
3 times, most recently
from
2f585ca
to
aab8353
Compare
|
With these changes, you can pass YAML pipeline config parameter default lists into the API using |
hylje
force-pushed
the
memona/support-list-as-pipeline-parameter
branch
2 times, most recently
from
29e324b
to
108f642
Compare
hylje
force-pushed
the
memona/support-list-as-pipeline-parameter
branch
from
108f642
to
ca128bf
Compare
| rules: Dict[str, Any] | ||
|
|
||
|
|
||
| ExpressionValue = Union[str, int, bool, float, VariantExpression] |
There was a problem hiding this comment.
How would this VariantExpression dictionary affect they way roi and/or valohai-utils parsing the value now? Btw, should we call it ExpressionValueVariant?
There was a problem hiding this comment.
Backend accepts variant expressions as parameter values, and it is the only way it accepts list parameters. The format works for all parameter value types though.
Valohai-utils is not involved in this part of the process at all. We pass a list in structured format to backend, and backend receives a list in structured format and keeps it as a list.
When the execution is prepared by backend, if you pass a list parameter as a command line parameter and parse command line parameters in a Python script using valohai-utils, it must use comma separated values as that is the only format valohai-utils accepts. Other scripts or programs can accept different parameter passing styles, which is why we support custom multiple-separators.
You can also use file based configuration to read the structured list parameter as JSON (valohai/config/parameters.json), which skips the whole parsing step and you get the original list. This is what valohai-utils automatically does for parameters that haven't been overridden as command line parameters.
| args = { | ||
| "param0-float": 1.5, | ||
| "param0-int": 3, | ||
| "param0-string": "iceberg", | ||
| "param0-flag": True, | ||
| "listings": ["msg", "pfa"], | ||
| } |
There was a problem hiding this comment.
where does this come from in practice?
There was a problem hiding this comment.
I added pipeline argument parsing when converting for a future CLI enhancement. This PR lets you send lists as pipeline parameters if you make the values you want default in YAML config, but CLI needs some extra stuff to let you specify what you want to send and override the defaults. That extra stuff should also get converted right away, so we don't need to reimplement the conversion on CLI end.
Resolves 300
Changes in this PR: