You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
When adding a step to a pipeline it's possible to override parameters. When overriding parameters, it's possible to do variable replacement of the parameters for parameters from the prior stages of the pipeline, e.g. parameter_override={'Args/input_file': '${stage3.id}' }. However, this feature doesn't appear to currently support overriding multiple parameters in the same argument, e.g. parameter_override={'Args/input_file': '[${stage3_task_A.id}, '${stage3_task_B.id}', '${stage3_task_C.id}']}. There are a lot of potential use cases for such a feature.
It is possible to do something like this using a dedicated argument for each variable being replaced. But that method would then create a huge number of potential variables that would need to be discovered and iterated over when it would be much simpler to have a single argument with a list of values.
This should be a fairly non-intrusive change to the codebase. I'd happily create a PR for it if someone would be willing to direct me to the relevant parts of the code (I've looked with the help of someone on Slack but it wasn't obvious).
Motivation
I have a pipeline that has a variable number of parent stages that will feed into one child. It would be much easier to return a list and iterate over the list rather than discover each variable.
Hey @natephysics, we always appreciate any contributions, and I'll be glad to help you! That being said, I believe to support your suggestion, you'd need to check the _parse_step_ref method at clearml/automation/controller.py:2779 and extend it's functionality to handle multiple references per parameter
Proposal Summary
Tested with pipelines from tasks.
When adding a step to a pipeline it's possible to override parameters. When overriding parameters, it's possible to do variable replacement of the parameters for parameters from the prior stages of the pipeline, e.g.
parameter_override={'Args/input_file': '${stage3.id}' }
. However, this feature doesn't appear to currently support overriding multiple parameters in the same argument, e.g.parameter_override={'Args/input_file': '[${stage3_task_A.id}, '${stage3_task_B.id}', '${stage3_task_C.id}']}
. There are a lot of potential use cases for such a feature.It is possible to do something like this using a dedicated argument for each variable being replaced. But that method would then create a huge number of potential variables that would need to be discovered and iterated over when it would be much simpler to have a single argument with a list of values.
This should be a fairly non-intrusive change to the codebase. I'd happily create a PR for it if someone would be willing to direct me to the relevant parts of the code (I've looked with the help of someone on Slack but it wasn't obvious).
Motivation
I have a pipeline that has a variable number of parent stages that will feed into one child. It would be much easier to return a list and iterate over the list rather than discover each variable.
Related Discussion
https://clearml.slack.com/archives/CTK20V944/p1689609314325429
The text was updated successfully, but these errors were encountered: