Repository navigation
Support inputs with multiple arguments #1933
Description
Activity
is this feature planned to be implemented in the upcoming releases? It would be great to have it in KFP
is this feature planned to be implemented in the upcoming releases? It would be great to have it in KFP
At this moment there we have no bandwidth for implementing this feature. It needs many places to change.
What is the scenario where you might want this feature?
The use case is the following: we have K-fold a cross-validation task somewhere in the middle of our pipelines, and the number of fold is the parameter to the pipeline. The desire is to distribute computation of each fold to a separate task and run them in parallel since this is a time-consuming operation. Basically this requires the following:
Task C now requires a variable number of input parameters.Can we offer our help and contribute to the implementation of this feature under this ticket?
Reacted by Sascha Heyer, Oskar Lilja, Yusuke Horibe and julian.leeDoes anyone have a pattern to make something like this work? The only hacks I've thought of would be maybe something like a simple shell call that takes the values from TaskB-1...TaskB-N and concatenates them to a single string (json or just space deliminated) or maybe to lines in a file, then TaskC takes that concatenation as input and knows how to parse it
Other ideas might be that TaskB-* concatenate their results somewhere central and TaskC grabs from there. So maybe TaskB-* write results to the same location, and TaskC grep's from there?
This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.
This issue has been automatically closed because it has not had recent activity. Please comment "/reopen" to reopen it.
Any news on this feature ? :)
Thank you !Reacted by hans, devrivne, Sascha Heyer, Amitg1, julian.lee, Aditya Soni and minikomiThis feature request is closely related to #3412
Are there any updates on that this is a request I have seen already multiple times.- added a commit that references this issue
on Oct 22, 2023 Closing this issue. No activity for more than a year.
/close
@rimolive: Closing this issue.
Details
In response to this:
Closing this issue. No activity for more than a year.
/close
Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.
Need to support variadic inputs in ComponentSpec, Lightweight components, load_component etc.
In command-line we can pass multiple inputs as repeated command-line parameters:
--a 1 --a 2.In TaskSpec we can make argument a list:
arguments: {a: [1, 2]}.In inputSpec we could add
repeated: trueorallowMultiple: trueproperty.In ContainerImplementation we could specify the command args as follows:
{map: {source: {inputValue: RepInput}, selector: [--a, {item}] }}Related: #1901