Support inputs with multiple arguments

[Ark-kun]

Need to support variadic inputs in ComponentSpec, Lightweight components, load_component etc.

In command-line we can pass multiple inputs as repeated command-line parameters: --a 1 --a 2.
In TaskSpec we can make argument a list: arguments: {a: [1, 2]}.
In inputSpec we could add repeated: true or allowMultiple: true property.
In ContainerImplementation we could specify the command args as follows: {map: {source: {inputValue: RepInput}, selector: [--a, {item}] }}

Related: #1901

[areshytko]

is this feature planned to be implemented in the upcoming releases? It would be great to have it in KFP

[Ark-kun]

is this feature planned to be implemented in the upcoming releases? It would be great to have it in KFP

At this moment there we have no bandwidth for implementing this feature. It needs many places to change.

What is the scenario where you might want this feature?

[areshytko]

The use case is the following: we have K-fold a cross-validation task somewhere in the middle of our pipelines, and the number of fold is the parameter to the pipeline. The desire is to distribute computation of each fold to a separate task and run them in parallel since this is a time-consuming operation. Basically this requires the following:

Task C now requires a variable number of input parameters.

Can we offer our help and contribute to the implementation of this feature under this ticket?

[ca-scribner]

Does anyone have a pattern to make something like this work? The only hacks I've thought of would be maybe something like a simple shell call that takes the values from TaskB-1...TaskB-N and concatenates them to a single string (json or just space deliminated) or maybe to lines in a file, then TaskC takes that concatenation as input and knows how to parse it

Other ideas might be that TaskB-* concatenate their results somewhere central and TaskC grabs from there. So maybe TaskB-* write results to the same location, and TaskC grep's from there?

[stale]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.

[stale]

This issue has been automatically closed because it has not had recent activity. Please comment "/reopen" to reopen it.