Support for dynamic LoopArguments (not currently supported in Tekton)

[ckadner]

/kind feature

Description:

Since Tekton doesn't have loop/recursion support, PR #67 introduced a workaround for static loop parameters by flattening the loop with parallel TaskRuns.
But for dynamic loop parameters, Tekton does not have a feature to support that.

Additional information:

• PR Extend compiler to support parallelFor with loop static params #67 introduced static loop parameter support
• Issue #70 discusses support and workarounds for dynamic looping in Tekton

[issue-label-bot]

Issue-Label Bot is automatically applying the labels:

Label	Probability
kind/feature	0.96

Please mark this comment with 👍 or 👎 to give our bot feedback!
Links: app homepage, dashboard and code for this bot.

[bobcatfish]

Hi there! I'm kinda jumping into this without a lot of context, but I noticed it linked from #87 and I wanted to point out that Tekton has been explicitly avoiding adding support for recursion and dynamic behavior (though DSLs built on top could potentially support it). Let me know if you want to discuss more and/or feel free to join our working group https://github.com/tektoncd/community/blob/master/working-groups.md

(If this is a deal breaker for using Tekton then maybe Argo is a better fit for this kind of thing??)

[fenglixa]

/assign

[jlewi]

@animeshsingh @ckadner per @bobcatfish's comment. My sense is Tekton has features like loops that amount to generating the DAG at runtime. This seems reasonable as I think one of the values of a DAG scheduler is that you know at job submit time the entire graph so its easy to reason about.

Have you considered building any dynamic features ontop of Tekton rather than baking it into Tekton?

For example, for dynamic graphs you could continue to use Tekton Tasks but define a new CR (or DSL) that describes how these tasks should be executed in loops?

For dynamic graphs my mental model is a set of tasks that get triggered in response to events in some queue and then emit events to some other queue. So you could potentially Tekton events/triggers for that as well.

The best example of this I think is Amazon Simple Workflow.

What I've heard from a couple folks that have worked SWF is that they like the fact that they end up writing their code pretty much the same way they would writ synchronous code.

/cc @amygdala
/cc @animeshsingh
/cc @Jeffwan

[animeshsingh]

thanks @jlewi current goal is for Tekton community to address it. We are driving an open issue there on the topic, as it will help coming with an official implementation which can be maintained by them
tektoncd/pipeline#2050

[bobcatfish]

While discussing tektoncd/pipeline#2050 in a recent working group, it was pointed out that "looping" can refer to different sorts of loops, and the one that we'd be most likely to add into Tekton would be a kind of "for each" functionality, like a Python list comprehension. It's (so far) not likely that we'd add generic dynamic looping, e.g. looping based on some state outside of the Pipeline.

[fenglixa]

Thanks @bobcatfish Yes, it's correct, we are also requiring the looping based on inside of the pipeline. Acturally the "dynamic looping" is mean the loop params rely on the result of other tasks in the same pipeline.

[stale]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.

[bobcatfish]

quick update from tekton: @GregDritschler has been working on adding looping functionality to Tekton and it is currently possible via a custom tasks (https://github.com/tektoncd/experimental/tree/master/task-loops#task-loop-extension) - next steps for this are to decide if we want to add it to the Tekton API or alternatively promote this custom task to a top level Tekton project

[stale]

This issue has been automatically closed because it has not had recent activity. Please comment "/reopen" to reopen it.