docs: Add Tekton backend design doc #65

FogDong · 2020-09-28T10:46:47Z

What changes were proposed in this pull request?

Chore(docs): add tekton backend design doc
Ref: #44

Why are the changes needed?

Does this PR introduce any user-facing change?

How was this patch tested?

CLAassistant · 2020-09-28T10:46:51Z

All committers have signed the CLA.

merlintang

thanks for you to start this process. this would be useful to tekton community.

I also want to know, how can you define the workflow for the tekton? do we follow the explicitly or implicitly way of couler ?

terrytangyuan · 2020-09-28T14:15:03Z

docs/couler-tekton-design.md

+```
+
+A `whenExpression` is made of `Input`, `Operator` and `Values`. The `Input` can be static inputs or variables (Parameters or Results in Tekton). The `Operator` can be either `in` or `notin`.
+As we can see, there is no `else_op` in Tekton, so we need to convert the `else` in the Tekton backend, which is pretty simple since there is only `in` and `notin` operator, we can simply change the operator in the "else" task like:


Good to see this. Also note that else_op should be optional. I'll add a note in Couler APIs design on this.

terrytangyuan · 2020-09-28T14:17:21Z

docs/couler-tekton-design.md

+
+#### `while_loop(cond, func, *args, **kwargs)`
+
+There is no native support for loops in Tekton for now.


Is tektoncd/pipeline#2050 a good issue to track this feature in Tekton?

terrytangyuan · 2020-09-28T14:19:55Z

docs/couler-tekton-design.md

+
+The `get_status`, `get_logs` remains the same as the Argo design.
+
+* `submit(config=workflow_config(schedule="* * * * 1"))`: There is no native support for cron in Tekton, see this [issue](https://github.com/tektoncd/triggers/issues/69) for more information. The community recommend cronjob in kubernetes, see [example](https://github.com/tektoncd/triggers/blob/master/examples/cron/README.md) here.


Argo Workflow's CronWorkflow is based on k8s CronJob anyways. Would it be easy to wrap CronJob in a Tekton workflow so that it's easy for users to start a scheduled workflow by simply specifying a schedule argument?

Great idea, we can wrap cron in the couler's workflow and let backend to handle the work

terrytangyuan · 2020-09-28T14:20:45Z

docs/couler-tekton-design.md

+The `get_status`, `get_logs` remains the same as the Argo design.
+
+* `submit(config=workflow_config(schedule="* * * * 1"))`: There is no native support for cron in Tekton, see this [issue](https://github.com/tektoncd/triggers/issues/69) for more information. The community recommend cronjob in kubernetes, see [example](https://github.com/tektoncd/triggers/blob/master/examples/cron/README.md) here.
+* `delete_workflow(workflow_name)`: To be discussed: should we delete the tasks when delete the pipeline?


Yes we should but we should probably make this configurable for users to decide.

Something like delete_workflow(workflow_name, all) and delete_workflow(workflow_name, only)?

I am thinking something like delete_workflow(workflow_name, dependencies=True) where dependencies indicates whether to delete this workflow's dependencies/resources it owns. Just my two cents and open to other ideas as well.

terrytangyuan · 2020-09-28T14:22:04Z

docs/couler-tekton-design.md

+
+* `submit(config=workflow_config(schedule="* * * * 1"))`: There is no native support for cron in Tekton, see this [issue](https://github.com/tektoncd/triggers/issues/69) for more information. The community recommend cronjob in kubernetes, see [example](https://github.com/tektoncd/triggers/blob/master/examples/cron/README.md) here.
+* `delete_workflow(workflow_name)`: To be discussed: should we delete the tasks when delete the pipeline?
+* `list_workflow_records(workflow_name)`: After submitting the workflow(create PipelineRun in Tekton), there will be numbers of workflow records. A list function is necessary for users to show all the records.


Can the name be simplified to list_workflows()? We can probably add this in core Couler APIs as well.

Also do we have API for dealing with workflow runs? For argo is that a feature only accessible via argo serve API?

@binarycrayon Could you clarify what you meant by "dealing with workflow runs"?

By workflow records, I mean workflow runs, which is pipeline run in tekton. Should we change the name to list_workflow_runs and add another api like list_workflows?

PipelineRun is just an instantiation of a Tekton Pipeline. In Argo, there's no such additional concept. Should we abstract such backend specific details away so that users will only focus on defining Workflow that will be submitted and won't be confused with these two concepts?

In other words, we only provide one API list_workflows() in Couler. What do you think? If really necessary, we can perhaps consider exposing list_workflow_runs at Tekton backend level.

@terrytangyuan We could, I can add argo server API support as part of argo-client-python

I think we can keep both list_workflows and list_workflow_runs. Though this is less applicable to Argo, it's quite common for other backends - they have the concept of pipeline/workflow to represent the execution logic and pipeline/workflow runs to represent the result and state for each run either triggered by cron or on-demand.

For Argo, I think CronWorkflow also has the idea of (though not explicit) the workflow vs run.

Actually for Tekton users, my understanding is that they may not need to get a list of Pipelines at all. Instead, they want to get a list of PipelineRuns, which is actually "equivalent" to Argo's Workflow. Am I right?

If so, we actually only need one API which can be named either list_workflow_runs or list_workflows where list_workflows might be more generalized and concise. Correct me if I am wrong though.

terrytangyuan · 2020-09-28T14:24:29Z

docs/couler-tekton-design.md

+
+### Other Tekton Utilities
+
+* [Finally](https://github.com/tektoncd/pipeline/blob/master/docs/pipelines.md#adding-finally-to-the-pipeline)


Good to see this. The current Argo backend supports exit handlers in Argo Workflows as well. We can consider offer this in core Couler APIs. Could you add a function signature on how this API will look like?

docs/couler-tekton-design.md

merlintang · 2020-10-06T10:26:56Z

Meanwhile, can you give an example of the workflow for Tekton based on Couler in this doc? This would help others to follow and understand.

terrytangyuan

Merging this for now but feel free to continue the discussions here. Thanks! @FogDong

If anyone would like to help out with the implementation, please let us know!

merlintang reviewed Sep 28, 2020

View reviewed changes

terrytangyuan reviewed Sep 28, 2020

View reviewed changes

terrytangyuan mentioned this pull request Sep 28, 2020

Any plan to develop Python SDK for Tekton? tektoncd/pipeline#2441

Closed

terrytangyuan reviewed Sep 28, 2020

View reviewed changes

docs/couler-tekton-design.md Outdated Show resolved Hide resolved

Chore(docs): add tekton backend design doc

c2e29c7

FogDong force-pushed the master branch from 4b7f467 to c2e29c7 Compare September 30, 2020 02:42

terrytangyuan changed the title ~~[RFC]Chore(docs): add tekton backend design doc~~ docs: Add Tekton backend design doc Oct 7, 2020

terrytangyuan approved these changes Oct 13, 2020

View reviewed changes

terrytangyuan merged commit ce533cb into couler-proj:master Oct 13, 2020

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

docs: Add Tekton backend design doc #65

docs: Add Tekton backend design doc #65

FogDong commented Sep 28, 2020 •

edited

Loading

CLAassistant commented Sep 28, 2020 •

edited

Loading

merlintang left a comment

terrytangyuan Sep 28, 2020

terrytangyuan Sep 28, 2020

terrytangyuan Sep 28, 2020

FogDong Sep 30, 2020

terrytangyuan Sep 28, 2020

FogDong Sep 30, 2020

terrytangyuan Sep 30, 2020 •

edited

Loading

terrytangyuan Sep 28, 2020

binarycrayon Sep 28, 2020

terrytangyuan Sep 29, 2020

FogDong Sep 30, 2020 •

edited

Loading

terrytangyuan Sep 30, 2020 •

edited

Loading

binarycrayon Sep 30, 2020

xinbinhuang Sep 30, 2020

terrytangyuan Sep 30, 2020 •

edited

Loading

terrytangyuan Sep 28, 2020

merlintang commented Oct 6, 2020

terrytangyuan left a comment •

edited

Loading


		#### `while_loop(cond, func, args, *kwargs)`

		There is no native support for loops in Tekton for now.


		The `get_status`, `get_logs` remains the same as the Argo design.

		* `submit(config=workflow_config(schedule="* * * * 1"))`: There is no native support for cron in Tekton, see this [issue](https://github.com/tektoncd/triggers/issues/69) for more information. The community recommend cronjob in kubernetes, see [example](https://github.com/tektoncd/triggers/blob/master/examples/cron/README.md) here.


		### Other Tekton Utilities

		* [Finally](https://github.com/tektoncd/pipeline/blob/master/docs/pipelines.md#adding-finally-to-the-pipeline)

docs: Add Tekton backend design doc #65

docs: Add Tekton backend design doc #65

Conversation

FogDong commented Sep 28, 2020 • edited Loading

What changes were proposed in this pull request?

Why are the changes needed?

Does this PR introduce any user-facing change?

How was this patch tested?

CLAassistant commented Sep 28, 2020 • edited Loading

merlintang left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

terrytangyuan Sep 30, 2020 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

FogDong Sep 30, 2020 • edited Loading

Choose a reason for hiding this comment

terrytangyuan Sep 30, 2020 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

terrytangyuan Sep 30, 2020 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

merlintang commented Oct 6, 2020

terrytangyuan left a comment • edited Loading

Choose a reason for hiding this comment

FogDong commented Sep 28, 2020 •

edited

Loading

CLAassistant commented Sep 28, 2020 •

edited

Loading

terrytangyuan Sep 30, 2020 •

edited

Loading

FogDong Sep 30, 2020 •

edited

Loading

terrytangyuan Sep 30, 2020 •

edited

Loading

terrytangyuan Sep 30, 2020 •

edited

Loading

terrytangyuan left a comment •

edited

Loading