[RFC] Add initial design doc for core Couler API #24

terrytangyuan · 2020-08-31T16:00:45Z

Hi community,

We are opening this PR to share our thoughts on supporting multiple workflow engines. We'd appreciate any feedback and suggestions. In addition, if you are interested in contributing either a new backend or functionalities of the existing backend, please let us know in this pull request.

Signed-off-by: terrytangyuan <terrytangyuan@gmail.com>

FogDong · 2020-09-01T15:23:39Z

docs/couler-api-design.md

+
+Core operations (`couler.ops`):
+
+* `run_step(step_def)` where `step_def` is the step definition that can be a container spec, Python function,


Will this name step a little bit confusing since step means differently in Tekton and Argo?

In addition, I am a little curious about the design of the DAG in Couler. In Argo, DAG is one of the templates, but in Tekton, there is no concept of templates and users can simply run their tasks in DAG.

Great question! Thanks for chiming in.

I think it should be fine as long as we document it properly. In Couler, a step simply means a node in the workflow graph (the "smallest" unit in some sense) which is essentially the same for Argo and Tekton. In Tekton, a task consists of multiple steps and a pipeline consists of multiple tasks (example).

Perhaps we can model the analogy like the following:

step (Couler) = step (Argo) = step (Tekton)

reusable step (Couler) = template (Argo) = task (Tekton)

workflow (Couler) = workflow (Argo) = pipeline (Tekton)

Even though this may not be 100% accurate but we can try model this as closely as possible to cover most use cases. What do you think?

Just to extend the analogy above to other systems like Airflow, Dagster, and Prefect

Workflow Engine Couler Argo Tekton Airflow Dagster Prefect

Step Step step step task solid task

Composite steps Reusable step template task SubDag or TaskGroup composite solid ???

Worfklow Workflow workflow pipeline DAG pipeline Flow

I am not sure about the definition of Reusable step, but I assume it means composite steps where it references task for Tekton?

I think it will be nice to have a comparison table and keep it updated to help adoption for people from other backends and for implementation for new backends.

Thanks! Great idea. I'll add this table to the doc. By "reusable step", I meant parameterized template that can be used to define a step where users only have to specify a few parameters.

chinazj · 2020-09-03T10:43:04Z

some suggestions:

Do we need an initialization step?
tekton's PipelineResources should be added as a parameter to the step funcition. These parameters have different types. ** Treat parameters as a structure or class**.

type parameter struct {
      param_type 
      url
      value
}

terrytangyuan · 2020-09-03T12:51:00Z

@chinazj Thanks for the great questions. See my replies below:

Do you mean the initialization of the workflow? Currently the workflow and its attributes are managed globally and we have an experimental couler._cleanup() to clean up the states so users can start defining a new workflow. This may not be ideal/friendly to some users and it's definitely good to have an object/scope to perform initialization and hold the states. cc @chunyang-wen who had similar thoughts on this.
Yes, parameters/artifacts are definitely things we plan to support in the future. For Argo backend, we have input/output as args in the step function but we have not thought of an unified API for this yet. The plan is to provide a minimal set of standardized APIs first and then the community can help bring up and contribute new proposals on designs like parameters/artifacts.

xinbinhuang · 2020-09-03T18:23:37Z

Hi thanks for the great work!

I am curious about a few things about the goals of couler:

Who is the main audience?
What are the preferred programming paradigm (i.e. functional?)
What is the high level system design/model on how couler interact and manage backends ?
What are the main use cases and current workloads you have in production in Alibaba?

terrytangyuan · 2020-09-03T19:21:33Z

@xinbinhuang Thanks for the feedback and questions. Please see my replies below:

Data scientists and engineers who want to define workflows for machine learning application or DevOps/automation tasks, at ease without learning any complicated OOP and DSL.
Functional and imperative.
Have you seen the following in the doc? Basically we want to use this mechanism to switch backends as well as potentially providing configurations in submit(config=workflow_config(schedule="* * * * 1")) that may be backend-specific. Implementations for each backend can be flexible as long as the APIs specific to the backend can work together.
```
Backends (`couler.backends`):
* `get_backend()`
* `use_backend("argo")`
```
We cannot disclose any specific use cases but we currently use Couler for ML tasks mostly. The main benefits of Couler are: the standardized interface across different engines so it's easy to switch engines for different use cases for better performance; reusable templates; shared best practices/validation/optimizations, etc.

linkerzhang · 2020-09-07T08:16:50Z

docs/couler-api-design.md

+* `when(cond, if_op, else_op)` where
+    * `cond` can be any of the following predicates: `equal()`, `not_equal()`, `bigger()`, `smaller()`, `bigger_equal()`, `smaller_equal()`.
+    * The operation defined in `if_op` will be executed when `cond` is true. Otherwise, `else_op` will be executed.
+* `while_loop(cond, func, *args, **kwargs)` where


naming suggestion: while_loop -> loop?

"loop" by itself might be a bit confusing as it can also be interpreted as "for loop"

binarycrayon · 2020-09-12T16:37:35Z

docs/couler-api-design.md

+
+* `get_backend()`
+* `use_backend("argo")`
+


Can we add a parameters section? (or at least think about that behavior?)

I think the matrix definition proposed above is great, also would love to see how parameters are handled. I know that you may define global parameters, which were shared across the steps in a workflow (for airflow at least). This could be achieved in argo, I'm not sure how that is handled in tekton.

OK I see that is a future thing (input/output/parameters), though from my perspective this is one thing we want to get to early

We currently support input parameters via args, e.g. see this test for examples. However I have not put thoughts into other backends for parameters yet. It might require a dedicated design doc for that.

binarycrayon · 2020-09-13T04:10:20Z

docs/couler-api-design.md

+
+result = flip_coin()
+couler.when(couler.equal(result, "heads"), lambda: heads())
+couler.when(couler.equal(result, "tails"), lambda: tails())


I understand the functional beauty of this but having to specify the when away from the involved step function spreads the logics into two places

I'm not sure how tekton does this, but could we have it available as a decorator, e.g.

from couler import when, equal @when(equal(flip_coin().result, "heads") def heads(): ...

I feel it's more concise to "see" the dag flow when such logics are located together, especially when you have large dag that involves tons of steps.

That's definitely a good thought by having the conditional logic and the op closer to each other. One concern is that users like data scientists and analysts might not know decorators well and may get confused. Though we can consider this decorator as an additional convenience method that's equivalent to the existing API. Feel free to propose again in the future when the existing implementation is in place and we'll take a look together with other backends as well.

Good discussion. I have a little bit different opinion on it. Having a "decorator" looks to have the conditional logic sitting together with the op logic, however, it does not really show the full workflow, instead, the line 77/78/79 are showing the full workflow, like a "main" function.

I have a similar comment as @binarycrayon. I think it will be nice to provide @when, @task or other constructs to users, but I agree that we can provide another set of these "higher-level" functional APIs which utilize the "lower-level" APIs (i.e. run_step, etc)

yxue-kabam · 2020-09-15T04:11:57Z

docs/couler-api-design.md

+* `when(cond, if_op, else_op)` where
+    * `cond` can be any of the following predicates: `equal()`, `not_equal()`, `bigger()`, `smaller()`, `bigger_equal()`, `smaller_equal()`.
+    * The operation defined in `if_op` will be executed when `cond` is true. Otherwise, `else_op` will be executed.


are we missing a next op that simply specifies the order of steps? (e.g. a linear workflow)

Actually it was intentional since Couler's APIs are imperative so users can define the op and once it's declared it will be added automatically to the workflow as one of the steps. Users can also set the dependencies explicitly to construct a linear workflow if they want.

Should we add dag and set_dependencies in this doc?

Based on the current dag design, if users want to add a when judge, they can only add couler.when outside of the couler.dag. What's the best practice of this case?

paguos · 2020-09-27T21:33:46Z

Does couler support local execution of workflows? And it would be great to expand the API to support serverless frameworks like AWS lambda

terrytangyuan · 2020-09-28T00:30:51Z

@paguos Either local execution or serverless frameworks has not been taken into consideration yet. For Argo backend, Argo Events might be a potential direction. In the meantime, feel free to share any thoughts and possible approaches with us.

merlintang · 2020-09-28T15:01:42Z

/lgtm

terrytangyuan added 2 commits August 31, 2020 11:14

[RFC] Add initial design of core Couler API

454fd35

Signed-off-by: terrytangyuan <terrytangyuan@gmail.com>

Update

dc9cb87

Signed-off-by: terrytangyuan <terrytangyuan@gmail.com>

FogDong reviewed Sep 1, 2020

View reviewed changes

terrytangyuan mentioned this pull request Sep 3, 2020

Long term plan and documentations #34

Open

linkerzhang reviewed Sep 7, 2020

View reviewed changes

This was referenced Sep 9, 2020

Support Apache Airflow as a backend #43

Closed

Support Tekton Pipelines as a backend #44

Closed

binarycrayon reviewed Sep 12, 2020

View reviewed changes

binarycrayon reviewed Sep 13, 2020

View reviewed changes

yxue-kabam reviewed Sep 15, 2020

View reviewed changes

terrytangyuan mentioned this pull request Sep 28, 2020

docs: Add Tekton backend design doc #65

Merged

terrytangyuan added 2 commits September 28, 2020 10:48

Address comments

437a000

Update couler-api-design.md

fadc304

merlintang approved these changes Sep 28, 2020

View reviewed changes

terrytangyuan merged commit 9c3addb into master Sep 28, 2020

terrytangyuan deleted the api-design branch September 28, 2020 17:51

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[RFC] Add initial design doc for core Couler API #24

[RFC] Add initial design doc for core Couler API #24

terrytangyuan commented Aug 31, 2020 •

edited

Loading

FogDong Sep 1, 2020

terrytangyuan Sep 1, 2020 •

edited

Loading

FogDong Sep 2, 2020

xinbinhuang Sep 3, 2020

terrytangyuan Sep 3, 2020

chinazj commented Sep 3, 2020

terrytangyuan commented Sep 3, 2020

xinbinhuang commented Sep 3, 2020

terrytangyuan commented Sep 3, 2020 •

edited

Loading

linkerzhang Sep 7, 2020

terrytangyuan Sep 8, 2020 •

edited

Loading

binarycrayon Sep 12, 2020

binarycrayon Sep 12, 2020

terrytangyuan Sep 14, 2020

binarycrayon Sep 13, 2020

terrytangyuan Sep 14, 2020

linkerzhang Sep 15, 2020

xinbinhuang Sep 28, 2020

yxue-kabam Sep 15, 2020

terrytangyuan Sep 15, 2020

FogDong Sep 20, 2020

paguos commented Sep 27, 2020

terrytangyuan commented Sep 28, 2020 •

edited

Loading

merlintang commented Sep 28, 2020


		Core operations (`couler.ops`):

		* `run_step(step_def)` where `step_def` is the step definition that can be a container spec, Python function,

Workflow Engine	Couler	Argo	Tekton	Airflow	Dagster	Prefect
Step	Step	step	step	task	solid	task
Composite steps	Reusable step	template	task	SubDag or TaskGroup	composite solid	???
Worfklow	Workflow	workflow	pipeline	DAG	pipeline	Flow

[RFC] Add initial design doc for core Couler API #24

[RFC] Add initial design doc for core Couler API #24

Conversation

terrytangyuan commented Aug 31, 2020 • edited Loading

Choose a reason for hiding this comment

terrytangyuan Sep 1, 2020 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

chinazj commented Sep 3, 2020

terrytangyuan commented Sep 3, 2020

xinbinhuang commented Sep 3, 2020

terrytangyuan commented Sep 3, 2020 • edited Loading

Choose a reason for hiding this comment

terrytangyuan Sep 8, 2020 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

paguos commented Sep 27, 2020

terrytangyuan commented Sep 28, 2020 • edited Loading

merlintang commented Sep 28, 2020

terrytangyuan commented Aug 31, 2020 •

edited

Loading

terrytangyuan Sep 1, 2020 •

edited

Loading

terrytangyuan commented Sep 3, 2020 •

edited

Loading

terrytangyuan Sep 8, 2020 •

edited

Loading

terrytangyuan commented Sep 28, 2020 •

edited

Loading