Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Support DAG glow #1

Open
hanxuanliang opened this issue Feb 28, 2024 · 2 comments
Open

Support DAG glow #1

hanxuanliang opened this issue Feb 28, 2024 · 2 comments

Comments

@hanxuanliang
Copy link

Will the kis-flow support DAG scheduling later?

pipeline:
    - depend_on: flow1

like the above config

@aceld
Copy link
Owner

aceld commented Mar 1, 2024

The current Kis-flow does not yet support DAG (Directed Acyclic Graph) scheduling, but we can perform parallel streaming operations.

For example, in cases like combining flow1 and flow2 to generate flow3:

We can have FunctionS and FunctionL nodes, where FunctionS acts as a storage node and FunctionL serves as a read/write node. These nodes can link the two flows together. Alternatively, we can have flow1's result sent to a message queue, and then flow2 can use this as a data source, initiating another flow to execute.

For configurations like:

pipeline:
    - depend_on: flow1
like the above config

This configuration format is not yet supported. However, it's a very good suggestion. The approaches mentioned for Kis-flow can be adapted to a configuration-based approach.

@hanxuanliang
Copy link
Author

Maybe DAG is a little more common. It may also be necessary to introduce the docker runtime environment

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

2 participants