Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add support for more complex dependencies inside a pipeline #564

Open
fran6co opened this issue Mar 25, 2024 · 1 comment
Open

Add support for more complex dependencies inside a pipeline #564

fran6co opened this issue Mar 25, 2024 · 1 comment

Comments

@fran6co
Copy link
Contributor

fran6co commented Mar 25, 2024

To reduce latency we have the case of creating pipelines that have some parallel tasks inside that would make this graph:

a -> b -> d -> h -> i
  -> c -> e ->
  -> f -> g ->

where b, d, c,e and f,g run in parallel to each other and each one of them can be parallel or serial (so the next line waits for them accordingly). The pipeline waits for both of them to finish to move on to h.
I tried implementing this with a sub taskflow that has the parallel tasks, but it doesn't capture the dependencies the same way as it would need to make it serial if any of sub tasks are serial.

@fran6co
Copy link
Contributor Author

fran6co commented Mar 25, 2024

I checked Graph Processing Pipeline article but I don't see how would it work if there are more than 2 "forks" in the graph

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

1 participant