Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[Question] Dependencies between parallel subtasks? #1657

Open
FredericoCoelhoNunes opened this issue Oct 8, 2021 · 0 comments
Open

[Question] Dependencies between parallel subtasks? #1657

FredericoCoelhoNunes opened this issue Oct 8, 2021 · 0 comments

Comments

@FredericoCoelhoNunes
Copy link

FredericoCoelhoNunes commented Oct 8, 2021

Is there any way to run two tasks (with subtasks) in parallel, but setting some dependencies between the subtasks?

A concrete example:

  • task A: A1, A2, A3
  • task B: B1, B2, B3

Requirements:

  • A and B run in parallel
  • B3 must wait for B2 and A2 to finish

Using Airflow (the only other orchestrator I have used), this would be done in the following way:

A1 >> A2 >> A3
B1 >> B2 >> B3
A2 >> B3

Is there any way to replicate this functionality using DigDag's declarative syntax?

Thank you.


EDIT: I have thought about two options,

option 1: doesn't exactly work because it makes a3 also wait for b2:

+task1:
  _parallel: true
  +a:
    +a1: ...
    +a2: ...
  +b:
    +b1: ...
    +b2: ...

+task2:
  _parallel: true
  +a:
    +a3: ...
  +b:
    +b3: ...

option 2: doesn't exactly work because then b3 also has to wait for a3

+task1:
  _parallel: true
  +a:
    +a1: ...
    +a2: ...
    +a3: ...
  +b:
    +b1: ...
    +b2: ...

+task2:
  +b:
    +b3: ...
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

1 participant