Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add TaskGroup support #29

Merged
merged 5 commits into from
Apr 11, 2023
Merged

Add TaskGroup support #29

merged 5 commits into from
Apr 11, 2023

Conversation

jlaneve
Copy link
Collaborator

@jlaneve jlaneve commented Apr 11, 2023

closes #27

This PR makes a few small changes to support for Airflow TaskGroups in a Databricks workflow:

  • it traverses up the tree of task groups to see if the task is in a Databricks Workflow TaskGroup instead of just the immediate parent
  • instead of iterating over direct children of the Databricks Workflow TaskGroup, it first flattens the children by recursively removing task groups
  • small bugfix in the Tiltfile to fix the dev folder sync
  • adds an example dag to the dev directory using TaskGroups

I've run the example DAG in Astronomer's Databricks account, and it worked like a charm! Worth noting that Databricks themselves don't have support for TaskGroups/dependency groups, so this was pretty cool to put together.

Also worth noting that in the process of putting this together, I uncovered a bug I filed to be addressed separately.

Screen Shot 2023-04-11 at 12 23 38 AM

@codecov-commenter
Copy link

Codecov Report

Patch coverage: 76.66% and project coverage change: -0.55 ⚠️

Comparison is base (0538602) 78.82% compared to head (1ddcdc5) 78.28%.

Additional details and impacted files
@@            Coverage Diff             @@
##             main      #29      +/-   ##
==========================================
- Coverage   78.82%   78.28%   -0.55%     
==========================================
  Files           5        5              
  Lines         425      442      +17     
  Branches       47       52       +5     
==========================================
+ Hits          335      346      +11     
- Misses         73       76       +3     
- Partials       17       20       +3     
Impacted Files Coverage Δ
src/astro_databricks/operators/notebook.py 89.52% <63.63%> (-3.19%) ⬇️
src/astro_databricks/operators/workflow.py 83.46% <84.21%> (-0.57%) ⬇️

Help us with your feedback. Take ten seconds to tell us how you rate us. Have a feature suggestion? Share it here.

☔ View full report in Codecov by Sentry.
📢 Do you have feedback about the report comment? Let us know in this issue.

@jlaneve jlaneve merged commit ea9d8cf into main Apr 11, 2023
@jlaneve jlaneve deleted the task-groups branch April 11, 2023 19:17
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

Add support for TaskGroups from within the Databricks Workflow
3 participants