New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Add the ability to interact between local dataframes and Delta tables… #1397
Conversation
Codecov ReportBase: 88.91% // Head: 97.32% // Increases project coverage by
Additional details and impacted files@@ Coverage Diff @@
## main #1397 +/- ##
==========================================
+ Coverage 88.91% 97.32% +8.41%
==========================================
Files 60 19 -41
Lines 3120 672 -2448
Branches 368 0 -368
==========================================
- Hits 2774 654 -2120
+ Misses 252 18 -234
+ Partials 94 0 -94 Help us with your feedback. Take ten seconds to tell us how you rate us. Have a feature suggestion? Share it here. ☔ View full report at Codecov. |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Thanks @dimberman, I'm happy for this to be merged once:
- row limit is -1 by default (and that is when it is unlimited)
- we remove Postgres from the test name
25a4a81
to
ea5fe75
Compare
ea5fe75
to
ea4f424
Compare
…pushing local dataframes into spark delta tables
ea4f424
to
b206bb5
Compare
… in databricks
Description
Currently there is no established path for moving a dataframe into a databricks delta table except to manually save that dataframe as a local file and then push that file into delta using aql.load_file.
What is the new behavior?
This PR allows users to pass local dataframes into delta files using all of the existing methods withing the astro SDK (e.g. passing it into a transform function). Users should not need to learn any new functionality as all of this will work out of the box with existing aql based DAGs.
Does this introduce a breaking change?
No.
Checklist