-
Notifications
You must be signed in to change notification settings - Fork 13.7k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Databricks SQL operators #21363
Merged
Merged
Databricks SQL operators #21363
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
alexott
changed the title
[WIP-do-not-merge] Databricks SQL operator
Databricks SQL operator
Feb 6, 2022
pateash
reviewed
Feb 6, 2022
Thank you for review @pateash , but this is really far from review state - more refactoring is coming |
mik-laj
reviewed
Feb 6, 2022
alexott
force-pushed
the
databricks-sql-operator
branch
from
February 13, 2022 13:55
da9a71f
to
29bded7
Compare
docs/apache-airflow-providers-databricks/connections/databricks.rst
Outdated
Show resolved
Hide resolved
alexott
force-pushed
the
databricks-sql-operator
branch
2 times, most recently
from
February 20, 2022 10:57
e74f505
to
74d2e87
Compare
@potiuk Jarek - would it be possible to review the changes? |
potiuk
approved these changes
Feb 26, 2022
You need to rebase @alexott |
No documentation & tests yet
Still need to fix existing tests & add tests for Databricks SQL hook & operator
This includes: * identifying SQL Endpoint by name * allow to output results into a CSV/JSON/JSONL file * fix tests for DatabricksHook * address most of the comments
…rator Co-authored-by: Lennart Kats (databricks) <lennart.kats@databricks.com>
Split documentation for operators into separate pages & add more content and examples.
alexott
force-pushed
the
databricks-sql-operator
branch
from
February 27, 2022 10:22
d95f421
to
958c6be
Compare
@potiuk done. thank you for review |
Tests are failing though :( |
alexott
force-pushed
the
databricks-sql-operator
branch
from
February 27, 2022 12:31
958c6be
to
855aee4
Compare
🤦 forgot that tests are referring to the |
potiuk
approved these changes
Feb 27, 2022
jedcunningham
added
the
changelog:skip
Changes that should be skipped from the changelog (CI, tests, etc..)
label
Feb 28, 2022
84 tasks
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Labels
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
This PR adds new operators to Databricks provider:
DatabrickSqlOperator
that allows to execute SQL commands against Databricks SQL Endpoints and Databricks clusters.DatabricksCopyIntoOperator
(built on top ofDatabrickSqlOperator
) that allows to import data into Databricks tables.This operator uses the same connection as other Databricks operators (although it could be discussed), if having a dedicated connection make sense as we can further customize it with specific input fields, etc.
Another possible improvement - make the
databricks-sql-connector
dependency optional, but I'm not sure how to make it correctly in Airflowcloses: #21030
closes: #21376