Databricks: Add support for execution of SQL commands against clusters/SQL endpoints

[alexott]

Description

Airflow supports communication with different databases using native interfaces or JDBC. But it's harder to use JDBC operator with Databricks as it requires additional downloads, etc. It would be really useful to allow to extract data from Databricks SQL endpoints/clusters without need to install additional objects, like, JDBC/ODBC drivers.

I'm opening issue just to visibility, as I'm starting to work on this functionality and wants to avoid doing double work

Use case/motivation

No response

Related issues

No response

Are you willing to submit a PR?

• Yes I am willing to submit a PR!

Code of Conduct

• I agree to follow this project's Code of Conduct

[boring-cyborg]

Thanks for opening your first issue here! Be sure to follow the issue template!

[potiuk]

ASsigned you to it @alexott

[mik-laj]

Do you know where the source code for the databricks-sql-connector package is available? The metadata for this package specifies that it is released under the Apache Software License (http://www.apache.org/licenses/LICENSE-2.0) license, so the source code should be public.

[alexott]

Not sure, need to check

[mik-laj]

It would be great if this information were added to the package metadata.

airflow/setup.cfg

Lines 60 to 66 in d5cafc9

	project_urls =
	Documentation=https://airflow.apache.org/docs/
	Bug Tracker=https://github.com/apache/airflow/issues
	Source Code=https://github.com/apache/airflow
	Slack Chat=https://s.apache.org/airflow-slack
	Twitter=https://twitter.com/ApacheAirflow
	YouTube=https://www.youtube.com/channel/UCSXwxpWZQ7XZ1WL3wqevChA/

[mik-laj]

@juliuszsompolski @NiallEgan Do you know where the source code for the databricks-sql-connector package is available?

[NiallEgan]

Hi @mik-laj, we're still working on hosting the source code in a public repo. We hope to make it more publicly available over the coming months.