Skip to content

Add direct GCS export to DatabricksSqlOperator with Parquet/Avro support #60501

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Closed
haseebmalik18 wants to merge 20 commits into from
Closed

Add direct GCS export to DatabricksSqlOperator with Parquet/Avro support #60501

haseebmalik18 wants to merge 20 commits into from

Conversation

@haseebmalik18
Copy link
Contributor

@haseebmalik18 haseebmalik18 commented Jan 14, 2026

Add direct GCS export capability to DatabricksSqlOperator with Parquet and Avro format support.

  • Add gcp_conn_id parameter for GCS authentication
  • Add parquet and avro to supported output_format values
  • Support gs:// URIs in output_path for direct GCS export
  • Implement type mapping from Databricks SQL types to PyArrow/Avro schemas

closes: #55128

haseebmalik18 and others added 20 commits December 22, 2025 22:22
@haseebmalik18
Add E2E tests for DAG audit log functionality (apache#59684)
b2844a0
@vatsrahul1001
Merge branch 'main' into main
4cdcbad
@haseebmalik18
Add E2E tests for DAG audit log functionality apache#59684
66f1251
@haseebmalik18
Merge branch 'main' into main
9df2836
@vatsrahul1001
Merge branch 'main' into main
4604abf
@haseebmalik18
Add E2E tests for DAG audit log functionality apache#59684
3dbc6fc
@haseebmalik18
Merge branch 'main' of https://github.com/haseebmalik18/airflow
26b0de8
@vatsrahul1001
Merge branch 'main' into main
e070948
@haseebmalik18
Add E2E tests for DAG audit log functionality apache#59684
55b8d71
@haseebmalik18
Add E2E tests for DAG audit log functionality apache#59684
834fba1
@haseebmalik18
Add E2E tests for DAG audit log functionality apache#59684
ec2bf04
@haseebmalik18
Add E2E tests for DAG audit log functionality apache#59684
40853ed
@haseebmalik18
Add E2E tests for DAG audit log functionality apache#59684
0b91e49
@vatsrahul1001
Merge branch 'main' into main
09d1239
@haseebmalik18
Add E2E tests for DAG audit log functionality apache#59684
9a59687
@haseebmalik18
Merge branch 'main' of https://github.com/haseebmalik18/airflow
cf2dd57
@vatsrahul1001
Merge branch 'main' into main
df4870d
@haseebmalik18
Add E2E tests for DAG audit log functionality apache#59684
e5a3d49
@vatsrahul1001
Merge branch 'main' into main
f74fc4d
@haseebmalik18
Add direct GCS export to DatabricksSqlOperator with Parquet/Avro supp…
eed446a 
…ort apache#55128
@boring-cyborg boring-cyborg bot added area:providers area:UI Related to UI/UX. For Frontend Developers. provider:databricks labels Jan 14, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@vatsrahul1001 vatsrahul1001 Awaiting requested review from vatsrahul1001 vatsrahul1001 is a code owner

@choo121600 choo121600 Awaiting requested review from choo121600 choo121600 is a code owner

Assignees

No one assigned

Labels

area:providers area:UI Related to UI/UX. For Frontend Developers. provider:databricks

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

Add direct GCS export to DatabricksSqlOperator with Parquet/Avro support

2 participants

@haseebmalik18 @vatsrahul1001