🎉 Destination Databricks: Support Azure storage (#15140) by abaerptc · Pull Request #15329 · airbytehq/airbyte

Ashley Baer (abaerptc) · 2022-08-04T19:24:30Z

What

Adds Azure storage as an option for Databricks destination.
This PR resolves #15140.

How

Adds Azure storage as an alternative to S3.

🚨 User Impact 🚨

No breaking changes.

Pre-merge Checklist

Updating a connector

Community member or Airbyter

Grant edit access to maintainers (instructions)
Secrets in the connector's spec are annotated with airbyte_secret
Unit & integration tests added and passing. Community members, please provide proof of success locally e.g: screenshot or copy-paste unit, integration, and acceptance test output. To run acceptance tests for a Python connector, follow instructions in the README. For java connectors run ./gradlew :airbyte-integrations:connectors:<name>:integrationTest.
Code reviews completed
Documentation updated
- Connector's README.md
- Connector's bootstrap.md. See description and examples
- Changelog updated in docs/integrations/<source or destination>/<name>.md including changelog. See changelog example
PR name follows PR naming conventions

Airbyter

If this is a community PR, the Airbyte engineer reviewing this PR is responsible for the below items.

Create a non-forked branch based on this PR and test the below items on it
Build is successful
If new credentials are required for use in CI, add them to GSM. Instructions.
/test connector=connectors/<name> command is passing
New Connector version released on Dockerhub and connector version bumped by running the /publish command described here

Tests

Unit

Integration

None.

Acceptance

Cannot run S3 acceptance test locally.

CLAassistant · 2022-08-04T19:24:35Z

All committers have signed the CLA.

Ashley Baer (abaerptc) · 2022-08-09T13:29:24Z

The Azure integration test passes, but I don't have access to Databricks on AWS, and it's proving difficult to configure Azure Databricks to write to S3, so I'm not able to fully run the S3 integration tests. I have confirmed that the data gets written to S3 fine, but still need to confirm that the tables can be created from the S3 location in Databricks. Could someone from Airbyte please trigger the integration tests to run from this PR?

M. Marx (marcosmarxm) · 2022-08-09T17:10:07Z

/test connector=connectors/destination-databricks

🕑 connectors/destination-databricks https://github.com/airbytehq/airbyte/actions/runs/2827034508
❌ connectors/destination-databricks https://github.com/airbytehq/airbyte/actions/runs/2827034508
🐛 https://gradle.com/s/77uzxvmvh373e

Build Failed

Test summary info:

Could not find result summary

Ashley Baer (abaerptc) · 2022-08-09T21:52:25Z

I pushed a new commit that should fix the error seen in the previous run of the S3 acceptance test.

The Azure acceptance test will require the addition of a new secrets file by an Airbyte member (see sample_secrets/azure_config.json).

M. Marx (marcosmarxm) · 2022-08-09T22:42:31Z

/test connector=connectors/destination-databricks

I added azure_config to Airbyte secret manager.

Liren Tu (tuliren) · 2022-08-10T03:16:58Z

Ashley Baer (@abaerptc), thank you for the PR! Could you add me as a collaborator to your fork? In this way I can push changes to this branch if needed as I am working on merging it.

M. Marx (@marcosmarxm), there is another PR pending merging for Databricks. Would you mind me doing the coordination and merging it later?

M. Marx (marcosmarxm) · 2022-08-10T11:47:43Z

/test connector=connectors/destination-databricks

🕑 connectors/destination-databricks https://github.com/airbytehq/airbyte/actions/runs/2832361238
❌ connectors/destination-databricks https://github.com/airbytehq/airbyte/actions/runs/2832361238
🐛 https://gradle.com/s/xtf6vwhphaki2

Build Failed

Test summary info:

Could not find result summary

M. Marx (marcosmarxm) · 2022-08-10T11:48:06Z

M. Marx (@marcosmarxm), there is another PR pending merging for Databricks. Would you mind me doing the coordination and merging it later?

Sure Liren Tu (@tuliren) I'd just to confirm tests are running properly here.

M. Marx (marcosmarxm) · 2022-08-10T12:37:43Z

/test connector=connectors/destination-databricks

🕑 connectors/destination-databricks https://github.com/airbytehq/airbyte/actions/runs/2832664101
❌ connectors/destination-databricks https://github.com/airbytehq/airbyte/actions/runs/2832664101
🐛 https://gradle.com/s/nazigo4vbe2pe

Build Failed

Test summary info:

Could not find result summary

Ashley Baer (abaerptc) · 2022-08-10T12:43:46Z

Liren Tu (@tuliren) You're added as a collaborator.

M. Marx (marcosmarxm) · 2022-08-10T13:00:52Z

Ashley Baer (@abaerptc) now the integration tests are running with your changes!

Ashley Baer (abaerptc) · 2022-08-10T13:08:25Z

Ashley Baer (@abaerptc) now the integration tests are running with your changes!

M. Marx (@marcosmarxm) I see a few failures with messages like "Invalid configuration value detected for fs.azure.account.key". I think you need to give your Databricks instance access to your Azure account via the Spark configuration property fs.azure.account.key.

TamGB · 2022-09-05T15:26:30Z

Awesome PR Ashley Baer (@abaerptc) !

Just a side note, the AWS version supports destination pattern matching (see airbyte-integrations/connectors/destination-databricks/src/main/java/io/airbyte/integrations/destination/databricks/DatabricksS3StorageConfig.java), is this something you would consider adding? (of course, I am happy to help!)

Ashley Baer (abaerptc) · 2022-09-06T13:33:58Z

Awesome PR Ashley Baer (@abaerptc) !

Just a side note, the AWS version supports destination pattern matching (see airbyte-integrations/connectors/destination-databricks/src/main/java/io/airbyte/integrations/destination/databricks/DatabricksS3StorageConfig.java), is this something you would consider adding? (of course, I am happy to help!)

TamGB I don't object to having pattern matching added. However, I don't have the availability to work on it. Please go ahead and implement it if you'd find it valuable.

Pavel Kudinov (pkudinov) · 2022-09-23T19:06:46Z

Is this abstracted enough to add support for Databricks on Google Cloud (i.e. add GCS as a target bucket along with AWS S3, and Azure Blob)?

Ashley Baer (abaerptc) · 2022-09-23T19:20:43Z

Is this abstracted enough to add support for Databricks on Google Cloud (i.e. add GCS as a target bucket along with AWS S3, and Azure Blob)?

You should be able to add another implementation of the data storage method pretty easily based on what I've done here, yes.

Ashley Baer (abaerptc) · 2022-10-10T14:29:49Z

M. Marx (@marcosmarxm) Liren Tu (@tuliren) Any update on getting this merged in?

Nataly Merezhuk (natalyjazzviolin) · 2022-10-12T14:05:41Z

Ashley Baer (@abaerptc) looks like there are conflicts, could you resolve them please?

Liren Tu (@tuliren) there is a lot of community interest for this PR, would you be able to move it up in your queue for review/merging? Thank you in advance :)

Ashley Baer (abaerptc) · 2022-10-12T18:54:59Z

Nataly Merezhuk (@natalyjazzviolin) Conflicts are resolved.

M. Marx (marcosmarxm) · 2022-10-13T23:18:44Z

So sorry Ashley Baer (@abaerptc) for the long delay to reply. Liren Tu (@tuliren) is actively working to improve databricks destination for other features and to implement your contribution too. Soon he will return to you!

Liren Tu (tuliren) · 2022-10-14T19:56:16Z

Sorry about the delay.

To test the Azure integration, we would need a separate Databricks cluster. I was not able to do that so far. Given that it has been delayed for a long time, I will do some local testing, and mute the integration test related to Azure for now.

I target to merge it today.

Liren Tu (tuliren) · 2022-10-15T08:06:50Z

/test connector=connectors/destination-databricks

🕑 connectors/destination-databricks https://github.com/airbytehq/airbyte/actions/runs/3254956793
✅ connectors/destination-databricks https://github.com/airbytehq/airbyte/actions/runs/3254956793
No Python unittests run

Build Passed

Test summary info:

All Passed

Liren Tu (tuliren)

🎉

Liren Tu (tuliren) · 2022-10-15T08:22:12Z

/publish connector=connectors/destination-databricks run-tests=false

🕑 Publishing the following connectors:
connectors/destination-databricks
https://github.com/airbytehq/airbyte/actions/runs/3255000004

Connector	Did it publish?	Were definitions generated?
connectors/destination-databricks	✅	✅

if you have connectors that successfully published but failed definition generation, follow step 4 here ▶️

…airbyte into databricks-azure-15140

Liren Tu (tuliren) · 2022-10-15T08:39:33Z

Ashley Baer (@abaerptc), thank you very much for your contribution. Sorry about the long delay.

Please don't forget to remove me from your repo.

…rbytehq#15329) * 🎉 Destination Databricks: Support Azure storage (airbytehq#15140) * Update docs and version. * Revert unintentional change to S3 config parsing. * Revert unnecessary additional table property. * change spec.json * Ignore azure integration test * Add issue link * auto-bump connector version Co-authored-by: Marcos Marx <marcosmarxm@users.noreply.github.com> Co-authored-by: marcosmarxm <marcosmarxm@gmail.com> Co-authored-by: Liren Tu <tuliren@gmail.com> Co-authored-by: Octavia Squidington III <octavia-squidington-iii@users.noreply.github.com>

🎉 Destination Databricks: Support Azure storage (#15140)

962411b

Ashley Baer (abaerptc) requested a review from a team as a code owner August 4, 2022 19:24

github-actions Bot added the area/connectors Connector related issues label Aug 4, 2022

Octavia Squidington III (octavia-squidington-iii) added bounty community connectors/destination/azure-blob-storage connectors/destination/databricks labels Aug 4, 2022

Update docs and version.

65b9c01

github-actions Bot added the area/documentation Improvements or additions to documentation label Aug 4, 2022

Sajarin (sajarin) added internal and removed bounty labels Aug 5, 2022

Ashley Baer (abaerptc) added 2 commits August 8, 2022 14:27

Revert unintentional change to S3 config parsing.

29c3d3e

Merge branch 'master' into databricks-azure-15140

0914d64

Revert unnecessary additional table property.

ede2d24

M. Marx (marcosmarxm) and others added 2 commits August 10, 2022 08:12

Merge branch 'master' into databricks-azure-15140

baffd1c

change spec.json

2d0757e

Ashley Baer (abaerptc) mentioned this pull request Oct 10, 2022

🎉 Destination Databricks: Support Azure storage #15140

Closed

Merge remote-tracking branch 'master' into databricks-azure-15140

04bcca2

Ignore azure integration test

efd63a5

Liren Tu (tuliren) approved these changes Oct 15, 2022

View reviewed changes

Liren Tu (tuliren) and others added 3 commits October 15, 2022 01:31

Add issue link

14d186d

auto-bump connector version

9280509

Merge branch 'databricks-azure-15140' of https://github.com/abaerptc/…

6afe9a7

…airbyte into databricks-azure-15140

Liren Tu (tuliren) merged commit 1b134d8 into airbytehq:master Oct 15, 2022

Octavia Squidington III (octavia-squidington-iii) mentioned this pull request Oct 20, 2022

Bump Airbyte version from 0.40.15 to 0.40.16 #18268

Merged

Conversation

Ashley Baer (abaerptc) commented Aug 4, 2022 • edited by tuliren Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

What

How

Recommended reading order

🚨 User Impact 🚨

Pre-merge Checklist

Community member or Airbyter

Airbyter

Tests

Uh oh!

CLAassistant commented Aug 4, 2022 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Ashley Baer (abaerptc) commented Aug 9, 2022

Uh oh!

M. Marx (marcosmarxm) commented Aug 9, 2022 • edited by github-actions Bot Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Build Failed

Uh oh!

Ashley Baer (abaerptc) commented Aug 9, 2022

Uh oh!

M. Marx (marcosmarxm) commented Aug 9, 2022

Uh oh!

Liren Tu (tuliren) commented Aug 10, 2022

Uh oh!

M. Marx (marcosmarxm) commented Aug 10, 2022 • edited by github-actions Bot Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Build Failed

Uh oh!

M. Marx (marcosmarxm) commented Aug 10, 2022

Uh oh!

M. Marx (marcosmarxm) commented Aug 10, 2022 • edited by github-actions Bot Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Build Failed

Uh oh!

Ashley Baer (abaerptc) commented Aug 10, 2022

Uh oh!

M. Marx (marcosmarxm) commented Aug 10, 2022

Uh oh!

Ashley Baer (abaerptc) commented Aug 10, 2022

Uh oh!

TamGB commented Sep 5, 2022

Uh oh!

Ashley Baer (abaerptc) commented Sep 6, 2022

Uh oh!

Pavel Kudinov (pkudinov) commented Sep 23, 2022

Uh oh!

Ashley Baer (abaerptc) commented Sep 23, 2022

Uh oh!

Ashley Baer (abaerptc) commented Oct 10, 2022

Uh oh!

Nataly Merezhuk (natalyjazzviolin) commented Oct 12, 2022

Uh oh!

Ashley Baer (abaerptc) commented Oct 12, 2022

Uh oh!

M. Marx (marcosmarxm) commented Oct 13, 2022

Uh oh!

Liren Tu (tuliren) commented Oct 14, 2022

Uh oh!

Liren Tu (tuliren) commented Oct 15, 2022 • edited by github-actions Bot Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Build Passed

Uh oh!

Liren Tu (tuliren) left a comment

Choose a reason for hiding this comment

Uh oh!

Liren Tu (tuliren) commented Oct 15, 2022 • edited by github-actions Bot Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Liren Tu (tuliren) commented Oct 15, 2022

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

Ashley Baer (abaerptc) commented Aug 4, 2022 •

edited by tuliren

Loading

CLAassistant commented Aug 4, 2022 •

edited

Loading

M. Marx (marcosmarxm) commented Aug 9, 2022 •

edited by github-actions Bot

Loading

M. Marx (marcosmarxm) commented Aug 10, 2022 •

edited by github-actions Bot

Loading

M. Marx (marcosmarxm) commented Aug 10, 2022 •

edited by github-actions Bot

Loading

Liren Tu (tuliren) commented Oct 15, 2022 •

edited by github-actions Bot

Loading

Liren Tu (tuliren) commented Oct 15, 2022 •

edited by github-actions Bot

Loading