Skip to content

🐛 Destination Databricks: Fix destination check test table collisions when multiple connections write to same schema. - #55232

Merged
M. Marx (marcosmarxm) merged 12 commits into
airbytehq:masterfrom
sheyd:sheyd/concurrency-issue-causing-table-creation-failures
Mar 13, 2025
Merged

🐛 Destination Databricks: Fix destination check test table collisions when multiple connections write to same schema.#55232
M. Marx (marcosmarxm) merged 12 commits into
airbytehq:masterfrom
sheyd:sheyd/concurrency-issue-causing-table-creation-failures

Conversation

@sheyd

@sheyd Sena Heydari (sheyd) commented Mar 6, 2025

Copy link
Copy Markdown
Contributor

What

This PR is an attempt to resolve #44929.

In our environment, we're trying to create connections from multiple upstream sources (e.g. one connection per PostgreSQL source table) all writing to the same Unity Catalog based schema. When we run multiple connections, we get destination errors that arise from what appears to be a race condition where the airbyte_check_test_table names are all trying to be created/dropped concurrently.

How

This fix adds a UUID to the airbyte_check_test_table to ensure each connection check table value is unique.

Review guide

  1. airbyte-integrations/connectors/destination-databricks/src/main/kotlin/io/airbyte/integrations/destination/databricks/DatabricksDestination.kt

User Impact

  • End-users shouldn't notice any impact, aside from logs having UUID names related to airbyte_check_test_table.

Can this PR be safely reverted and rolled back?

  • YES 💚
  • NO ❌

@sheyd
Sena Heydari (sheyd) requested a review from a team as a code owner March 6, 2025 17:24
@vercel

vercel Bot commented Mar 6, 2025

Copy link
Copy Markdown

Sena Heydari (@sheyd) is attempting to deploy a commit to the Airbyte Growth Team on Vercel.

A member of the Team first needs to authorize it.

@CLAassistant

CLAassistant commented Mar 6, 2025

Copy link
Copy Markdown

CLA assistant check
All committers have signed the CLA.

@marcosmarxm

M. Marx (marcosmarxm) commented Mar 7, 2025

Copy link
Copy Markdown
Contributor

/bump-version type="patch" changelog="fix table name collision multiple connections same schema"

Bump Version job started... Check job output.

✅ Changes applied successfully. (70bc5ba)

@vercel

vercel Bot commented Mar 7, 2025

Copy link
Copy Markdown

The latest updates on your projects. Learn more about Vercel for Git ↗︎

Name Status Preview Comments Updated (UTC)
airbyte-docs ✅ Ready (Inspect) Visit Preview 💬 Add feedback Mar 7, 2025 0:01am

@marcosmarxm M. Marx (marcosmarxm) left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Code change looks good. Sena Heydari (@sheyd) need final approval from destination team
cc Edward Gao (@edgao)

@github-project-automation github-project-automation Bot moved this from Backlog to Ready to Ship in 🧑‍🏭 Community Pull Requests Mar 13, 2025

@edgao Edward Gao (edgao) left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

why is the version bumping to 3.3.5 instead of 3.3.4?

otherwise lgtm

@sheyd

Sena Heydari (sheyd) commented Mar 13, 2025

Copy link
Copy Markdown
Contributor Author

Edward Gao (@edgao) I bumped it manually in this commit, and then the bump from GH comment job automatically versioned it an additional time. My mistake for bumping it manually (I think I misread the Contribution Guidelines on how to bump versions), so we ended up two PATCH points ahead instead of one.

Happy to follow your guidance on how to remedy to get us to 3.3.4 if it's a blocker (e.g. pushing a commit to mark it down to 3.3.3 and have the comment Patch generation job re-run to get to 3.3.4), but also happy to leave as is and merge if that works.

@marcosmarxm
M. Marx (marcosmarxm) merged commit e44d30a into airbytehq:master Mar 13, 2025
@github-project-automation github-project-automation Bot moved this from Ready to Ship to Done in 🧑‍🏭 Community Pull Requests Mar 13, 2025
@sheyd
Sena Heydari (sheyd) deleted the sheyd/concurrency-issue-causing-table-creation-failures branch March 17, 2025 19:45
Sven Pöche (Valgard) pushed a commit to mayflower/airbyte that referenced this pull request Mar 25, 2025
… when multiple connections write to same schema. (airbytehq#55232)

Co-authored-by: Octavia Squidington III <octavia-squidington-iii@users.noreply.github.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

area/connectors Connector related issues area/documentation Improvements or additions to documentation community connectors/destination/databricks

Projects

Development

Successfully merging this pull request may close these issues.

[destination-databricks] concurrency issue causing table creation failures

6 participants