🐛 Destination Databricks: Fix destination check test table collisions when multiple connections write to same schema. - #55232
Conversation
|
Sena Heydari (@sheyd) is attempting to deploy a commit to the Airbyte Growth Team on Vercel. A member of the Team first needs to authorize it. |
|
/bump-version type="patch" changelog="fix table name collision multiple connections same schema"
|
|
The latest updates on your projects. Learn more about Vercel for Git ↗︎
|
M. Marx (marcosmarxm)
left a comment
There was a problem hiding this comment.
Code change looks good. Sena Heydari (@sheyd) need final approval from destination team
cc Edward Gao (@edgao)
Edward Gao (edgao)
left a comment
There was a problem hiding this comment.
why is the version bumping to 3.3.5 instead of 3.3.4?
otherwise lgtm
|
Edward Gao (@edgao) I bumped it manually in this commit, and then the bump from GH comment job automatically versioned it an additional time. My mistake for bumping it manually (I think I misread the Contribution Guidelines on how to bump versions), so we ended up two PATCH points ahead instead of one. Happy to follow your guidance on how to remedy to get us to 3.3.4 if it's a blocker (e.g. pushing a commit to mark it down to 3.3.3 and have the comment Patch generation job re-run to get to 3.3.4), but also happy to leave as is and merge if that works. |
… when multiple connections write to same schema. (airbytehq#55232) Co-authored-by: Octavia Squidington III <octavia-squidington-iii@users.noreply.github.com>
What
This PR is an attempt to resolve #44929.
In our environment, we're trying to create connections from multiple upstream sources (e.g. one connection per PostgreSQL source table) all writing to the same Unity Catalog based schema. When we run multiple connections, we get destination errors that arise from what appears to be a race condition where the
airbyte_check_test_tablenames are all trying to be created/dropped concurrently.How
This fix adds a UUID to the
airbyte_check_test_tableto ensure each connection check table value is unique.Review guide
airbyte-integrations/connectors/destination-databricks/src/main/kotlin/io/airbyte/integrations/destination/databricks/DatabricksDestination.ktUser Impact
airbyte_check_test_table.Can this PR be safely reverted and rolled back?