Skip to content

Destination Bigquery: RC2 - fix column name handling - #60302

Merged
Edward Gao (edgao) merged 3 commits into
masterfrom
edgao/bigquery_fix_special_chars
May 15, 2025
Merged

Edward Gao (edgao) merged 3 commits into
masterfrom
edgao/bigquery_fix_special_chars

Conversation

@edgao

Copy link
Copy Markdown
Contributor

two fixes here, in separate commits:

  1. Fix SQL generation - we were generating invalid SQL when the cursor column of a dedup sync contains a special character, because we were using the original column name, rather than the escaped column name
  2. Fix column collision handling - we weren't correctly detecting name collisions because we were comparing the display name in addition to the canonical name.

this will fix:

  • source-google-ads - the cursor is segments.date, which gets turned into segments_date
  • some random source-mongodb sync, which had two columns allowsMultiUse and AllowsMultiUse

also, intellij told me to declare the TableId type explicitly on toTableId 🤷

@vercel

vercel Bot commented May 15, 2025 •

Copy link
Copy Markdown

The latest updates on your projects. Learn more about Vercel for Git ↗︎

Name Status Preview Comments Updated (UTC)
airbyte-docs ✅ Ready (Inspect) Visit Preview 💬 Add feedback May 15, 2025 8:30pm

// "order" is a reserved word in many sql engines
"order" to stringType,
"ProperCase" to stringType,
"Foo.Bar" to stringType,

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

this doesn't actually catch the bug (b/c the bug was specifically in dedup syncs). But it's cheap to add, so why not 🤷

@edgao
Edward Gao (edgao) marked this pull request as ready for review May 15, 2025 20:22
@edgao
Edward Gao (edgao) requested a review from a team as a code owner May 15, 2025 20:22
""
} else if (importType.cursor.size == 1) {
val columnName = importType.cursor.first()
val columnName = columnNameMapping[importType.cursor.first()]!!

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Nit: No tests for this change?

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

it's the new testFunkyCharactersDedup case

@github-actions

Copy link
Copy Markdown
Contributor

destination-bigquery Connector Test Results

172 tests   150 ✅  50m 49s ⏱️
 11 suites   22 💤
 11 files      0 ❌

Results for commit 9aad3dc.

@edgao
Edward Gao (edgao) enabled auto-merge (squash) May 15, 2025 21:22
@edgao
Edward Gao (edgao) merged commit 6fad1d4 into master May 15, 2025
@edgao
Edward Gao (edgao) deleted the edgao/bigquery_fix_special_chars branch May 15, 2025 21:24

This branch was successfully deployed

1 active deployment
Preview — 9aad3dc9 Deployed May 15, 2025 by vercel[bot]
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants