Skip to content

🐛 Destination snowflake: Create final tables with uppercase naming - #30056

Merged
Octavia The Approver (octavia-approvington) merged 22 commits into
masterfrom
edgao/dv2/snowflake/case_sensitivity
Sep 5, 2023
Merged

Octavia The Approver (octavia-approvington) merged 22 commits into
masterfrom
edgao/dv2/snowflake/case_sensitivity

Conversation

@edgao

@edgao Edward Gao (edgao) commented Aug 31, 2023 •

Copy link
Copy Markdown
Contributor

closes #30010

The interesting changes are in SnowflakeDestinationHandler / SnowflakeSqlGenerator; everything else is just updating tests. (T+D tests are passing locally, at least)

See also #30068, which implements the migration for existing tables. I'll merge that PR into this one, then release this PR to master.

todo:

@github-actions

github-actions Bot commented Aug 31, 2023 •

Copy link
Copy Markdown
Contributor

Before Merging a Connector Pull Request

Wow! What a great pull request you have here! 🎉

To merge this PR, ensure the following has been done/considered for each connector added or updated:

  • PR name follows PR naming conventions
  • Breaking changes are considered. If a Breaking Change is being introduced, ensure an Airbyte engineer has created a Breaking Change Plan.
  • Connector version has been incremented in the Dockerfile and metadata.yaml according to our Semantic Versioning for Connectors guidelines
  • You've updated the connector's metadata.yaml file any other relevant changes, including a breakingChanges entry for major version bumps. See metadata.yaml docs
  • Secrets in the connector's spec are annotated with airbyte_secret
  • All documentation files are up to date. (README.md, bootstrap.md, docs.md, etc...)
  • Changelog updated in docs/integrations/<source or destination>/<name>.md with an entry for the new version. See changelog example
  • Migration guide updated in docs/integrations/<source or destination>/<name>-migrations.md with an entry for the new version, if the version is a breaking change. See migration guide example
  • If set, you've ensured the icon is present in the platform-internal repo. (Docs)

If the checklist is complete, but the CI check is failing,

  1. Check for hidden checklists in your PR description

  2. Toggle the github label checklist-action-run on/off to re-run the checklist CI.

@edgao
Edward Gao (edgao) force-pushed the edgao/dv2/snowflake/case_sensitivity branch from b226c19 to befa893 Compare August 31, 2023 20:26
@edgao
Edward Gao (edgao) marked this pull request as ready for review August 31, 2023 21:26
@edgao
Edward Gao (edgao) requested a review from a team as a code owner August 31, 2023 21:26
@github-actions

This comment was marked as outdated.

@evantahler Evan Tahler (evantahler) left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

There's a lot of toUpperCase() in the implementation code. I thought we wanted to let the DB evaluate these these to UPPERCASE lazily?

Comment on lines +48 to +50
unescapeIdentifier(databaseName).toUpperCase(),
unescapeIdentifier(schema).toUpperCase(),
unescapeIdentifier(table).toUpperCase()).stream()

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is a test util, so maybe it doens't matter, but I thought we wanted to let the DB evaluate the upper-case-ness of table names

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I overruled that decision https://airbytehq-team.slack.com/archives/C03C4AVJWG4/p1693495091006359 (but open to continued discussion!)

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yes!

with snowflake uppercasing via not-quoting, we’re losing the ability to use special characters / reserved words in namespace/table/column names. That opens up a whole can of worms around escaping things, which I’d rather not deal with right now.

That makes a lot of sense. 👍

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

this is still technically a user-facing decision (e.g. would they rather get "FOO$BAR" or FOO_BAR). But that's a much smaller blast radius

@github-actions

This comment was marked as outdated.

@octavia-squidington-iii Octavia Squidington III (octavia-squidington-iii) added the area/documentation Improvements or additions to documentation label Sep 1, 2023

### Snowflake

1. `destination-snowflake` is now case-sensitive, and was not previously. This means that if you have a source stream "users", `destination-snowflake` would have previously created a "USERS" table in your data warehouse. We now correctly create a "users" table.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Alexandre Cuoci (@Hesperide) fyi - I'm deleting the case-sensitivity thing from snowflake's upgrade guide.

@edgao Edward Gao (edgao) changed the title Destination snowflake: Create final tables with uppercase naming 🐛 Destination snowflake: Create final tables with uppercase naming Sep 1, 2023
@github-actions

This comment was marked as outdated.

@github-actions

This comment was marked as outdated.

@github-actions

This comment was marked as outdated.

@github-actions

This comment was marked as outdated.

@github-actions

This comment was marked as outdated.

@github-actions

This comment was marked as outdated.

@github-actions

This comment was marked as outdated.

@github-actions

This comment was marked as outdated.

@github-actions

This comment was marked as off-topic.

@github-actions

github-actions Bot commented Sep 5, 2023

Copy link
Copy Markdown
Contributor

destination-bigquery test report (commit d8c27a2e81) - ✅

⏲️ Total pipeline duration: 10mn44s

Step Result
Java Connector Unit Tests ✅
Build connector tar ✅
Build destination-bigquery docker image for platform linux/x86_64 ✅
Java Connector Integration Tests ✅
Validate airbyte-integrations/connectors/destination-bigquery/metadata.yaml ✅
Connector version semver check ✅
Connector version increment check ✅
QA checks ✅

🔗 View the logs here

☁️ View runs for commit in Dagger Cloud

Please note that tests are only run on PR ready for review. Please set your PR to draft mode to not flood the CI engine and upstream service on following commits.
You can run the same pipeline locally on this branch with the airbyte-ci tool with the following command

airbyte-ci connectors --name=destination-bigquery test

@github-actions

github-actions Bot commented Sep 5, 2023

Copy link
Copy Markdown
Contributor

destination-snowflake test report (commit d8c27a2e81) - ✅

⏲️ Total pipeline duration: 03mn13s

Step Result
Java Connector Unit Tests ✅
Build connector tar ✅
Build destination-snowflake docker image for platform linux/x86_64 ✅
Java Connector Integration Tests ✅
Validate airbyte-integrations/connectors/destination-snowflake/metadata.yaml ✅
Connector version semver check ✅
Connector version increment check ✅
QA checks ✅

🔗 View the logs here

☁️ View runs for commit in Dagger Cloud

Please note that tests are only run on PR ready for review. Please set your PR to draft mode to not flood the CI engine and upstream service on following commits.
You can run the same pipeline locally on this branch with the airbyte-ci tool with the following command

airbyte-ci connectors --name=destination-snowflake test

@edgao

Copy link
Copy Markdown
Contributor Author

source-hubplanner is an unrelated autoformat change. Will approve-and-merge this PR.

@edgao

Copy link
Copy Markdown
Contributor Author

/approve-and-merge reason="failing test is due to unrelated connector; test only ran because of autoformat. bigquery+snowflake have green CI"

@octavia-approvington

Copy link
Copy Markdown
Contributor

Send it!
rocket socks

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Revert case-senstivity in Snowflake final tables (and don't do it in Redshift)

5 participants