🐛 Fix normalization issue with quoted & case sensitive columns #9317
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
ChristopheDuong
changed the base branch from
master
to
chris/fix-bq-normalization-scd-float
|
/test connector=bases/base-normalization
|
|
/test connector=bases/base-normalization
|
tuliren
approved these changes
Jan 5, 2022
Comment on lines
+27
to
+28
| "User id", | ||
| "user id", |
There was a problem hiding this comment.
Sorry that I don't understand how the first User id is not converted to lower case. The main change in the Python file is this one I think:
- if not is_quoted and not self.needs_quotes(input_name):
- result = input_name.lower()
+ result = input_name.lower()which seems to universally convert names to lower case. Can you elaborate?
There was a problem hiding this comment.
This file you commented on is the output for Postgres, not mssql (i think the mssql one is not versioned in git).
- In Postgres, when quoted, the identifier's case matters (so "User Id", "User id" and "user id" are different column names). So nothing has changed here.
- But as the integration test reveals and as the reported On-Call issue reports, MSSQL does not make a difference with these 3 columns names with different cases and considers them all to be equal, and thus conflicting.
So, to mirror this behavior with MSSQL in normalization code, I universally lowercase all column identifiers for MSSQL only even if they are quoted. As a consequence, normalization now detects column name conflicts and will resolve it by renaming conflicting names with extra '_1', '_2' etc
There was a problem hiding this comment.
Actually, I'm going to change the PR and isolate it to normalize this sort of casing only when looking for column name conflicts...
There was a problem hiding this comment.
I see. Thanks for the explanation.
|
/test connector=bases/base-normalization
|
ChristopheDuong
commented
Jan 6, 2022
Comment on lines
10
to
14
| json_value(_airbyte_data, ''$."User Id"'') as "User Id", | ||
| json_value(_airbyte_data, ''$."user_id"'') as user_id, | ||
| json_value(_airbyte_data, ''$."User id"'') as "User id_1", | ||
| json_value(_airbyte_data, ''$."user id"'') as "user id_2", | ||
| json_value(_airbyte_data, ''$."UserId"'') as userid, |
There was a problem hiding this comment.
@tuliren here is the output of the new test for MSSQL
ChristopheDuong
changed the title
Base automatically changed from
chris/fix-bq-normalization-scd-float
to
master
January 6, 2022 17:49
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
What
Relates to https://github.com/airbytehq/oncall/issues/84
How
It seems MSSQL does care about case sensitivity of columns (even quoted ones).
(This might be depending on a "collation" settings of the server too)
But this PR makes it so we always resolve column conflicts by lowercasing all identifiers.
When I added the new test case, it surfaced similar behavior/errors on other destinations, so I'm fixing it there too.
(bigquery/mysql)
Recommended reading order
x.javay.python