Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Postgres: column name truncation spike #36805

Merged
merged 12 commits into from
Apr 10, 2024

Conversation

edgao
Copy link
Contributor

@edgao edgao commented Apr 3, 2024

stacked on #36620. See also #36808 for "just throw an error".

  • tried to just add a truncate thing
  • naive catalogparser collision detection doesn't help here
    • it just enters an infinite loop
  • add more logic to the collision solver to handle the truncation case

we end up with e.g. ddddddddddddddddddddddddddddddddddddddddddddddddddddddddddddddd (63 chars long) and ddddddddddddddddddddddddddddd461dddddddddddddddddddddd_rirqs2 (note the 461 in the middle), when the original column names are d<510>d_rirqs1 and d<510>d_rirqs2. Not convinced this is actually great behavior.

also, fix the catalogparser tests to have more stringent asserts. Previously they were actually just completely useless, because we were mocking the wrong method. The asserts should now catch this problem, since we assert on the actual table/column names.

submitted https://github.com/airbytehq/airbyte-internal-issues/issues/6989 to make collision logic better in general.

Copy link

vercel bot commented Apr 3, 2024

The latest updates on your projects. Learn more about Vercel for Git ↗︎

Name Status Preview Comments Updated (UTC)
airbyte-docs ✅ Ready (Inspect) Visit Preview 💬 Add feedback Apr 10, 2024 3:59pm

@octavia-squidington-iii octavia-squidington-iii added area/connectors Connector related issues CDK Connector Development Kit connectors/destination/postgres labels Apr 3, 2024
@edgao edgao marked this pull request as ready for review April 3, 2024 18:46
@edgao edgao requested a review from a team as a code owner April 3, 2024 18:46
@octavia-squidington-iv octavia-squidington-iv requested review from a team April 3, 2024 18:52
Copy link
Contributor

@jbfbell jbfbell left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

General thumbs up, just the toStreamConfig method is getting a little unruly and we should break it up into sub methods

@@ -161,6 +161,45 @@ constructor(
var i = 1
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[Re: line 104]

wise suggestion we should probably follow

See this comment inline on Graphite.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

done in c610197?w=1. also cleaned up some of the kotlin autoconvert cruft.

// but this interface + our superclass are weirdly complicated, so plausibly something is missing
@Override
public String getIdentifier(final String name) {
return truncate(super.getIdentifier(name));
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I already made another mess here unify that with call to getIdentifier ?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

done in 8291ba5.

@gisripa gisripa force-pushed the gireesh/03-28-postgres-cdkk-fixes branch 8 times, most recently from 3ca93b5 to 5bbae1f Compare April 5, 2024 17:52
@edgao edgao force-pushed the edgao/postgres_column_name_truncation branch from c1204a2 to f29d835 Compare April 5, 2024 18:11
@gisripa gisripa force-pushed the gireesh/03-28-postgres-cdkk-fixes branch 2 times, most recently from 7ddd15b to f2bf8e0 Compare April 5, 2024 19:26
Base automatically changed from gireesh/03-28-postgres-cdkk-fixes to master April 5, 2024 20:02
@edgao edgao force-pushed the edgao/postgres_column_name_truncation branch from 8291ba5 to 74dde57 Compare April 5, 2024 20:40
@edgao edgao requested a review from a team as a code owner April 10, 2024 15:22
@edgao
Copy link
Contributor Author

edgao commented Apr 10, 2024

/publish-java-cdk

🕑 https://github.com/airbytehq/airbyte/actions/runs/8633799390
✅ Successfully published Java CDK version=0.29.10!

@octavia-squidington-iii octavia-squidington-iii added the area/documentation Improvements or additions to documentation label Apr 10, 2024
@edgao edgao enabled auto-merge (squash) April 10, 2024 15:52
@edgao edgao merged commit 7ecb1d3 into master Apr 10, 2024
31 checks passed
@edgao edgao deleted the edgao/postgres_column_name_truncation branch April 10, 2024 16:12
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
area/connectors Connector related issues area/documentation Improvements or additions to documentation CDK Connector Development Kit connectors/destination/postgres connectors/destination/postgres-strict-encrypt
Projects
None yet
Development

Successfully merging this pull request may close these issues.

4 participants