Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Destination postgres (dv2): add indexes to raw table #34236

Merged
merged 121 commits into from Jan 19, 2024
Merged

Destination postgres (dv2): add indexes to raw table #34236

merged 121 commits into from Jan 19, 2024

Conversation

edgao
Copy link
Contributor

@edgao edgao commented Jan 12, 2024

copied from #34225, but without screwy merge. Still targeting the #34177 branch.

There was still one open discussion:

@evantahler

... should JdbcDestinationHandler detect index changes? We got that working in snowflake. It seems like a feature we'll want in the future maybe?

@edgao

I think eventually yes, but not literally today. If any beta users want us to change things indexes, I feel ok asking them to manually drop all their tables? Having autodetections feels like a "after we release to GA and we want to change our indexes" feature
it's probably not a huge lift (something like fetch jdbc metadata -> dump into the TableDefinition struct -> diff appropriately) and is a pure bolt-on addition in the future, so I feel ok deferring it for now?

@edgao
make methods public
7ec0e4a
@edgao
allow overriding timestamptz type
b0f6035
@edgao
better?
e65e315
@edgao
move jdbc base test into db-destinations testFixtures
e9c6e34
@edgao
override annotation
6ee0add
@edgao
fixes
aade64a
@edgao
typing_deduping core code uses Sql struct
d8fde89
@edgao
update bigquery
d7949a0
@edgao
update snowflake
30f8a4d
@edgao
update redshift
cd6b03b
@edgao
autoappend semicolon
803c758
@edgao
move insert statement to base class
af0d77a
@edgao
Merge branch 'dv2_cdk_changes' into edgao/dv2_sql_refactor
f9a4193
@edgao
propagate safe_cast option to subclass
3f31078
@edgao
Merge branch 'dv2_cdk_changes' into edgao/dv2_sql_refactor
d952aab
@edgao
fix redshift sqlgenerator
d25cb85
@edgao
Merge branch 'dv2_cdk_changes' into edgao/dv2_sql_refactor
ef2ddd4
@edgao
make methods public
b21a539
@edgao
allow overriding timestamptz type
8d55bac
@edgao
better?
a37d096
@edgao
move jdbc base test into db-destinations testFixtures
0f7cf74
@edgao
override annotation
2583a70
@edgao
fixes
2e6e931
@edgao
move insert statement to base class
b4ffdba
@edgao
propagate safe_cast option to subclass
4d045ca
@edgao
fix redshift sqlgenerator
3c53c37
@edgao
Merge branch 'dv2_cdk_changes' into edgao/dv2_sql_refactor
54350e9
@edgao
fix redshift test
b0a667d
@edgao
Merge branch 'dv2_cdk_changes' into edgao/dv2_sql_refactor
df833f9
@edgao
use local cdk
9463896
@gisripa
redshift cdk update
4be9a32 
Signed-off-by: Gireesh Sreepathi <gisripa@gmail.com>
@gisripa
revert bigquery and snowflake cdk breaking changes
ddd719e 
Signed-off-by: Gireesh Sreepathi <gisripa@gmail.com>
@octavia-squidington-iii octavia-squidington-iii removed the area/documentation Improvements or additions to documentation label Jan 18, 2024
@gisripa
format
1daf383 
Signed-off-by: Gireesh Sreepathi <gisripa@gmail.com>
@gisripa
bump connector versions and cdk version
ac45f4f 
Signed-off-by: Gireesh Sreepathi <gisripa@gmail.com>
@octavia-squidington-iii octavia-squidington-iii added the area/documentation Improvements or additions to documentation label Jan 18, 2024
@gisripa gisripa changed the title Postgres dv2: add indexes Destination postgres (dv2): add indexes to raw table Jan 18, 2024
@gisripa
bump redshift connector
ae1ec15 
Signed-off-by: Gireesh Sreepathi <gisripa@gmail.com>
@gisripa
Copy link
Contributor

gisripa commented Jan 18, 2024 •
edited by github-actions bot

/publish-java-cdk

🕑 https://github.com/airbytehq/airbyte/actions/runs/7577063045
✅ Successfully published Java CDK version=0.13.1!

@gisripa
fix readme for cdk
2a11dcc 
Signed-off-by: Gireesh Sreepathi <gisripa@gmail.com>
@gisripa
turnoff local cdk for postgres
9797031 
Signed-off-by: Gireesh Sreepathi <gisripa@gmail.com>
@gisripa
Merge branch 'master' into edgao/dv2/postgres/indexes_take2
5ae13c0
edgao
edgao commented Jan 18, 2024
View reviewed changes
airbyte-integrations/connectors/destination-postgres-strict-encrypt/metadata.yaml
@@ -19,8 +13,14 @@ data:
normalizationIntegrationType: postgres
normalizationRepository: airbyte/normalization
normalizationTag: 0.4.1
registries:
cloud:
dockerImageTag: 0.4.0
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think we should remove this, even if it's ignored

airbyte-integrations/connectors/destination-redshift/build.gradle
@@ -4,7 +4,7 @@ plugins {
}

airbyteJavaConnector {
cdkVersionRequired = '0.12.1'
cdkVersionRequired = '0.13.0'
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

might as well bump to 0.13.1?

@gisripa gisripa merged commit d610ad1 into master Jan 19, 2024
28 checks passed
@gisripa gisripa deleted the edgao/dv2/postgres/indexes_take2 branch January 19, 2024 00:21
rodireich pushed a commit that referenced this pull request Jan 19, 2024
@edgao @gisripa @rodireich
Destination postgres (dv2): add indexes to raw table (#34236)
a1e2dae 
Signed-off-by: Gireesh Sreepathi <gisripa@gmail.com>
Co-authored-by: Gireesh Sreepathi <gisripa@gmail.com>
jatinyadav-cc pushed a commit to ollionorg/datapipes-airbyte that referenced this pull request Feb 26, 2024
@edgao @gisripa @jatinyadav-cc
Destination postgres (dv2): add indexes to raw table (airbytehq#34236)
7cec13d 
Signed-off-by: Gireesh Sreepathi <gisripa@gmail.com>
Co-authored-by: Gireesh Sreepathi <gisripa@gmail.com>
jatinyadav-cc pushed a commit to ollionorg/datapipes-airbyte that referenced this pull request Feb 26, 2024
@edgao @gisripa @jatinyadav-cc
Destination postgres (dv2): add indexes to raw table (airbytehq#34236)
4f7d9e7 
Signed-off-by: Gireesh Sreepathi <gisripa@gmail.com>
Co-authored-by: Gireesh Sreepathi <gisripa@gmail.com>
jatinyadav-cc pushed a commit to ollionorg/datapipes-airbyte that referenced this pull request Feb 26, 2024
@edgao @gisripa @jatinyadav-cc
Destination postgres (dv2): add indexes to raw table (airbytehq#34236)
59d4198 
Signed-off-by: Gireesh Sreepathi <gisripa@gmail.com>
Co-authored-by: Gireesh Sreepathi <gisripa@gmail.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@evantahler evantahler evantahler approved these changes

Assignees
No one assigned
Labels
area/connectors Connector related issues area/documentation Improvements or additions to documentation CDK Connector Development Kit checklist-action-run connectors/destination/postgres connectors/destination/postgres-strict-encrypt connectors/destination/redshift
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.

None yet
4 participants
@edgao @gisripa @evantahler @octavia-squidington-iii