Improve check connection performance for all JDBC-related DBs #12820

alexandr-shegeda · 2022-05-12T19:31:35Z

What

Describe what the change is solving
It helps to add screenshots if it affects the frontend.

How

Optimized check operation by removing the SELECT SCHEMA part.

🚨 User Impact 🚨

No user impact. Removed redundant schema retrieve operation in the scope of the check method.

Pre-merge Checklist

Issue acceptance criteria met
PR name follows PR naming conventions

Tests

Unit

Put your unit tests output here.

Integration

Put your integration tests output here.

Acceptance

Put your acceptance tests output here.

# Conflicts: # docs/integrations/destinations/redshift.md

alexandr-shegeda · 2022-05-17T18:02:12Z

/publish connector=connectors/destination-snowflake

🕑 connectors/destination-snowflake https://github.com/airbytehq/airbyte/actions/runs/2340495556
❌ Failed to publish connectors/destination-snowflake
❌ Couldn't auto-bump version for connectors/destination-snowflake

alexandr-shegeda · 2022-05-17T19:31:29Z

/test connector=connectors/destination-snowflake

🕑 connectors/destination-snowflake https://github.com/airbytehq/airbyte/actions/runs/2340955612
✅ connectors/destination-snowflake https://github.com/airbytehq/airbyte/actions/runs/2340955612
Python tests coverage:

Name                                                              Stmts   Miss  Cover
-------------------------------------------------------------------------------------
normalization/transform_config/__init__.py                            2      0   100%
normalization/transform_catalog/reserved_keywords.py                 13      0   100%
normalization/transform_catalog/__init__.py                           2      0   100%
normalization/destination_type.py                                    13      0   100%
normalization/__init__.py                                             4      0   100%
normalization/transform_catalog/destination_name_transformer.py     155      8    95%
normalization/transform_config/transform.py                         159     31    81%
normalization/transform_catalog/table_name_registry.py              174     34    80%
normalization/transform_catalog/utils.py                             38      9    76%
normalization/transform_catalog/dbt_macro.py                         22      7    68%
normalization/transform_catalog/catalog_processor.py                147     80    46%
normalization/transform_catalog/transform.py                         61     38    38%
normalization/transform_catalog/stream_processor.py                 543    352    35%
-------------------------------------------------------------------------------------
TOTAL                                                              1333    559    58%

alexandr-shegeda · 2022-05-17T19:32:16Z

/test connector=connectors/destination-azure-blob-storage

🕑 connectors/destination-azure-blob-storage https://github.com/airbytehq/airbyte/actions/runs/2340960074
✅ connectors/destination-azure-blob-storage https://github.com/airbytehq/airbyte/actions/runs/2340960074
No Python unittests run

alexandr-shegeda · 2022-05-17T19:32:23Z

/test connector=connectors/destination-clickhouse

🕑 connectors/destination-clickhouse https://github.com/airbytehq/airbyte/actions/runs/2340960710
✅ connectors/destination-clickhouse https://github.com/airbytehq/airbyte/actions/runs/2340960710
No Python unittests run

alexandr-shegeda · 2022-05-17T19:32:32Z

/test connector=connectors/destination-clickhouse-strict-encrypt

🕑 connectors/destination-clickhouse-strict-encrypt https://github.com/airbytehq/airbyte/actions/runs/2340961044
✅ connectors/destination-clickhouse-strict-encrypt https://github.com/airbytehq/airbyte/actions/runs/2340961044
No Python unittests run

alexandr-shegeda · 2022-05-17T19:32:37Z

/test connector=connectors/destination-dynamodb

🕑 connectors/destination-dynamodb https://github.com/airbytehq/airbyte/actions/runs/2340961897
✅ connectors/destination-dynamodb https://github.com/airbytehq/airbyte/actions/runs/2340961897
No Python unittests run

alexandr-shegeda · 2022-05-17T19:32:50Z

/test connector=connectors/destination-gcs

🕑 connectors/destination-gcs https://github.com/airbytehq/airbyte/actions/runs/2340963148
✅ connectors/destination-gcs https://github.com/airbytehq/airbyte/actions/runs/2340963148
No Python unittests run

alexandr-shegeda · 2022-05-17T19:32:56Z

/test connector=connectors/destination-mariadb-columnstore

🕑 connectors/destination-mariadb-columnstore https://github.com/airbytehq/airbyte/actions/runs/2340963518

alexandr-shegeda · 2022-05-17T19:33:01Z

/test connector=connectors/destination-mssql

🕑 connectors/destination-mssql https://github.com/airbytehq/airbyte/actions/runs/2340964426
❌ connectors/destination-mssql https://github.com/airbytehq/airbyte/actions/runs/2340964426
🐛

alexandr-shegeda · 2022-05-17T19:33:08Z

/test connector=connectors/destination-mssql-strict-encrypt

🕑 connectors/destination-mssql-strict-encrypt https://github.com/airbytehq/airbyte/actions/runs/2340964959
❌ connectors/destination-mssql-strict-encrypt https://github.com/airbytehq/airbyte/actions/runs/2340964959
🐛

alexandr-shegeda · 2022-05-17T19:33:14Z

/test connector=connectors/destination-mysql

🕑 connectors/destination-mysql https://github.com/airbytehq/airbyte/actions/runs/2340965102
❌ connectors/destination-mysql https://github.com/airbytehq/airbyte/actions/runs/2340965102
🐛

alexandr-shegeda · 2022-05-17T19:33:21Z

/test connector=connectors/destination-mysql-strict-encrypt

🕑 connectors/destination-mysql-strict-encrypt https://github.com/airbytehq/airbyte/actions/runs/2340965746
❌ connectors/destination-mysql-strict-encrypt https://github.com/airbytehq/airbyte/actions/runs/2340965746
🐛

alexandr-shegeda · 2022-05-17T19:33:28Z

/test connector=connectors/destination-oracle

🕑 connectors/destination-oracle https://github.com/airbytehq/airbyte/actions/runs/2340965974
✅ connectors/destination-oracle https://github.com/airbytehq/airbyte/actions/runs/2340965974
Python tests coverage:

Name                                                              Stmts   Miss  Cover
-------------------------------------------------------------------------------------
normalization/transform_config/__init__.py                            2      0   100%
normalization/transform_catalog/reserved_keywords.py                 13      0   100%
normalization/transform_catalog/__init__.py                           2      0   100%
normalization/destination_type.py                                    13      0   100%
normalization/__init__.py                                             4      0   100%
normalization/transform_catalog/destination_name_transformer.py     155      8    95%
normalization/transform_config/transform.py                         159     31    81%
normalization/transform_catalog/table_name_registry.py              174     34    80%
normalization/transform_catalog/utils.py                             38      9    76%
normalization/transform_catalog/dbt_macro.py                         22      7    68%
normalization/transform_catalog/catalog_processor.py                147     80    46%
normalization/transform_catalog/transform.py                         61     38    38%
normalization/transform_catalog/stream_processor.py                 543    352    35%
-------------------------------------------------------------------------------------
TOTAL                                                              1333    559    58%

alexandr-shegeda · 2022-05-17T19:33:34Z

/test connector=connectors/destination-oracle-strict-encrypt

🕑 connectors/destination-oracle-strict-encrypt https://github.com/airbytehq/airbyte/actions/runs/2340966922
❌ connectors/destination-oracle-strict-encrypt https://github.com/airbytehq/airbyte/actions/runs/2340966922
🐛

alexandr-shegeda · 2022-05-19T07:12:06Z

/publish connector=connectors/destination-oracle-strict-encrypt

🕑 connectors/destination-oracle-strict-encrypt https://github.com/airbytehq/airbyte/actions/runs/2350318285
❌ Failed to publish connectors/destination-oracle-strict-encrypt
❌ Couldn't auto-bump version for connectors/destination-oracle-strict-encrypt

alexandr-shegeda · 2022-05-19T07:12:17Z

/publish connector=connectors/destination-oracle

🕑 connectors/destination-oracle https://github.com/airbytehq/airbyte/actions/runs/2350318975
❌ Failed to publish connectors/destination-oracle
❌ Couldn't auto-bump version for connectors/destination-oracle

alexandr-shegeda · 2022-05-19T07:12:41Z

/test connector=connectors/destination-mssql-strict-encrypt

🕑 connectors/destination-mssql-strict-encrypt https://github.com/airbytehq/airbyte/actions/runs/2350320963
❌ connectors/destination-mssql-strict-encrypt https://github.com/airbytehq/airbyte/actions/runs/2350320963
🐛 https://gradle.com/s/frzcz3oxj4qis

alexandr-shegeda · 2022-05-19T07:12:55Z

/test connector=connectors/destination-mssql

🕑 connectors/destination-mssql https://github.com/airbytehq/airbyte/actions/runs/2350321714
✅ connectors/destination-mssql https://github.com/airbytehq/airbyte/actions/runs/2350321714
Python tests coverage:

Name                                                              Stmts   Miss  Cover
-------------------------------------------------------------------------------------
normalization/transform_config/__init__.py                            2      0   100%
normalization/transform_catalog/reserved_keywords.py                 13      0   100%
normalization/transform_catalog/__init__.py                           2      0   100%
normalization/destination_type.py                                    13      0   100%
normalization/__init__.py                                             4      0   100%
normalization/transform_catalog/destination_name_transformer.py     155      8    95%
normalization/transform_config/transform.py                         159     31    81%
normalization/transform_catalog/table_name_registry.py              174     34    80%
normalization/transform_catalog/utils.py                             38      9    76%
normalization/transform_catalog/dbt_macro.py                         22      7    68%
normalization/transform_catalog/catalog_processor.py                147     80    46%
normalization/transform_catalog/transform.py                         61     38    38%
normalization/transform_catalog/stream_processor.py                 543    352    35%
-------------------------------------------------------------------------------------
TOTAL                                                              1333    559    58%

alexandr-shegeda · 2022-05-19T15:41:30Z

/publish connector=connectors/destination-mssql

🕑 connectors/destination-mssql https://github.com/airbytehq/airbyte/actions/runs/2353082006
❌ Failed to publish connectors/destination-mssql
❌ Couldn't auto-bump version for connectors/destination-mssql

alexandr-shegeda · 2022-05-19T15:41:50Z

/publish connector=connectors/destination-mssql-strict-encrypt

🕑 connectors/destination-mssql-strict-encrypt https://github.com/airbytehq/airbyte/actions/runs/2353084384
❌ Failed to publish connectors/destination-mssql-strict-encrypt
❌ Couldn't auto-bump version for connectors/destination-mssql-strict-encrypt

alexandr-shegeda · 2022-05-19T16:35:48Z

/publish connector=connectors/destination-mssql auto-bump-version=false

🕑 connectors/destination-mssql https://github.com/airbytehq/airbyte/actions/runs/2353349802
❌ Failed to publish connectors/destination-mssql

alexandr-shegeda · 2022-05-19T16:48:10Z

/publish connector=connectors/destination-redshift auto-bump-version=false

🕑 connectors/destination-redshift https://github.com/airbytehq/airbyte/actions/runs/2353399852
🚀 Successfully published connectors/destination-redshift
✅ connectors/destination-redshift https://github.com/airbytehq/airbyte/actions/runs/2353399852

marcosmarxm · 2022-05-20T14:33:23Z

@alexandr-shegeda can you bump the redshift destination again? I didn't find information about the 35 version and excluded from Dockerhub and publish #12940 with it.

marcosmarxm · 2022-05-20T14:34:11Z

Also would be better change the title of your PR. It's doing way more than editing Snowflake destination.

# Conflicts: # docs/integrations/destinations/redshift.md

alexandr-shegeda · 2022-05-23T11:59:23Z

/publish connector=connectors/destination-redshift

🕑 connectors/destination-redshift https://github.com/airbytehq/airbyte/actions/runs/2371102365
🚀 Successfully published connectors/destination-redshift
❌ Couldn't auto-bump version for connectors/destination-redshift

…ehq#12820) * removed redundant read permission check for destination * bump docker image versions, update definitions and changelog * bump docker image versions, update definitions and changelog * update changelog * fix for MSSQL and Oracle tests * fix for MSSQL strict encrypt * fixed conflicts * revert versions that were not published

removed redundant read permission check for destination

98aeda7

alexandr-shegeda requested a review from grishick May 12, 2022 19:31

github-actions bot added the area/connectors Connector related issues label May 12, 2022

alexandr-shegeda mentioned this pull request May 12, 2022

10376 Destination Snowflake: improve check connection performance #12196

Closed

2 tasks

alexandr-shegeda linked an issue May 12, 2022 that may be closed by this pull request

Destination Snowflake: improve check connection performance #10376

Closed

1 task

VitaliiMaltsev mentioned this pull request May 13, 2022

Destination Redshift: improve check performance #11580

Closed

grishick approved these changes May 16, 2022

View reviewed changes

bump docker image versions, update definitions and changelog

dff32c6

github-actions bot added the area/documentation Improvements or additions to documentation label May 17, 2022

alexandr-shegeda added 2 commits May 17, 2022 20:55

Merge branch 'master' into osheg/10376-improved-jdbc-check-perf

5871627

# Conflicts: # docs/integrations/destinations/redshift.md

bump docker image versions, update definitions and changelog

95518cc

alexandr-shegeda temporarily deployed to more-secrets May 17, 2022 17:59 Inactive

update changelog

dbfe2ad

alexandr-shegeda temporarily deployed to more-secrets May 17, 2022 18:02 Inactive

alexandr-shegeda temporarily deployed to more-secrets May 18, 2022 18:49 Inactive

fix for MSSQL strict encrypt

d3328d6

alexandr-shegeda requested a deployment to more-secrets May 19, 2022 15:30 Abandoned

alexandr-shegeda temporarily deployed to more-secrets May 19, 2022 15:30 Inactive

alexandertsukanov mentioned this pull request May 20, 2022

Destination Redshift: Fixed maximum record size for SUPER type #12940

Merged

14 tasks

alexandr-shegeda changed the title ~~10376 Destination Snowflake: improve check connection performance~~ Improve check connection performance for all JDBC-related DBs May 20, 2022

alexandr-shegeda added 2 commits May 23, 2022 14:51

Merge branch 'master' into osheg/10376-improved-jdbc-check-perf

e275428

# Conflicts: # docs/integrations/destinations/redshift.md

fixed conflicts

b904854

alexandr-shegeda temporarily deployed to more-secrets May 23, 2022 12:01 Inactive

revert versions that were not published

591728f

alexandr-shegeda merged commit 44a43f0 into master May 24, 2022

alexandr-shegeda deleted the osheg/10376-improved-jdbc-check-perf branch May 24, 2022 13:06

alexandr-shegeda temporarily deployed to more-secrets May 24, 2022 13:06 Inactive

octavia-squidington-iii mentioned this pull request May 25, 2022

Bump Airbyte version from 0.39.1-alpha to 0.39.2-alpha #13188

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Improve check connection performance for all JDBC-related DBs #12820

Improve check connection performance for all JDBC-related DBs #12820

alexandr-shegeda commented May 12, 2022

alexandr-shegeda commented May 17, 2022 •

edited by github-actions bot

Loading

alexandr-shegeda commented May 17, 2022 •

edited by github-actions bot

Loading

alexandr-shegeda commented May 17, 2022 •

edited by github-actions bot

Loading

alexandr-shegeda commented May 17, 2022 •

edited by github-actions bot

Loading

alexandr-shegeda commented May 17, 2022 •

edited by github-actions bot

Loading

alexandr-shegeda commented May 17, 2022 •

edited by github-actions bot

Loading

alexandr-shegeda commented May 17, 2022 •

edited by github-actions bot

Loading

alexandr-shegeda commented May 17, 2022 •

edited by github-actions bot

Loading

alexandr-shegeda commented May 17, 2022 •

edited by github-actions bot

Loading

alexandr-shegeda commented May 17, 2022 •

edited by github-actions bot

Loading

alexandr-shegeda commented May 17, 2022 •

edited by github-actions bot

Loading

alexandr-shegeda commented May 17, 2022 •

edited by github-actions bot

Loading

alexandr-shegeda commented May 17, 2022 •

edited by github-actions bot

Loading

alexandr-shegeda commented May 17, 2022 •

edited by github-actions bot

Loading

alexandr-shegeda commented May 19, 2022 •

edited by github-actions bot

Loading

alexandr-shegeda commented May 19, 2022 •

edited by github-actions bot

Loading

alexandr-shegeda commented May 19, 2022 •

edited by github-actions bot

Loading

alexandr-shegeda commented May 19, 2022 •

edited by github-actions bot

Loading

alexandr-shegeda commented May 19, 2022 •

edited by github-actions bot

Loading

alexandr-shegeda commented May 19, 2022 •

edited by github-actions bot

Loading

alexandr-shegeda commented May 19, 2022 •

edited by github-actions bot

Loading

alexandr-shegeda commented May 19, 2022 •

edited by github-actions bot

Loading

marcosmarxm commented May 20, 2022

marcosmarxm commented May 20, 2022

alexandr-shegeda commented May 23, 2022 •

edited by github-actions bot

Loading

Improve check connection performance for all JDBC-related DBs #12820

Improve check connection performance for all JDBC-related DBs #12820

Conversation

alexandr-shegeda commented May 12, 2022

What

How

Recommended reading order

🚨 User Impact 🚨

Pre-merge Checklist

Tests

alexandr-shegeda commented May 17, 2022 • edited by github-actions bot Loading

alexandr-shegeda commented May 17, 2022 • edited by github-actions bot Loading

alexandr-shegeda commented May 17, 2022 • edited by github-actions bot Loading

alexandr-shegeda commented May 17, 2022 • edited by github-actions bot Loading

alexandr-shegeda commented May 17, 2022 • edited by github-actions bot Loading

alexandr-shegeda commented May 17, 2022 • edited by github-actions bot Loading

alexandr-shegeda commented May 17, 2022 • edited by github-actions bot Loading

alexandr-shegeda commented May 17, 2022 • edited by github-actions bot Loading

alexandr-shegeda commented May 17, 2022 • edited by github-actions bot Loading

alexandr-shegeda commented May 17, 2022 • edited by github-actions bot Loading

alexandr-shegeda commented May 17, 2022 • edited by github-actions bot Loading

alexandr-shegeda commented May 17, 2022 • edited by github-actions bot Loading

alexandr-shegeda commented May 17, 2022 • edited by github-actions bot Loading

alexandr-shegeda commented May 17, 2022 • edited by github-actions bot Loading

alexandr-shegeda commented May 19, 2022 • edited by github-actions bot Loading

alexandr-shegeda commented May 19, 2022 • edited by github-actions bot Loading

alexandr-shegeda commented May 19, 2022 • edited by github-actions bot Loading

alexandr-shegeda commented May 19, 2022 • edited by github-actions bot Loading

alexandr-shegeda commented May 19, 2022 • edited by github-actions bot Loading

alexandr-shegeda commented May 19, 2022 • edited by github-actions bot Loading

alexandr-shegeda commented May 19, 2022 • edited by github-actions bot Loading

alexandr-shegeda commented May 19, 2022 • edited by github-actions bot Loading

marcosmarxm commented May 20, 2022

marcosmarxm commented May 20, 2022

alexandr-shegeda commented May 23, 2022 • edited by github-actions bot Loading

alexandr-shegeda commented May 17, 2022 •

edited by github-actions bot

Loading

alexandr-shegeda commented May 17, 2022 •

edited by github-actions bot

Loading

alexandr-shegeda commented May 17, 2022 •

edited by github-actions bot

Loading

alexandr-shegeda commented May 17, 2022 •

edited by github-actions bot

Loading

alexandr-shegeda commented May 17, 2022 •

edited by github-actions bot

Loading

alexandr-shegeda commented May 17, 2022 •

edited by github-actions bot

Loading

alexandr-shegeda commented May 17, 2022 •

edited by github-actions bot

Loading

alexandr-shegeda commented May 17, 2022 •

edited by github-actions bot

Loading

alexandr-shegeda commented May 17, 2022 •

edited by github-actions bot

Loading

alexandr-shegeda commented May 17, 2022 •

edited by github-actions bot

Loading

alexandr-shegeda commented May 17, 2022 •

edited by github-actions bot

Loading

alexandr-shegeda commented May 17, 2022 •

edited by github-actions bot

Loading

alexandr-shegeda commented May 17, 2022 •

edited by github-actions bot

Loading

alexandr-shegeda commented May 17, 2022 •

edited by github-actions bot

Loading

alexandr-shegeda commented May 19, 2022 •

edited by github-actions bot

Loading

alexandr-shegeda commented May 19, 2022 •

edited by github-actions bot

Loading

alexandr-shegeda commented May 19, 2022 •

edited by github-actions bot

Loading

alexandr-shegeda commented May 19, 2022 •

edited by github-actions bot

Loading

alexandr-shegeda commented May 19, 2022 •

edited by github-actions bot

Loading

alexandr-shegeda commented May 19, 2022 •

edited by github-actions bot

Loading

alexandr-shegeda commented May 19, 2022 •

edited by github-actions bot

Loading

alexandr-shegeda commented May 19, 2022 •

edited by github-actions bot

Loading

alexandr-shegeda commented May 23, 2022 •

edited by github-actions bot

Loading