🎉Source-redshift: added an optional field for schema\s selection #9721

etsybaev · 2022-01-23T13:29:59Z

What

Currently we are not able to select a particular schema for selection. That causes a mess and inconvenience if you have multiple schemas but want to migrate only some of them.

How

Added a new optional field on UI for schema selection similarly to Postgres and Oracle source connectors.

No schema selected explicitly- will process all of existing in source DB
One or more explicitly selected on UI - will show and process only selected.

!!!!!!!TODO ONCE PR APPROVED: !!!!!!!!!!

update docs
finish all PR's checklist

Tested locally:
All schemas are shown If no schemas selected explicitly

Tested with particular schema selection

🚨 User Impact 🚨

The newly added field is an optional one. So breaks in backward compatibility are expected.

Pre-merge Checklist

Expand the relevant checklist and delete the others.

New Connector

Community member or Airbyter

Airbyter

If this is a community PR, the Airbyte engineer reviewing this PR is responsible for the below items.

Create a non-forked branch based on this PR and test the below items on it
Build is successful
Credentials added to Github CI. Instructions.
/test connector=connectors/<name> command is passing.
New Connector version released on Dockerhub by running the /publish command described here
After the connector is published, connector added to connector index as described here
Seed specs have been re-generated by building the platform and committing the changes to the seed spec files, as described here

Updating a connector

Community member or Airbyter

Grant edit access to maintainers (instructions)
Secrets in the connector's spec are annotated with airbyte_secret
Unit & integration tests added and passing. Community members, please provide proof of success locally e.g: screenshot or copy-paste unit, integration, and acceptance test output. To run acceptance tests for a Python connector, follow instructions in the README. For java connectors run ./gradlew :airbyte-integrations:connectors:<name>:integrationTest.
Code reviews completed
Documentation updated
- Connector's README.md
- Connector's bootstrap.md. See description and examples
- Changelog updated in docs/integrations/<source or destination>/<name>.md including changelog. See changelog example
PR name follows PR naming conventions

Airbyter

If this is a community PR, the Airbyte engineer reviewing this PR is responsible for the below items.

Create a non-forked branch based on this PR and test the below items on it
Build is successful
Credentials added to Github CI. Instructions.
/test connector=connectors/<name> command is passing.
New Connector version released on Dockerhub by running the /publish command described here
After the new connector version is published, connector version bumped in the seed directory as described here
Seed specs have been re-generated by building the platform and committing the changes to the seed spec files, as described here

etsybaev · 2022-01-23T16:21:48Z

/test connector=connectors/source-redshift

🕑 connectors/source-redshift https://github.com/airbytehq/airbyte/actions/runs/1736398244
✅ connectors/source-redshift https://github.com/airbytehq/airbyte/actions/runs/1736398244
No Python unittests run

...rs/source-redshift/src/main/java/io/airbyte/integrations/source/redshift/RedshiftSource.java

airbyte-integrations/connectors/source-redshift/src/main/resources/spec.json

.../airbyte/integrations/io/airbyte/integration_tests/sources/RedshiftSourceAcceptanceTest.java

…election # Conflicts: # airbyte-integrations/connectors/source-redshift/src/main/resources/spec.json

...rs/source-redshift/src/main/java/io/airbyte/integrations/source/redshift/RedshiftSource.java

airbyte-integrations/connectors/source-redshift/src/main/resources/spec.json

tuliren · 2022-01-25T21:26:41Z

.../airbyte/integrations/io/airbyte/integration_tests/sources/RedshiftSourceAcceptanceTest.java

@@ -52,6 +53,9 @@ protected void setupEnvironment(final TestDestinationEnv environment) throws Exc
        RedshiftSource.DRIVER_CLASS);

    schemaName = Strings.addRandomSuffix("integration_test", "_", 5).toLowerCase();
+
+    config = config.set("schemas", Jsons.jsonNode(List.of(schemaName)));
+
    final String createSchemaQuery = String.format("CREATE SCHEMA %s", schemaName);


Can you also create a second schema (like unselected_schema) with some tables, and verify in the acceptance test that the catalog will not include anything in the second schema?

Otherwise, I think the schema selection is not tested any where currently.

Moved data creation to a separate method and added one more schema creation to have some noise. Thanks

I just realized that the SourceAcceptanceTest does not actually verify the catalog. I created an issue to track it:
#9818

etsybaev · 2022-01-26T15:08:19Z

/test connector=connectors/source-redshift

🕑 connectors/source-redshift https://github.com/airbytehq/airbyte/actions/runs/1751477075
✅ connectors/source-redshift https://github.com/airbytehq/airbyte/actions/runs/1751477075
No Python unittests run

…election

etsybaev · 2022-01-26T19:42:20Z

/publish connector=connectors/source-redshift

🕑 connectors/source-redshift https://github.com/airbytehq/airbyte/actions/runs/1752699724
✅ connectors/source-redshift https://github.com/airbytehq/airbyte/actions/runs/1752699724

[9525] source-redshift: added schema selection

73890b0

github-actions bot added the area/connectors Connector related issues label Jan 23, 2022

etsybaev linked an issue Jan 23, 2022 that may be closed by this pull request

Support schema selection for Redshift source connector #9525

Closed

octavia-squidington-iii temporarily deployed to more-secrets January 23, 2022 16:24 Inactive

etsybaev changed the title ~~[DRAFT_PR][9525] source-redshift: added schema selection~~ [DRAFT_PR]🎉Source-redshift: added an optional fields for schema\s selection Jan 24, 2022

etsybaev requested a review from mkhokh-33 January 24, 2022 09:10

mkhokh-33 reviewed Jan 24, 2022

View reviewed changes

...rs/source-redshift/src/main/java/io/airbyte/integrations/source/redshift/RedshiftSource.java Outdated Show resolved Hide resolved

mkhokh-33 approved these changes Jan 24, 2022

View reviewed changes

etsybaev changed the title ~~[DRAFT_PR]🎉Source-redshift: added an optional fields for schema\s selection~~ 🎉Source-redshift: added an optional field for schema\s selection Jan 24, 2022

minor refactor

28c99ab

etsybaev requested a review from subodh1810 January 24, 2022 11:54

etsybaev marked this pull request as ready for review January 24, 2022 11:54

etsybaev temporarily deployed to more-secrets January 24, 2022 11:54 Inactive

mkhokh-33 approved these changes Jan 24, 2022

View reviewed changes

etsybaev requested a review from edgao January 25, 2022 18:20

edgao reviewed Jan 25, 2022

View reviewed changes

etsybaev added 2 commits January 25, 2022 22:44

minor refactor

c313e42

Merge branch 'master' into etsybaev/9525-source-redshift-add-schema-s…

ef7fa6e

…election # Conflicts: # airbyte-integrations/connectors/source-redshift/src/main/resources/spec.json

etsybaev temporarily deployed to more-secrets January 25, 2022 20:52 Inactive

etsybaev requested a review from edgao January 25, 2022 21:05

tuliren reviewed Jan 25, 2022

View reviewed changes

etsybaev added 2 commits January 26, 2022 15:36

added schema to ignore in test as filtering verification

6d5ec87

added schema to ignore in test as filtering verification

18bdb13

etsybaev temporarily deployed to more-secrets January 26, 2022 13:39 Inactive

typo fix

a537304

etsybaev requested a review from tuliren January 26, 2022 15:08

octavia-squidington-iii temporarily deployed to more-secrets January 26, 2022 15:10 Inactive

etsybaev temporarily deployed to more-secrets January 26, 2022 16:35 Inactive

Merge branch 'master' into etsybaev/9525-source-redshift-add-schema-s…

5cdedf6

…election

etsybaev temporarily deployed to more-secrets January 26, 2022 17:50 Inactive

tuliren approved these changes Jan 26, 2022

View reviewed changes

bumbed version

31f4c78

github-actions bot added the area/documentation Improvements or additions to documentation label Jan 26, 2022

etsybaev temporarily deployed to more-secrets January 26, 2022 19:29 Inactive

octavia-squidington-iii temporarily deployed to more-secrets January 26, 2022 19:44 Inactive

updated source schema

cde71b2

etsybaev temporarily deployed to more-secrets January 26, 2022 20:02 Inactive

etsybaev merged commit 5bce24c into master Jan 26, 2022

etsybaev deleted the etsybaev/9525-source-redshift-add-schema-selection branch January 26, 2022 20:09

tuliren mentioned this pull request Jan 28, 2022

Verify source redshift schema selection in tests #9862

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

🎉Source-redshift: added an optional field for schema\s selection #9721

🎉Source-redshift: added an optional field for schema\s selection #9721

etsybaev commented Jan 23, 2022 •

edited

etsybaev commented Jan 23, 2022 •

edited by github-actions bot

tuliren Jan 25, 2022

etsybaev Jan 26, 2022

tuliren Jan 26, 2022

etsybaev commented Jan 26, 2022 •

edited by github-actions bot

etsybaev commented Jan 26, 2022 •

edited by github-actions bot

🎉Source-redshift: added an optional field for schema\s selection #9721

🎉Source-redshift: added an optional field for schema\s selection #9721

Conversation

etsybaev commented Jan 23, 2022 • edited

What

How

Recommended reading order

🚨 User Impact 🚨

Pre-merge Checklist

Community member or Airbyter

Airbyter

Community member or Airbyter

Airbyter

etsybaev commented Jan 23, 2022 • edited by github-actions bot

tuliren Jan 25, 2022

Choose a reason for hiding this comment

etsybaev Jan 26, 2022

Choose a reason for hiding this comment

tuliren Jan 26, 2022

Choose a reason for hiding this comment

etsybaev commented Jan 26, 2022 • edited by github-actions bot

etsybaev commented Jan 26, 2022 • edited by github-actions bot

etsybaev commented Jan 23, 2022 •

edited

etsybaev commented Jan 23, 2022 •

edited by github-actions bot

etsybaev commented Jan 26, 2022 •

edited by github-actions bot

etsybaev commented Jan 26, 2022 •

edited by github-actions bot