Destination Redshift: Install normalization #26165

edgao · 2023-05-16T23:09:59Z

basically just copy #25782 but for redshift

I ran a sync locally (with platform-side hack to turn on in-connector normalization) and it worked:

github-actions · 2023-05-16T23:10:36Z

Before Merging a Connector Pull Request

Wow! What a great pull request you have here! 🎉

To merge this PR, ensure the following has been done/considered for each connector added or updated:

PR name follows PR naming conventions
Breaking changes are considered. If a Breaking Change is being introduced, ensure an Airbyte engineer has created a Breaking Change Plan and you've followed all steps in the Breaking Changes Checklist
Connector version has been incremented in the Dockerfile and metadata.yaml according to our Semantic Versioning for Connectors guidelines
Secrets in the connector's spec are annotated with airbyte_secret
All documentation files are up to date. (README.md, bootstrap.md, docs.md, etc...)
Changelog updated in docs/integrations/<source or destination>/<name>.md with an entry for the new version. See changelog example
You, or an Airbyter, have run /test successfully on this PR - or on a non-forked branch
You, or an Airbyter, have run /publish successfully on this PR - or on a non-forked branch
You've updated the connector's metadata.yaml file new!

If the checklist is complete, but the CI check is failing,

Check for hidden checklists in your PR description
Toggle the github label checklist-action-run on/off to re-run the checklist CI.

edgao · 2023-05-16T23:11:14Z

/test connector=connectors/destination-redshift

🕑 connectors/destination-redshift https://github.com/airbytehq/airbyte/actions/runs/4997523247
✅ connectors/destination-redshift https://github.com/airbytehq/airbyte/actions/runs/4997523247
Python tests coverage:

Name                                                              Stmts   Miss  Cover
-------------------------------------------------------------------------------------
normalization/transform_config/__init__.py                            2      0   100%
normalization/transform_catalog/reserved_keywords.py                 15      0   100%
normalization/transform_catalog/__init__.py                           2      0   100%
normalization/destination_type.py                                    18      0   100%
normalization/__init__.py                                             4      0   100%
normalization/transform_catalog/destination_name_transformer.py     171     10    94%
normalization/transform_catalog/table_name_registry.py              174     34    80%
normalization/transform_config/transform.py                         195     48    75%
normalization/transform_catalog/utils.py                             51     14    73%
normalization/transform_catalog/dbt_macro.py                         22      7    68%
normalization/transform_catalog/catalog_processor.py                147     80    46%
normalization/transform_catalog/transform.py                         65     39    40%
normalization/transform_catalog/stream_processor.py                 603    407    33%
-------------------------------------------------------------------------------------
TOTAL                                                              1469    639    57%

Build Passed

Test summary info:

All Passed

github-actions · 2023-05-16T23:14:11Z

Affected Connector Report

NOTE ⚠️ Changes in this PR affect the following connectors. Make sure to do the following as needed:

Run integration tests
Bump connector or module version
Add changelog
Publish the new version

✅ Sources (0)

Connector	Version	Changelog	Publish

See "Actionable Items" below for how to resolve warnings and errors.

❌ Destinations (1)

Connector	Version	Changelog	Publish
`destination-redshift`	`0.4.8`	✅	❌ (diff seed version)

See "Actionable Items" below for how to resolve warnings and errors.

👀 Other Modules (1)

base-normalization

Actionable Items

(click to expand)

Category	Status	Actionable Item
Version	❌ mismatch	The version of the connector is different from its normal variant. Please bump the version of the connector.
	⚠ doc not found	The connector does not seem to have a documentation file. This can be normal (e.g. basic connector like `source-jdbc` is not published or documented). Please double-check to make sure that it is not a bug.
Changelog	⚠ doc not found	The connector does not seem to have a documentation file. This can be normal (e.g. basic connector like `source-jdbc` is not published or documented). Please double-check to make sure that it is not a bug.
	❌ changelog missing	There is no chnagelog for the current version of the connector. If you are the author of the current version, please add a changelog.
Publish	⚠ not in seed	The connector is not in the cloud or oss registry, so its publication status cannot be checked. This can be normal (e.g. some connectors are cloud-specific, and only listed in the cloud seed file). Please double-check to make sure that you have added a metadata.yaml file and the expected registries are enabled.

evantahler · 2023-05-16T23:35:53Z

airbyte-integrations/connectors/destination-redshift/Dockerfile

@@ -18,3 +37,6 @@ COPY --from=build /airbyte /airbyte

 LABEL io.airbyte.version=0.4.7


Should there be a version bump here?

yeah, I haven't done the metadata.yaml / changelog either. just wanted to get the pr up and deal with the tedious bits later... i.e. today 😓

evantahler · 2023-05-16T23:36:16Z

tools/ci_connector_ops/ci_connector_ops/pipelines/actions/environments.py

@@ -566,7 +566,7 @@ def with_integration_base_java(context: PipelineContext, build_platform: Platfor
        "dockerfile": "redshift.Dockerfile",
        "dbt_adapter": "dbt-redshift==1.0.0",
        "integration_name": "redshift",
-        "supports_in_connector_normalization": False,
+        "supports_in_connector_normalization": True,


airbyte-config-oss/init-oss/src/main/resources/seed/oss_registry.json

edgao · 2023-05-17T17:26:13Z

/test connector=connectors/destination-redshift

🕑 connectors/destination-redshift https://github.com/airbytehq/airbyte/actions/runs/5006158169
✅ connectors/destination-redshift https://github.com/airbytehq/airbyte/actions/runs/5006158169
Python tests coverage:

Name                                                              Stmts   Miss  Cover
-------------------------------------------------------------------------------------
normalization/transform_config/__init__.py                            2      0   100%
normalization/transform_catalog/reserved_keywords.py                 15      0   100%
normalization/transform_catalog/__init__.py                           2      0   100%
normalization/destination_type.py                                    18      0   100%
normalization/__init__.py                                             4      0   100%
normalization/transform_catalog/destination_name_transformer.py     171     10    94%
normalization/transform_catalog/table_name_registry.py              174     34    80%
normalization/transform_config/transform.py                         195     48    75%
normalization/transform_catalog/utils.py                             51     14    73%
normalization/transform_catalog/dbt_macro.py                         22      7    68%
normalization/transform_catalog/catalog_processor.py                147     80    46%
normalization/transform_catalog/transform.py                         65     39    40%
normalization/transform_catalog/stream_processor.py                 603    407    33%
-------------------------------------------------------------------------------------
TOTAL                                                              1469    639    57%

Build Passed

Test summary info:

All Passed

edgao · 2023-05-25T18:24:35Z

/test connector=connectors/destination-redshift

🕑 connectors/destination-redshift https://github.com/airbytehq/airbyte/actions/runs/5083312708
✅ connectors/destination-redshift https://github.com/airbytehq/airbyte/actions/runs/5083312708
Python tests coverage:

Name                                                              Stmts   Miss  Cover
-------------------------------------------------------------------------------------
normalization/transform_config/__init__.py                            2      0   100%
normalization/transform_catalog/reserved_keywords.py                 15      0   100%
normalization/transform_catalog/__init__.py                           2      0   100%
normalization/destination_type.py                                    18      0   100%
normalization/__init__.py                                             4      0   100%
normalization/transform_catalog/destination_name_transformer.py     171     10    94%
normalization/transform_catalog/table_name_registry.py              174     34    80%
normalization/transform_config/transform.py                         195     48    75%
normalization/transform_catalog/utils.py                             51     14    73%
normalization/transform_catalog/dbt_macro.py                         22      7    68%
normalization/transform_catalog/catalog_processor.py                147     80    46%
normalization/transform_catalog/transform.py                         65     39    40%
normalization/transform_catalog/stream_processor.py                 603    407    33%
-------------------------------------------------------------------------------------
TOTAL                                                              1469    639    57%

Build Passed

Test summary info:

All Passed

* do things in-connector * Automated Change * version bumps + changelog * remove unnecessary yum package * derp * also build.gradle * Automated Change --------- Co-authored-by: edgao <edgao@users.noreply.github.com>

do things in-connector

604584a

octavia-squidington-iii added area/connectors Connector related issues connectors/destination/redshift labels May 16, 2023

edgao marked this pull request as ready for review May 16, 2023 23:11

edgao requested a review from a team as a code owner May 16, 2023 23:11

Automated Change

f2b84ae

evantahler approved these changes May 16, 2023

View reviewed changes

version bumps + changelog

814efe8

octavia-squidington-iii added the area/documentation Improvements or additions to documentation label May 17, 2023

remove unnecessary yum package

3b72664

edgao mentioned this pull request May 22, 2023

Implement in-destination normalization for clickhouse #25495

Closed

evantahler mentioned this pull request May 22, 2023

Destination MySQL: Install normalization #26376

Open

edgao mentioned this pull request May 23, 2023

[Epic] Move normalization into destinations #24791

Closed

edgao added 4 commits May 25, 2023 09:59

Merge branch 'master' into edgao/redshift_normalization_in_connector

0432799

derp

fc703cc

also build.gradle

c391a10

Merge branch 'master' into edgao/redshift_normalization_in_connector

54343b4

edgao and others added 2 commits May 25, 2023 18:43

Automated Change

d804955

Merge branch 'master' into edgao/redshift_normalization_in_connector

aab9b3e

edgao enabled auto-merge (squash) May 25, 2023 23:20

edgao merged commit bbdb3be into master May 25, 2023
19 checks passed

edgao deleted the edgao/redshift_normalization_in_connector branch May 25, 2023 23:40

edgao mentioned this pull request May 25, 2023

Implement in-destination normalization for redshift #25499

Closed

cynthiaxyin mentioned this pull request Dec 19, 2023

Add in container normalization for postgres #33458

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Destination Redshift: Install normalization #26165

Destination Redshift: Install normalization #26165

edgao commented May 16, 2023 •

edited

github-actions bot commented May 16, 2023 •

edited by edgao

edgao commented May 16, 2023 •

edited by github-actions bot

github-actions bot commented May 16, 2023 •

edited

✅ Sources (0)

❌ Destinations (1)

👀 Other Modules (1)

Actionable Items

evantahler May 16, 2023

edgao May 17, 2023

evantahler May 16, 2023

edgao commented May 17, 2023 •

edited by github-actions bot

edgao commented May 25, 2023 •

edited by github-actions bot

		@@ -18,3 +37,6 @@ COPY --from=build /airbyte /airbyte

		LABEL io.airbyte.version=0.4.7

Destination Redshift: Install normalization #26165

Destination Redshift: Install normalization #26165

Conversation

edgao commented May 16, 2023 • edited

github-actions bot commented May 16, 2023 • edited by edgao

Before Merging a Connector Pull Request

edgao commented May 16, 2023 • edited by github-actions bot

Build Passed

github-actions bot commented May 16, 2023 • edited

Affected Connector Report

✅ Sources (0)

❌ Destinations (1)

👀 Other Modules (1)

Actionable Items

evantahler May 16, 2023

Choose a reason for hiding this comment

edgao May 17, 2023

Choose a reason for hiding this comment

evantahler May 16, 2023

Choose a reason for hiding this comment

edgao commented May 17, 2023 • edited by github-actions bot

Build Passed

edgao commented May 25, 2023 • edited by github-actions bot

Build Passed

edgao commented May 16, 2023 •

edited

github-actions bot commented May 16, 2023 •

edited by edgao

edgao commented May 16, 2023 •

edited by github-actions bot

github-actions bot commented May 16, 2023 •

edited

edgao commented May 17, 2023 •

edited by github-actions bot

edgao commented May 25, 2023 •

edited by github-actions bot