Skip to content

In-connector normalization: make logs less misleading - #26585

Merged
Edward Gao (edgao) merged 3 commits into
masterfrom
edgao/normalization_script_tweak
May 25, 2023
Merged

Edward Gao (edgao) merged 3 commits into
masterfrom
edgao/normalization_script_tweak

Conversation

@edgao

@edgao Edward Gao (edgao) commented May 25, 2023 •

Copy link
Copy Markdown
Contributor

from https://github.com/airbytehq/oncall/issues/2122#issuecomment-1562574875

example with normalization enabled
image
image

example with normalization disabled
image

@github-actions

github-actions Bot commented May 25, 2023 •

Copy link
Copy Markdown
Contributor

Affected Connector Report

NOTE ⚠️ Changes in this PR affect the following connectors. Make sure to do the following as needed:

  • Run integration tests
  • Bump connector or module version
  • Add changelog
  • Publish the new version

❌ Sources (33)

Connector Version Changelog Publish
source-alloydb 2.0.28 ✅ ✅
source-alloydb-strict-encrypt 2.0.28 🔵
(ignored)
🔵
(ignored)
source-azure-blob-storage 0.1.0 ✅ ✅
source-bigquery 0.2.3 ✅ ✅
source-clickhouse 0.1.17 ✅ ✅
source-clickhouse-strict-encrypt 0.1.17 🔵
(ignored)
🔵
(ignored)
source-cockroachdb 0.1.22 ✅ ✅
source-cockroachdb-strict-encrypt 0.1.22 🔵
(ignored)
🔵
(ignored)
source-db2 0.1.19 ✅ ✅
source-db2-strict-encrypt 0.1.19 🔵
(ignored)
🔵
(ignored)
source-dynamodb 0.1.2 ✅ ✅
source-e2e-test 2.1.4 ✅ ✅
source-e2e-test-cloud 2.1.4 🔵
(ignored)
🔵
(ignored)
source-elasticsearch 0.1.1 ✅ ✅
source-jdbc 0.3.5 🔵
(ignored)
🔵
(ignored)
source-kafka 0.2.3 ✅ ✅
source-mongodb-strict-encrypt 0.1.19 🔵
(ignored)
🔵
(ignored)
source-mongodb-v2 0.1.19 ✅ ✅
source-mssql 1.0.17 ✅ ❌
(diff seed version)
source-mssql-strict-encrypt 1.0.17 🔵
(ignored)
🔵
(ignored)
source-mysql 2.0.24 ✅ ❌
(diff seed version)
source-mysql-strict-encrypt 2.0.24 🔵
(ignored)
🔵
(ignored)
source-oracle 0.3.24 ✅ ✅
source-oracle-strict-encrypt 0.3.24 🔵
(ignored)
🔵
(ignored)
source-postgres 2.0.30 ✅ ❌
(diff seed version)
source-postgres-strict-encrypt 2.0.30 🔵
(ignored)
🔵
(ignored)
source-redshift 0.3.16 ✅ ✅
source-relational-db 0.3.1 🔵
(ignored)
🔵
(ignored)
source-scaffold-java-jdbc 0.1.0 🔵
(ignored)
🔵
(ignored)
source-sftp 0.1.2 ✅ ✅
source-snowflake 0.1.34 ✅ ✅
source-teradata 0.1.0 ✅ ✅
source-tidb 0.2.4 ✅ ✅
  • See "Actionable Items" below for how to resolve warnings and errors.

❌ Destinations (49)

Connector Version Changelog Publish
destination-azure-blob-storage 0.2.0 ✅ ✅
destination-bigquery 1.4.4 ✅ ❌
(diff seed version)
destination-bigquery-denormalized 1.4.1 ✅ ✅
destination-cassandra 0.1.4 ✅ ✅
destination-clickhouse 0.2.3 ✅ ✅
destination-clickhouse-strict-encrypt 0.2.3 🔵
(ignored)
🔵
(ignored)
destination-csv 1.0.0 ✅ ✅
destination-databricks 1.0.2 ✅ ✅
destination-dev-null 0.3.0 🔵
(ignored)
🔵
(ignored)
destination-doris 0.1.0 ✅ ✅
destination-dynamodb 0.1.7 ✅ ✅
destination-e2e-test 0.3.0 ✅ ✅
destination-elasticsearch 0.1.6 ✅ ✅
destination-elasticsearch-strict-encrypt 0.1.6 🔵
(ignored)
🔵
(ignored)
destination-exasol 0.1.1 ✅ ✅
destination-gcs 0.3.0 ✅ ✅
destination-iceberg 0.1.0 ✅ ✅
destination-kafka 0.1.10 ✅ ✅
destination-keen 0.2.4 ✅ ✅
destination-kinesis 0.1.5 ✅ ✅
destination-local-json 0.2.11 ✅ ✅
destination-mariadb-columnstore 0.1.7 ✅ ✅
destination-mongodb 0.1.9 ✅ ✅
destination-mongodb-strict-encrypt 0.1.9 🔵
(ignored)
🔵
(ignored)
destination-mqtt 0.1.3 ✅ ✅
destination-mssql 0.1.23 ✅ ✅
destination-mssql-strict-encrypt 0.1.23 🔵
(ignored)
🔵
(ignored)
destination-mysql 0.1.20 ✅ ✅
destination-mysql-strict-encrypt ❌ 0.1.21
(mismatch: 0.1.20)
🔵
(ignored)
🔵
(ignored)
destination-oracle 0.1.19 ✅ ✅
destination-oracle-strict-encrypt 0.1.19 🔵
(ignored)
🔵
(ignored)
destination-postgres 0.3.27 ✅ ✅
destination-postgres-strict-encrypt 0.3.27 🔵
(ignored)
🔵
(ignored)
destination-pubsub 0.2.0 ✅ ✅
destination-pulsar 0.1.3 ✅ ✅
destination-r2 0.1.0 ✅ ✅
destination-redis 0.1.4 ✅ ✅
destination-redpanda 0.1.0 ✅ ✅
destination-redshift 0.4.7 ✅ ✅
destination-rockset 0.1.4 ✅ ✅
destination-s3 0.4.1 ✅ ✅
destination-s3-glue 0.1.7 ✅ ✅
destination-scylla 0.1.3 ✅ ✅
destination-selectdb 0.1.0 ✅ ✅
destination-snowflake 1.0.4 ✅ ✅
destination-starburst-galaxy 0.0.1 ✅ ✅
destination-teradata 0.1.1 ✅ ✅
destination-tidb 0.1.1 ✅ ❌
(diff seed version)
destination-yugabytedb 0.1.1 ✅ ✅
  • See "Actionable Items" below for how to resolve warnings and errors.

👀 Other Modules (1)

  • base-normalization

Actionable Items

(click to expand)

Category Status Actionable Item
Version ❌
mismatch
The version of the connector is different from its normal variant. Please bump the version of the connector.
⚠
doc not found
The connector does not seem to have a documentation file. This can be normal (e.g. basic connector like source-jdbc is not published or documented). Please double-check to make sure that it is not a bug.
Changelog ⚠
doc not found
The connector does not seem to have a documentation file. This can be normal (e.g. basic connector like source-jdbc is not published or documented). Please double-check to make sure that it is not a bug.
❌
changelog missing
There is no chnagelog for the current version of the connector. If you are the author of the current version, please add a changelog.
Publish ⚠
not in seed
The connector is not in the cloud or oss registry, so its publication status cannot be checked. This can be normal (e.g. some connectors are cloud-specific, and only listed in the cloud seed file). Please double-check to make sure that you have added a metadata.yaml file and the expected registries are enabled.

@edgao
Edward Gao (edgao) marked this pull request as ready for review May 25, 2023 16:42
@edgao
Edward Gao (edgao) requested a review from a team as a code owner May 25, 2023 16:42
@edgao
Edward Gao (edgao) requested a review from a team as a code owner May 25, 2023 17:52
@edgao
Edward Gao (edgao) enabled auto-merge (squash) May 25, 2023 17:53

@evantahler Evan Tahler (evantahler) left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

some grammar nits, but 👍

Comment thread airbyte-integrations/bases/base-java/run_with_normalization.sh Outdated
Comment thread airbyte-integrations/bases/base-java/run_with_normalization.sh Outdated
@edgao
Edward Gao (edgao) enabled auto-merge (squash) May 25, 2023 17:58
@edgao
Edward Gao (edgao) merged commit cf2ded2 into master May 25, 2023
@edgao
Edward Gao (edgao) deleted the edgao/normalization_script_tweak branch May 25, 2023 18:20
M. Marx (marcosmarxm) pushed a commit to natalia-miinto/airbyte that referenced this pull request Jun 8, 2023
* make logs less misleading

* version bumps + changelog

* tweak wording
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

area/connectors Connector related issues area/documentation Improvements or additions to documentation connectors/destination/bigquery

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants