Add namespace test in destination acceptance test by tuliren · Pull Request #10793 · airbytehq/airbyte

Liren Tu (tuliren) · 2022-03-02T12:59:29Z

What

Close Create a well defined charset for namespaces and test them in DAT #9351.
The new versions are published in Publish new snowflake and bigquery versions #11274.

How

Add a new test in DAT.
Run a simple data sync to a variety of namespaces on the destination.
The test is disabled by default. It is currently only activated for Snowflake and BigQuery.

🚨 User Impact 🚨

None expected.

Liren Tu (tuliren) · 2022-03-02T13:31:32Z

/test connector=connectors/destination-snowflake

🕑 connectors/destination-snowflake https://github.com/airbytehq/airbyte/actions/runs/1922475141
❌ connectors/destination-snowflake https://github.com/airbytehq/airbyte/actions/runs/1922475141
🐛

Liren Tu (tuliren) · 2022-03-02T13:31:39Z

/test connector=connectors/destination-bigquery

🕑 connectors/destination-bigquery https://github.com/airbytehq/airbyte/actions/runs/1922475558
❌ connectors/destination-bigquery https://github.com/airbytehq/airbyte/actions/runs/1922475558
🐛

Edward Gao (edgao)

Edward Gao (edgao) · 2022-03-02T21:16:25Z

@@ -0,0 +1,51 @@
+[
+  {
+    "id": "s1-1",


nitpick: can these IDs be a little more descriptive (e.g. all_caps) to make them easier to cross-reference? right now if a test fails, it'll just log that case s1-1 failed, and then the developer would need to find this file to understand why.

Done. I also updated the exception message to ask people to look into the namespace_test_case.json for details.

Liren Tu (tuliren) · 2022-03-02T23:27:57Z

/test connector=connectors/destination-snowflake

🕑 connectors/destination-snowflake https://github.com/airbytehq/airbyte/actions/runs/1925195221
❌ connectors/destination-snowflake https://github.com/airbytehq/airbyte/actions/runs/1925195221
🐛

Liren Tu (tuliren) · 2022-03-02T23:28:05Z

/test connector=connectors/destination-bigquery

🕑 connectors/destination-bigquery https://github.com/airbytehq/airbyte/actions/runs/1925195831
❌ connectors/destination-bigquery https://github.com/airbytehq/airbyte/actions/runs/1925195831
🐛

We don't want to introduce changes that will affect existing connections for now.

Liren Tu (tuliren) · 2022-03-18T09:48:34Z

/test connector=connectors/destination-bigquery-denormalized

🕑 connectors/destination-bigquery-denormalized https://github.com/airbytehq/airbyte/actions/runs/2003407941
✅ connectors/destination-bigquery-denormalized https://github.com/airbytehq/airbyte/actions/runs/2003407941
Python tests coverage:

Name                                                                                                                            Stmts   Miss  Cover
---------------------------------------------------------------------------------------------------------------------------------------------------
normalization/transform_config/__init__.py                                                                                          2      0   100%
normalization/transform_catalog/reserved_keywords.py                                                                               13      0   100%
normalization/transform_catalog/__init__.py                                                                                         2      0   100%
normalization/destination_type.py                                                                                                  13      0   100%
normalization/__init__.py                                                                                                           4      0   100%
/actions-runner/_work/airbyte/airbyte/airbyte-integrations/bases/airbyte-protocol/airbyte_protocol/models/airbyte_protocol.py     124      0   100%
/actions-runner/_work/airbyte/airbyte/airbyte-integrations/bases/airbyte-protocol/airbyte_protocol/models/__init__.py               1      0   100%
/actions-runner/_work/airbyte/airbyte/airbyte-integrations/bases/airbyte-protocol/airbyte_protocol/__init__.py                      2      0   100%
normalization/transform_catalog/destination_name_transformer.py                                                                   155      8    95%
normalization/transform_config/transform.py                                                                                       159     30    81%
normalization/transform_catalog/table_name_registry.py                                                                            174     34    80%
normalization/transform_catalog/utils.py                                                                                           33      7    79%
normalization/transform_catalog/catalog_processor.py                                                                              143     77    46%
normalization/transform_catalog/transform.py                                                                                       45     26    42%
normalization/transform_catalog/stream_processor.py                                                                               524    337    36%
---------------------------------------------------------------------------------------------------------------------------------------------------
TOTAL                                                                                                                            1394    519    63%

Liren Tu (tuliren) · 2022-03-18T09:53:34Z

/test connector=connectors/destination-bigquery

🕑 connectors/destination-bigquery https://github.com/airbytehq/airbyte/actions/runs/2003428337
❌ connectors/destination-bigquery https://github.com/airbytehq/airbyte/actions/runs/2003428337
🐛

Liren Tu (tuliren) · 2022-03-18T15:13:24Z

/test connector=connectors/destination-snowflake

🕑 connectors/destination-snowflake https://github.com/airbytehq/airbyte/actions/runs/2004904499
✅ connectors/destination-snowflake https://github.com/airbytehq/airbyte/actions/runs/2004904499
Python tests coverage:

Name                                                                                                                            Stmts   Miss  Cover
---------------------------------------------------------------------------------------------------------------------------------------------------
normalization/transform_config/__init__.py                                                                                          2      0   100%
normalization/transform_catalog/reserved_keywords.py                                                                               13      0   100%
normalization/transform_catalog/__init__.py                                                                                         2      0   100%
normalization/destination_type.py                                                                                                  13      0   100%
normalization/__init__.py                                                                                                           4      0   100%
/actions-runner/_work/airbyte/airbyte/airbyte-integrations/bases/airbyte-protocol/airbyte_protocol/models/airbyte_protocol.py     124      0   100%
/actions-runner/_work/airbyte/airbyte/airbyte-integrations/bases/airbyte-protocol/airbyte_protocol/models/__init__.py               1      0   100%
/actions-runner/_work/airbyte/airbyte/airbyte-integrations/bases/airbyte-protocol/airbyte_protocol/__init__.py                      2      0   100%
normalization/transform_catalog/destination_name_transformer.py                                                                   155      8    95%
normalization/transform_config/transform.py                                                                                       159     30    81%
normalization/transform_catalog/table_name_registry.py                                                                            174     34    80%
normalization/transform_catalog/utils.py                                                                                           33      7    79%
normalization/transform_catalog/catalog_processor.py                                                                              143     77    46%
normalization/transform_catalog/transform.py                                                                                       45     26    42%
normalization/transform_catalog/stream_processor.py                                                                               524    337    36%
---------------------------------------------------------------------------------------------------------------------------------------------------
TOTAL                                                                                                                            1394    519    63%

Liren Tu (tuliren) · 2022-03-18T15:13:30Z

/test connector=connectors/destination-bigquery

🕑 connectors/destination-bigquery https://github.com/airbytehq/airbyte/actions/runs/2004904984
✅ connectors/destination-bigquery https://github.com/airbytehq/airbyte/actions/runs/2004904984
Python tests coverage:

Name                                                                                                                            Stmts   Miss  Cover
---------------------------------------------------------------------------------------------------------------------------------------------------
normalization/transform_config/__init__.py                                                                                          2      0   100%
normalization/transform_catalog/reserved_keywords.py                                                                               13      0   100%
normalization/transform_catalog/__init__.py                                                                                         2      0   100%
normalization/destination_type.py                                                                                                  13      0   100%
normalization/__init__.py                                                                                                           4      0   100%
/actions-runner/_work/airbyte/airbyte/airbyte-integrations/bases/airbyte-protocol/airbyte_protocol/models/airbyte_protocol.py     124      0   100%
/actions-runner/_work/airbyte/airbyte/airbyte-integrations/bases/airbyte-protocol/airbyte_protocol/models/__init__.py               1      0   100%
/actions-runner/_work/airbyte/airbyte/airbyte-integrations/bases/airbyte-protocol/airbyte_protocol/__init__.py                      2      0   100%
normalization/transform_catalog/destination_name_transformer.py                                                                   155      8    95%
normalization/transform_config/transform.py                                                                                       159     30    81%
normalization/transform_catalog/table_name_registry.py                                                                            174     34    80%
normalization/transform_catalog/utils.py                                                                                           33      7    79%
normalization/transform_catalog/catalog_processor.py                                                                              143     77    46%
normalization/transform_catalog/transform.py                                                                                       45     26    42%
normalization/transform_catalog/stream_processor.py                                                                               524    337    36%
---------------------------------------------------------------------------------------------------------------------------------------------------
TOTAL                                                                                                                            1394    519    63%

Liren Tu (tuliren) · 2022-03-18T15:13:33Z

/test connector=connectors/destination-bigquery-denormalized

🕑 connectors/destination-bigquery-denormalized https://github.com/airbytehq/airbyte/actions/runs/2004905343
❌ connectors/destination-bigquery-denormalized https://github.com/airbytehq/airbyte/actions/runs/2004905343
🐛

Liren Tu (tuliren) · 2022-03-19T00:33:39Z

/test connector=connectors/destination-bigquery-denormalized

🕑 connectors/destination-bigquery-denormalized https://github.com/airbytehq/airbyte/actions/runs/2007062177
✅ connectors/destination-bigquery-denormalized https://github.com/airbytehq/airbyte/actions/runs/2007062177
Python tests coverage:

Name                                                                                                                            Stmts   Miss  Cover
---------------------------------------------------------------------------------------------------------------------------------------------------
normalization/transform_config/__init__.py                                                                                          2      0   100%
normalization/transform_catalog/reserved_keywords.py                                                                               13      0   100%
normalization/transform_catalog/__init__.py                                                                                         2      0   100%
normalization/destination_type.py                                                                                                  13      0   100%
normalization/__init__.py                                                                                                           4      0   100%
/actions-runner/_work/airbyte/airbyte/airbyte-integrations/bases/airbyte-protocol/airbyte_protocol/models/airbyte_protocol.py     124      0   100%
/actions-runner/_work/airbyte/airbyte/airbyte-integrations/bases/airbyte-protocol/airbyte_protocol/models/__init__.py               1      0   100%
/actions-runner/_work/airbyte/airbyte/airbyte-integrations/bases/airbyte-protocol/airbyte_protocol/__init__.py                      2      0   100%
normalization/transform_catalog/destination_name_transformer.py                                                                   155      8    95%
normalization/transform_config/transform.py                                                                                       159     30    81%
normalization/transform_catalog/table_name_registry.py                                                                            174     34    80%
normalization/transform_catalog/utils.py                                                                                           33      7    79%
normalization/transform_catalog/catalog_processor.py                                                                              143     77    46%
normalization/transform_catalog/transform.py                                                                                       45     26    42%
normalization/transform_catalog/stream_processor.py                                                                               524    337    36%
---------------------------------------------------------------------------------------------------------------------------------------------------
TOTAL                                                                                                                            1394    519    63%

Liren Tu (tuliren) · 2022-03-19T08:26:23Z

/test connector=connectors/destination-bigquery-denormalized

🕑 connectors/destination-bigquery-denormalized https://github.com/airbytehq/airbyte/actions/runs/2008100587
✅ connectors/destination-bigquery-denormalized https://github.com/airbytehq/airbyte/actions/runs/2008100587
Python tests coverage:

Name                                                                                                                            Stmts   Miss  Cover
---------------------------------------------------------------------------------------------------------------------------------------------------
normalization/transform_config/__init__.py                                                                                          2      0   100%
normalization/transform_catalog/reserved_keywords.py                                                                               13      0   100%
normalization/transform_catalog/__init__.py                                                                                         2      0   100%
normalization/destination_type.py                                                                                                  13      0   100%
normalization/__init__.py                                                                                                           4      0   100%
/actions-runner/_work/airbyte/airbyte/airbyte-integrations/bases/airbyte-protocol/airbyte_protocol/models/airbyte_protocol.py     124      0   100%
/actions-runner/_work/airbyte/airbyte/airbyte-integrations/bases/airbyte-protocol/airbyte_protocol/models/__init__.py               1      0   100%
/actions-runner/_work/airbyte/airbyte/airbyte-integrations/bases/airbyte-protocol/airbyte_protocol/__init__.py                      2      0   100%
normalization/transform_catalog/destination_name_transformer.py                                                                   155      8    95%
normalization/transform_config/transform.py                                                                                       159     30    81%
normalization/transform_catalog/table_name_registry.py                                                                            174     34    80%
normalization/transform_catalog/utils.py                                                                                           33      7    79%
normalization/transform_catalog/catalog_processor.py                                                                              143     77    46%
normalization/transform_catalog/transform.py                                                                                       45     26    42%
normalization/transform_catalog/stream_processor.py                                                                               524    337    36%
---------------------------------------------------------------------------------------------------------------------------------------------------
TOTAL                                                                                                                            1394    519    63%

Liren Tu (tuliren) · 2022-03-19T08:26:28Z

/test connector=connectors/destination-bigquery

🕑 connectors/destination-bigquery https://github.com/airbytehq/airbyte/actions/runs/2008100740
✅ connectors/destination-bigquery https://github.com/airbytehq/airbyte/actions/runs/2008100740
Python tests coverage:

Name                                                                                                                            Stmts   Miss  Cover
---------------------------------------------------------------------------------------------------------------------------------------------------
normalization/transform_config/__init__.py                                                                                          2      0   100%
normalization/transform_catalog/reserved_keywords.py                                                                               13      0   100%
normalization/transform_catalog/__init__.py                                                                                         2      0   100%
normalization/destination_type.py                                                                                                  13      0   100%
normalization/__init__.py                                                                                                           4      0   100%
/actions-runner/_work/airbyte/airbyte/airbyte-integrations/bases/airbyte-protocol/airbyte_protocol/models/airbyte_protocol.py     124      0   100%
/actions-runner/_work/airbyte/airbyte/airbyte-integrations/bases/airbyte-protocol/airbyte_protocol/models/__init__.py               1      0   100%
/actions-runner/_work/airbyte/airbyte/airbyte-integrations/bases/airbyte-protocol/airbyte_protocol/__init__.py                      2      0   100%
normalization/transform_catalog/destination_name_transformer.py                                                                   155      8    95%
normalization/transform_config/transform.py                                                                                       159     30    81%
normalization/transform_catalog/table_name_registry.py                                                                            174     34    80%
normalization/transform_catalog/utils.py                                                                                           33      7    79%
normalization/transform_catalog/catalog_processor.py                                                                              143     77    46%
normalization/transform_catalog/transform.py                                                                                       45     26    42%
normalization/transform_catalog/stream_processor.py                                                                               524    337    36%
---------------------------------------------------------------------------------------------------------------------------------------------------
TOTAL                                                                                                                            1394    519    63%

Liren Tu (tuliren) · 2022-03-19T08:26:34Z

/test connector=connectors/destination-snowflake

🕑 connectors/destination-snowflake https://github.com/airbytehq/airbyte/actions/runs/2008101018
✅ connectors/destination-snowflake https://github.com/airbytehq/airbyte/actions/runs/2008101018
Python tests coverage:

Name                                                                                                                            Stmts   Miss  Cover
---------------------------------------------------------------------------------------------------------------------------------------------------
normalization/transform_config/__init__.py                                                                                          2      0   100%
normalization/transform_catalog/reserved_keywords.py                                                                               13      0   100%
normalization/transform_catalog/__init__.py                                                                                         2      0   100%
normalization/destination_type.py                                                                                                  13      0   100%
normalization/__init__.py                                                                                                           4      0   100%
/actions-runner/_work/airbyte/airbyte/airbyte-integrations/bases/airbyte-protocol/airbyte_protocol/models/airbyte_protocol.py     124      0   100%
/actions-runner/_work/airbyte/airbyte/airbyte-integrations/bases/airbyte-protocol/airbyte_protocol/models/__init__.py               1      0   100%
/actions-runner/_work/airbyte/airbyte/airbyte-integrations/bases/airbyte-protocol/airbyte_protocol/__init__.py                      2      0   100%
normalization/transform_catalog/destination_name_transformer.py                                                                   155      8    95%
normalization/transform_config/transform.py                                                                                       159     30    81%
normalization/transform_catalog/table_name_registry.py                                                                            174     34    80%
normalization/transform_catalog/utils.py                                                                                           33      7    79%
normalization/transform_catalog/catalog_processor.py                                                                              143     77    46%
normalization/transform_catalog/transform.py                                                                                       45     26    42%
normalization/transform_catalog/stream_processor.py                                                                               524    337    36%
---------------------------------------------------------------------------------------------------------------------------------------------------
TOTAL                                                                                                                            1394    519    63%

Liren Tu (tuliren) added 2 commits March 2, 2022 04:54

Add namespace test for snowflake

2075116

Enable namespace test for bigquery

53e035b

github-actions Bot added the area/connectors Connector related issues label Mar 2, 2022

Format code

72cbd76

Liren Tu (tuliren) temporarily deployed to more-secrets March 2, 2022 13:02 Inactive

Octavia Squidington III (octavia-squidington-iii) temporarily deployed to more-secrets March 2, 2022 13:34 Inactive

Liren Tu (tuliren) requested a review from Edward Gao (edgao) March 2, 2022 17:17

Edward Gao (edgao) approved these changes Mar 2, 2022

View reviewed changes

Liren Tu (tuliren) added 2 commits March 2, 2022 15:26

Capitalize test case id

2292000

Update exception message to point to test case file

7c6fdc2

Liren Tu (tuliren) temporarily deployed to more-secrets March 2, 2022 23:28 Inactive

Octavia Squidington III (octavia-squidington-iii) temporarily deployed to more-secrets March 2, 2022 23:30 Inactive

Liren Tu (tuliren) added 10 commits March 17, 2022 19:04

Merge branch 'master' into liren/destination-namespace-charset

668e0fe

Update snowflake name transformer to prepend underscore

82b4f58

Override convertStreamName instead of getIdentifier

e2d3506

Add missing state message

e46e0f2

Remove unused import

1a5e73d

Disable more namespace test cases

d7eda39

We don't want to introduce changes that will affect existing connections for now.

Dry method that mutates namespace

1435264

Pass through null

e698272

Normalize namespace

241455a

Fix test case

cc7da57

Liren Tu (tuliren) added 3 commits March 18, 2022 02:26

Bump version for snowflake and bigquery

019d64a

Enable namespace test for bigquery denormalized

c14e92d

Dry bigquery denormalized acceptance test

3f9111b

github-actions Bot added the area/documentation Improvements or additions to documentation label Mar 18, 2022

Revert some of the variable scope change

e4be015

Liren Tu (tuliren) added 6 commits March 18, 2022 03:06

Fix unit test

6412b7b

Bump version

dcb8f98

Introduce getNamespace method

89f9f35

Implement getNamespace method for bigquery

bf159c0

Switch to getNamespace methods

b758324

Update comments

7b8708a

Liren Tu (tuliren) added 2 commits March 18, 2022 14:36

Fix bigquery denormalized acceptance test

7ca043d

Format code

9e93a4a

Liren Tu (tuliren) added 4 commits March 18, 2022 21:48

Dry bigquery destination test

c226dfc

Skip partition test for gcs mode

b9caead

Merge branch 'master' into liren/destination-namespace-charset

b4d789c

Bump version

c359284

Liren Tu (tuliren) merged commit 21ec23c into master Mar 20, 2022

Liren Tu (tuliren) deleted the liren/destination-namespace-charset branch March 20, 2022 00:47

Liren Tu (tuliren) mentioned this pull request Mar 20, 2022

Publish new snowflake and bigquery versions #11274

Merged

Octavia Squidington III (octavia-squidington-iii) mentioned this pull request Mar 21, 2022

Bump Airbyte version from 0.35.58-alpha to 0.35.59-alpha #11298

Merged

Conversation

Liren Tu (tuliren) commented Mar 2, 2022 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

What

How

Recommended reading order

🚨 User Impact 🚨

Uh oh!

Liren Tu (tuliren) commented Mar 2, 2022 • edited by github-actions Bot Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Liren Tu (tuliren) commented Mar 2, 2022 • edited by github-actions Bot Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Edward Gao (edgao) left a comment

Choose a reason for hiding this comment

Uh oh!

Edward Gao (edgao) Mar 2, 2022

Choose a reason for hiding this comment

Uh oh!

Liren Tu (tuliren) Mar 2, 2022

Choose a reason for hiding this comment

Uh oh!

Liren Tu (tuliren) commented Mar 2, 2022 • edited by github-actions Bot Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Liren Tu (tuliren) commented Mar 2, 2022 • edited by github-actions Bot Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Liren Tu (tuliren) commented Mar 18, 2022 • edited by github-actions Bot Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Liren Tu (tuliren) commented Mar 18, 2022 • edited by github-actions Bot Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Liren Tu (tuliren) commented Mar 18, 2022 • edited by github-actions Bot Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Liren Tu (tuliren) commented Mar 18, 2022 • edited by github-actions Bot Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Liren Tu (tuliren) commented Mar 18, 2022 • edited by github-actions Bot Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Liren Tu (tuliren) commented Mar 19, 2022 • edited by github-actions Bot Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Liren Tu (tuliren) commented Mar 19, 2022 • edited by github-actions Bot Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Liren Tu (tuliren) commented Mar 19, 2022 • edited by github-actions Bot Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Liren Tu (tuliren) commented Mar 19, 2022 • edited by github-actions Bot Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

Liren Tu (tuliren) commented Mar 2, 2022 •

edited

Loading

Liren Tu (tuliren) commented Mar 2, 2022 •

edited by github-actions Bot

Loading

Liren Tu (tuliren) commented Mar 2, 2022 •

edited by github-actions Bot

Loading

Liren Tu (tuliren) commented Mar 2, 2022 •

edited by github-actions Bot

Loading

Liren Tu (tuliren) commented Mar 2, 2022 •

edited by github-actions Bot

Loading

Liren Tu (tuliren) commented Mar 18, 2022 •

edited by github-actions Bot

Loading

Liren Tu (tuliren) commented Mar 18, 2022 •

edited by github-actions Bot

Loading

Liren Tu (tuliren) commented Mar 18, 2022 •

edited by github-actions Bot

Loading

Liren Tu (tuliren) commented Mar 18, 2022 •

edited by github-actions Bot

Loading

Liren Tu (tuliren) commented Mar 18, 2022 •

edited by github-actions Bot

Loading

Liren Tu (tuliren) commented Mar 19, 2022 •

edited by github-actions Bot

Loading

Liren Tu (tuliren) commented Mar 19, 2022 •

edited by github-actions Bot

Loading

Liren Tu (tuliren) commented Mar 19, 2022 •

edited by github-actions Bot

Loading

Liren Tu (tuliren) commented Mar 19, 2022 •

edited by github-actions Bot

Loading