🐛 Fix incremental normalization with empty tables #8394

ChristopheDuong · 2021-12-01T17:35:46Z

What

Closes #8286
Closes #8028

How

Handle incremental clauses when final table is empty
Upgrade dbt version for mssql (0.21.0 to handle incremental & schema change better)
Postgres index with btree instead of hash (so a comparison between dates with < or > is more performant)

🚨 User Impact 🚨

Are there any breaking changes? If yes, please make sure to include it here and in any changelogs with the 🚨🚨 emoji
What is the end result perceived by the user?

Pre-merge Checklist

Expand the relevant checklist and delete the others.

New Connector

Community member or Airbyter

Airbyter

If this is a community PR, the Airbyte engineer reviewing this PR is responsible for the below items.

Create a non-forked branch based on this PR and test the below items on it
Build is successful
Credentials added to Github CI. Instructions.
/test connector=connectors/<name> command is passing.
New Connector version released on Dockerhub by running the /publish command described here
After the connector is published, connector added to connector index as described here
Seed specs have been re-generated by building the platform and committing the changes to the seed spec files, as described here

Updating a connector

Community member or Airbyter

Grant edit access to maintainers (instructions)
Secrets in the connector's spec are annotated with airbyte_secret
Unit & integration tests added and passing. Community members, please provide proof of success locally e.g: screenshot or copy-paste unit, integration, and acceptance test output. To run acceptance tests for a Python connector, follow instructions in the README. For java connectors run ./gradlew :airbyte-integrations:connectors:<name>:integrationTest.
Code reviews completed
Documentation updated
- Connector's README.md
- Connector's bootstrap.md. See description and examples
- Changelog updated in docs/integrations/<source or destination>/<name>.md including changelog. See changelog example
PR name follows PR naming conventions

Airbyter

If this is a community PR, the Airbyte engineer reviewing this PR is responsible for the below items.

Create a non-forked branch based on this PR and test the below items on it
Build is successful
Credentials added to Github CI. Instructions.
/test connector=connectors/<name> command is passing.
New Connector version released on Dockerhub by running the /publish command described here
After the new connector version is published, connector version bumped in the seed directory as described here
Seed specs have been re-generated by building the platform and committing the changes to the seed spec files, as described here

Connector Generator

Issue acceptance criteria met
PR name follows PR naming conventions
If adding a new generator, add it to the list of scaffold modules being tested
The generator test modules (all connectors with -scaffold in their name) have been updated with the latest scaffold by running ./gradlew :airbyte-integrations:connector-templates:generator:testScaffoldTemplates then checking in your changes
Documentation which references the generator is updated as needed.

ChristopheDuong · 2021-12-01T17:51:02Z

/test connector=bases/base-normalization

🕑 bases/base-normalization https://github.com/airbytehq/airbyte/actions/runs/1527080035
❌ bases/base-normalization https://github.com/airbytehq/airbyte/actions/runs/1527080035
🐛

ChristopheDuong · 2021-12-01T20:07:46Z

/test connector=bases/base-normalization

🕑 bases/base-normalization https://github.com/airbytehq/airbyte/actions/runs/1527592332
✅ bases/base-normalization https://github.com/airbytehq/airbyte/actions/runs/1527592332
Python tests coverage:

	 ---------- coverage: platform linux, python 3.8.10-final-0 -----------
	 Name                                       Stmts   Miss  Cover
	 --------------------------------------------------------------
	 base_python/__init__.py                       13      0   100%
	 base_python/catalog_helpers.py                10      6    40%
	 base_python/cdk/__init__.py                    0      0   100%
	 base_python/cdk/abstract_source.py            89     64    28%
	 base_python/cdk/streams/__init__.py            0      0   100%
	 base_python/cdk/streams/auth/__init__.py       0      0   100%
	 base_python/cdk/streams/auth/core.py           8      1    88%
	 base_python/cdk/streams/auth/jwt.py            5      5     0%
	 base_python/cdk/streams/auth/oauth.py         37     26    30%
	 base_python/cdk/streams/auth/token.py          9      4    56%
	 base_python/cdk/streams/core.py               63     32    49%
	 base_python/cdk/streams/exceptions.py         10      2    80%
	 base_python/cdk/streams/http.py               67     33    51%
	 base_python/cdk/streams/rate_limiting.py      30     14    53%
	 base_python/cdk/utils/__init__.py              0      0   100%
	 base_python/cdk/utils/casing.py                4      0   100%
	 base_python/cdk/utils/event_timing.py         47      3    94%
	 base_python/client.py                         56     33    41%
	 base_python/entrypoint.py                     70     56    20%
	 base_python/integration.py                    52     25    52%
	 base_python/logger.py                         33     15    55%
	 base_python/schema_helpers.py                 56     41    27%
	 base_python/source.py                         51     34    33%
	 main_dev.py                                    3      3     0%
	 --------------------------------------------------------------
	 TOTAL                                        713    397    44%
	 ---------- coverage: platform linux, python 3.8.10-final-0 -----------
	 Name                                                              Stmts   Miss  Cover
	 -------------------------------------------------------------------------------------
	 main_dev_transform_catalog.py                                         3      3     0%
	 main_dev_transform_config.py                                          3      3     0%
	 normalization/__init__.py                                             4      0   100%
	 normalization/destination_type.py                                    12      0   100%
	 normalization/transform_catalog/__init__.py                           2      0   100%
	 normalization/transform_catalog/catalog_processor.py                143     77    46%
	 normalization/transform_catalog/destination_name_transformer.py     120      6    95%
	 normalization/transform_catalog/reserved_keywords.py                 11      0   100%
	 normalization/transform_catalog/stream_processor.py                 468    287    39%
	 normalization/transform_catalog/table_name_registry.py              174     34    80%
	 normalization/transform_catalog/transform.py                         45     26    42%
	 normalization/transform_catalog/utils.py                             33      7    79%
	 normalization/transform_config/__init__.py                            2      0   100%
	 normalization/transform_config/transform.py                         140     29    79%
	 -------------------------------------------------------------------------------------
	 TOTAL                                                              1160    472    59%
	 ---------- coverage: platform linux, python 3.8.10-final-0 -----------
	 Name                                                 Stmts   Miss  Cover
	 ------------------------------------------------------------------------
	 source_acceptance_test/__init__.py                       2      0   100%
	 source_acceptance_test/base.py                          10      4    60%
	 source_acceptance_test/config.py                        76      8    89%
	 source_acceptance_test/conftest.py                     108    108     0%
	 source_acceptance_test/plugin.py                        47     47     0%
	 source_acceptance_test/tests/__init__.py                 4      0   100%
	 source_acceptance_test/tests/test_core.py              235     95    60%
	 source_acceptance_test/tests/test_full_refresh.py       38     27    29%
	 source_acceptance_test/tests/test_incremental.py        69     38    45%
	 source_acceptance_test/utils/__init__.py                 6      0   100%
	 source_acceptance_test/utils/asserts.py                 37      2    95%
	 source_acceptance_test/utils/common.py                  54     24    56%
	 source_acceptance_test/utils/compare.py                 62     25    60%
	 source_acceptance_test/utils/connector_runner.py        82     49    40%
	 source_acceptance_test/utils/json_schema_helper.py     115     14    88%
	 ------------------------------------------------------------------------
	 TOTAL                                                  945    441    53%
	 ---------- coverage: platform linux, python 3.8.10-final-0 -----------
	 Name                                                              Stmts   Miss  Cover
	 -------------------------------------------------------------------------------------
	 main_dev_transform_catalog.py                                         3      3     0%
	 main_dev_transform_config.py                                          3      3     0%
	 normalization/__init__.py                                             4      0   100%
	 normalization/destination_type.py                                    12      0   100%
	 normalization/transform_catalog/__init__.py                           2      0   100%
	 normalization/transform_catalog/catalog_processor.py                143     77    46%
	 normalization/transform_catalog/destination_name_transformer.py     120      6    95%
	 normalization/transform_catalog/reserved_keywords.py                 11      0   100%
	 normalization/transform_catalog/stream_processor.py                 468    287    39%
	 normalization/transform_catalog/table_name_registry.py              174     34    80%
	 normalization/transform_catalog/transform.py                         45     26    42%
	 normalization/transform_catalog/utils.py                             33      7    79%
	 normalization/transform_config/__init__.py                            2      0   100%
	 normalization/transform_config/transform.py                         140     29    79%
	 -------------------------------------------------------------------------------------
	 TOTAL                                                              1160    472    59%
	 ---------- coverage: platform linux, python 3.8.10-final-0 -----------
	 Name                                                              Stmts   Miss  Cover
	 -------------------------------------------------------------------------------------
	 main_dev_transform_catalog.py                                         3      3     0%
	 main_dev_transform_config.py                                          3      3     0%
	 normalization/__init__.py                                             4      0   100%
	 normalization/destination_type.py                                    12      0   100%
	 normalization/transform_catalog/__init__.py                           2      0   100%
	 normalization/transform_catalog/catalog_processor.py                143     12    92%
	 normalization/transform_catalog/destination_name_transformer.py     120      4    97%
	 normalization/transform_catalog/reserved_keywords.py                 11      0   100%
	 normalization/transform_catalog/stream_processor.py                 468     37    92%
	 normalization/transform_catalog/table_name_registry.py              174     51    71%
	 normalization/transform_catalog/transform.py                         45     30    33%
	 normalization/transform_catalog/utils.py                             33      0   100%
	 normalization/transform_config/__init__.py                            2      0   100%
	 normalization/transform_config/transform.py                         140     45    68%
	 -------------------------------------------------------------------------------------
	 TOTAL                                                              1160    185    84%

ChristopheDuong · 2021-12-01T21:27:55Z

/publish connector=bases/base-normalization

🕑 bases/base-normalization https://github.com/airbytehq/airbyte/actions/runs/1527882817
✅ bases/base-normalization https://github.com/airbytehq/airbyte/actions/runs/1527882817

* Fix incremental with empty final tables * upgrade docker images * Regen SQL * Bumpversion & format

github-actions bot added area/documentation Improvements or additions to documentation normalization labels Dec 1, 2021

ChristopheDuong added 3 commits December 1, 2021 18:36

Fix incremental with empty final tables

6fb65a8

upgrade docker images

5c5a1ca

Regen SQL

7c60b99

ChristopheDuong force-pushed the chris/normalization-fix branch from a834c80 to 7c60b99 Compare December 1, 2021 17:36

ChristopheDuong temporarily deployed to more-secrets December 1, 2021 17:38 Inactive

ChristopheDuong requested review from sherifnada and tuliren December 1, 2021 17:41

jrhizor temporarily deployed to more-secrets December 1, 2021 17:53 Inactive

sherifnada approved these changes Dec 1, 2021

View reviewed changes

Fix MSSql tests

a813bf8

ChristopheDuong temporarily deployed to more-secrets December 1, 2021 20:09 Inactive

jrhizor temporarily deployed to more-secrets December 1, 2021 20:10 Inactive

ChristopheDuong added 2 commits December 1, 2021 21:35

Bumpversion & format

06d7f8f

Merge remote-tracking branch 'origin/master' into normalization-fix

47e56a4

github-actions bot added area/connectors Connector related issues area/platform issues related to the platform area/worker Related to worker labels Dec 1, 2021

ChristopheDuong temporarily deployed to more-secrets December 1, 2021 21:28 Inactive

jrhizor temporarily deployed to more-secrets December 1, 2021 21:30 Inactive

ChristopheDuong merged commit b424c1a into master Dec 1, 2021

ChristopheDuong deleted the chris/normalization-fix branch December 1, 2021 22:40

jrhizor mentioned this pull request Dec 2, 2021

Bump Airbyte version from 0.32.10-alpha to 0.32.11-alpha #8426

Merged

schlattk pushed a commit to schlattk/airbyte that referenced this pull request Jan 4, 2022

🐛 Fix incremental normalization with empty tables (airbytehq#8394)

a1f1d34

* Fix incremental with empty final tables * upgrade docker images * Regen SQL * Bumpversion & format

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

🐛 Fix incremental normalization with empty tables #8394

🐛 Fix incremental normalization with empty tables #8394

ChristopheDuong commented Dec 1, 2021 •

edited

Loading

ChristopheDuong commented Dec 1, 2021 •

edited by github-actions bot

Loading

ChristopheDuong commented Dec 1, 2021 •

edited by github-actions bot

Loading

ChristopheDuong commented Dec 1, 2021 •

edited by github-actions bot

Loading

🐛 Fix incremental normalization with empty tables #8394

🐛 Fix incremental normalization with empty tables #8394

Conversation

ChristopheDuong commented Dec 1, 2021 • edited Loading

What

How

Recommended reading order

🚨 User Impact 🚨

Pre-merge Checklist

Community member or Airbyter

Airbyter

Community member or Airbyter

Airbyter

ChristopheDuong commented Dec 1, 2021 • edited by github-actions bot Loading

ChristopheDuong commented Dec 1, 2021 • edited by github-actions bot Loading

ChristopheDuong commented Dec 1, 2021 • edited by github-actions bot Loading

ChristopheDuong commented Dec 1, 2021 •

edited

Loading

ChristopheDuong commented Dec 1, 2021 •

edited by github-actions bot

Loading

ChristopheDuong commented Dec 1, 2021 •

edited by github-actions bot

Loading

ChristopheDuong commented Dec 1, 2021 •

edited by github-actions bot

Loading