TiDB Destination: add append_dedup mode #19109

Daemonxiao · 2022-11-08T05:48:33Z

What

TiDB Destination support "deduped history" mode. So add append_dedup mode in spec.json.

How

Describe the solution

Pre-merge Checklist

Expand the relevant checklist and delete the others.

Updating a connector

Community member or Airbyter

Grant edit access to maintainers (instructions)
Secrets in the connector's spec are annotated with airbyte_secret
Unit & integration tests added and passing. Community members, please provide proof of success locally e.g: screenshot or copy-paste unit, integration, and acceptance test output. To run acceptance tests for a Python connector, follow instructions in the README. For java connectors run ./gradlew :airbyte-integrations:connectors:<name>:integrationTest.
Code reviews completed
Documentation updated
- Connector's README.md
- Connector's bootstrap.md. See description and examples
- Changelog updated in docs/integrations/<source or destination>/<name>.md including changelog. See changelog example
PR name follows PR naming conventions

Airbyter

If this is a community PR, the Airbyte engineer reviewing this PR is responsible for the below items.

Create a non-forked branch based on this PR and test the below items on it
[] Build is successful
If new credentials are required for use in CI, add them to GSM. Instructions.
/test connector=connectors/<name> command is passing
New Connector version released on Dockerhub and connector version bumped by running the /publish command described here

Tests

Integration

Daemonxiao · 2022-11-08T05:48:53Z

/test connector=connectors/destination-tidb

YiyangLi · 2022-11-18T18:26:47Z

/test connector=connectors/destination-tidb

🕑 connectors/destination-tidb https://github.com/airbytehq/airbyte/actions/runs/3499313396
✅ connectors/destination-tidb https://github.com/airbytehq/airbyte/actions/runs/3499313396
Python tests coverage:

Name                                                              Stmts   Miss  Cover
-------------------------------------------------------------------------------------
normalization/transform_config/__init__.py                            2      0   100%
normalization/transform_catalog/reserved_keywords.py                 14      0   100%
normalization/transform_catalog/__init__.py                           2      0   100%
normalization/destination_type.py                                    14      0   100%
normalization/__init__.py                                             4      0   100%
normalization/transform_catalog/destination_name_transformer.py     166      8    95%
normalization/transform_catalog/table_name_registry.py              174     34    80%
normalization/transform_config/transform.py                         189     48    75%
normalization/transform_catalog/utils.py                             51     14    73%
normalization/transform_catalog/dbt_macro.py                         22      7    68%
normalization/transform_catalog/catalog_processor.py                147     80    46%
normalization/transform_catalog/transform.py                         61     38    38%
normalization/transform_catalog/stream_processor.py                 595    400    33%
-------------------------------------------------------------------------------------
TOTAL                                                              1441    629    56%

Build Passed

Test summary info:

All Passed

Daemonxiao · 2022-11-22T02:21:34Z

@YiyangLi Is this PR OK to merge?

YiyangLi

YiyangLi · 2022-11-22T16:35:03Z

/publish connector=connectors/destination-tidb

🕑 Publishing the following connectors:
connectors/destination-tidb
https://github.com/airbytehq/airbyte/actions/runs/3525293788

Connector	Did it publish?	Were definitions generated?
connectors/destination-tidb	❌	❌

if you have connectors that successfully published but failed definition generation, follow step 4 here ▶️

YiyangLi · 2022-11-22T16:46:56Z

/publish connector=connectors/destination-tidb

🕑 Publishing the following connectors:
connectors/destination-tidb
https://github.com/airbytehq/airbyte/actions/runs/3525379516

Connector	Did it publish?	Were definitions generated?
connectors/destination-tidb	❌	❌

if you have connectors that successfully published but failed definition generation, follow step 4 here ▶️

marcosmarxm · 2022-11-22T20:24:57Z

Hello 👋, first thank you for this amazing contribution.

We really appreciate the effort you've made to improve the project.
We ask you patience for the code review. Last month our team was focused on Hacktoberfest event and that probably left some PR without the proper feedback. And this week, due to the Thanksgiving US Holiday, most our team is out of office with their families. Another important piece of information why code won't be merge this week is: as a safety measure the core team has decided to freeze merging code to main branch to keep the release stable. Next week we'll return to you with the proper code review and update the status of your contribution.

If you have any questions feel free to send me a message in Slack!
Thanks!

YiyangLi · 2022-11-29T19:33:57Z

/publish connector=connectors/destination-tidb

🕑 Publishing the following connectors:
connectors/destination-tidb
https://github.com/airbytehq/airbyte/actions/runs/3577403466

Connector	Did it publish?	Were definitions generated?
connectors/destination-tidb	❌	❌

if you have connectors that successfully published but failed definition generation, follow step 4 here ▶️

marcosmarxm · 2022-12-01T16:31:55Z

I don't think it will work only adding the append_dedup. Probably you must change the normalization factory in the worker. Also maybe there is some work to made in normalization module.

Daemonxiao · 2022-12-02T03:35:26Z

@marcosmarxm Thanks for your advice. 🙏
TiDB normalization module uses the dbt-tidb package. The default increment mode of dbt-tidb is delete+insert, which means it could auto-delete the duplicate row with unique keys and then insert the new row. This mode works like upsert. Besides, we have already passed the acceptance test and it has tested the append_dedup mode.

marcosmarxm · 2022-12-20T19:51:32Z

Hello 👋:skin-tone-2: and thank you for your contribution!

Airbyte has instituted a code freeze between 19 and 30 December, to make sure there are no disruptions during the holidays.
Because of this, reviewing and merging your contribution may take longer than usual.
We apologize for the delay, but we want everyone to have a quiet and happy holiday.

If you have any questions or need further clarification, please don't hesitate to ping via Slack.

Daemonxiao · 2023-02-16T02:41:55Z

Is this pr ready to be merged? Or is there anything else I can do?

marcosmarxm · 2023-05-19T20:25:09Z

/test connector=connectors/destination-tidb

🕑 connectors/destination-tidb https://github.com/airbytehq/airbyte/actions/runs/5028032356
❌ connectors/destination-tidb https://github.com/airbytehq/airbyte/actions/runs/5028032356
🐛 https://gradle.com/s/g34daxfsf4f2i

Build Failed

Test summary info:

Could not find result summary

marcosmarxm · 2023-05-19T20:36:38Z

/test connector=connectors/destination-tidb

🕑 connectors/destination-tidb https://github.com/airbytehq/airbyte/actions/runs/5028109127
✅ connectors/destination-tidb https://github.com/airbytehq/airbyte/actions/runs/5028109127
Python tests coverage:

Name                                                              Stmts   Miss  Cover
-------------------------------------------------------------------------------------
normalization/transform_config/__init__.py                            2      0   100%
normalization/transform_catalog/reserved_keywords.py                 15      0   100%
normalization/transform_catalog/__init__.py                           2      0   100%
normalization/destination_type.py                                    18      0   100%
normalization/__init__.py                                             4      0   100%
normalization/transform_catalog/destination_name_transformer.py     171     10    94%
normalization/transform_catalog/table_name_registry.py              174     34    80%
normalization/transform_config/transform.py                         195     48    75%
normalization/transform_catalog/utils.py                             51     14    73%
normalization/transform_catalog/dbt_macro.py                         22      7    68%
normalization/transform_catalog/catalog_processor.py                147     80    46%
normalization/transform_catalog/transform.py                         65     39    40%
normalization/transform_catalog/stream_processor.py                 603    407    33%
-------------------------------------------------------------------------------------
TOTAL                                                              1469    639    57%

Build Passed

Test summary info:

All Passed

* TiDB Destination: add append_dedup mode * update Dockerfile * update docs and metadata file --------- Co-authored-by: Marcos Marx <marcosmarxm@users.noreply.github.com> Co-authored-by: marcosmarxm <marcosmarxm@gmail.com>

TiDB Destination: add append_dedup mode

334ba05

github-actions bot added the area/connectors Connector related issues label Nov 8, 2022

octavia-squidington-iv added bounty community connectors/destination/tidb labels Nov 8, 2022

update Dockerfile

62d2909

YiyangLi approved these changes Nov 22, 2022

View reviewed changes

sajarin added the bounty-L Maintainer program: claimable large bounty PR label Dec 5, 2022

Daemonxiao added 3 commits January 3, 2023 10:30

Merge branch 'master' into tidb-destination-spec

61b088f

Merge branch 'master' into tidb-destination-spec

444174b

Merge branch 'master' into tidb-destination-spec

8a3fa4a

Merge branch 'master' into tidb-destination-spec

6f6e7e6

marcosmarxm added the pr-alpha-beta label Apr 14, 2023

marcosmarxm added community-legacy and removed community labels May 15, 2023

Merge branch 'master' into tidb-destination-spec

8926cbb

Merge branch 'master' into tidb-destination-spec

f2b83d6

update docs and metadata file

ef7ebdb

octavia-squidington-iii added the area/documentation Improvements or additions to documentation label May 23, 2023

marcosmarxm approved these changes May 23, 2023

View reviewed changes

marcosmarxm merged commit 1e1bdac into airbytehq:master May 23, 2023
35 of 43 checks passed

marcosmarxm added community and removed community-legacy labels Aug 21, 2023

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

TiDB Destination: add append_dedup mode #19109

TiDB Destination: add append_dedup mode #19109

Daemonxiao commented Nov 8, 2022 •

edited

Daemonxiao commented Nov 8, 2022

YiyangLi commented Nov 18, 2022 •

edited by github-actions bot

Daemonxiao commented Nov 22, 2022

YiyangLi left a comment

YiyangLi commented Nov 22, 2022 •

edited by github-actions bot

YiyangLi commented Nov 22, 2022 •

edited by github-actions bot

marcosmarxm commented Nov 22, 2022

YiyangLi commented Nov 29, 2022 •

edited by github-actions bot

marcosmarxm commented Dec 1, 2022

Daemonxiao commented Dec 2, 2022

marcosmarxm commented Dec 20, 2022

Daemonxiao commented Feb 16, 2023

marcosmarxm commented May 19, 2023 •

edited by github-actions bot

marcosmarxm commented May 19, 2023 •

edited by github-actions bot

TiDB Destination: add append_dedup mode #19109

TiDB Destination: add append_dedup mode #19109

Conversation

Daemonxiao commented Nov 8, 2022 • edited

What

How

Recommended reading order

Pre-merge Checklist

Community member or Airbyter

Airbyter

Tests

Daemonxiao commented Nov 8, 2022

YiyangLi commented Nov 18, 2022 • edited by github-actions bot

Build Passed

Daemonxiao commented Nov 22, 2022

YiyangLi left a comment

Choose a reason for hiding this comment

YiyangLi commented Nov 22, 2022 • edited by github-actions bot

YiyangLi commented Nov 22, 2022 • edited by github-actions bot

marcosmarxm commented Nov 22, 2022

YiyangLi commented Nov 29, 2022 • edited by github-actions bot

marcosmarxm commented Dec 1, 2022

Daemonxiao commented Dec 2, 2022

marcosmarxm commented Dec 20, 2022

Daemonxiao commented Feb 16, 2023

marcosmarxm commented May 19, 2023 • edited by github-actions bot

Build Failed

marcosmarxm commented May 19, 2023 • edited by github-actions bot

Build Passed

Daemonxiao commented Nov 8, 2022 •

edited

YiyangLi commented Nov 18, 2022 •

edited by github-actions bot

YiyangLi commented Nov 22, 2022 •

edited by github-actions bot

YiyangLi commented Nov 22, 2022 •

edited by github-actions bot

YiyangLi commented Nov 29, 2022 •

edited by github-actions bot

marcosmarxm commented May 19, 2023 •

edited by github-actions bot

marcosmarxm commented May 19, 2023 •

edited by github-actions bot