Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Standardize spec.json for DB connectors that support log-based CDC replication #12917

Closed
ThaliaBarrera opened this issue May 17, 2022 · 2 comments · Fixed by #16216 or #16215
Closed

Standardize spec.json for DB connectors that support log-based CDC replication #12917

ThaliaBarrera opened this issue May 17, 2022 · 2 comments · Fixed by #16216 or #16215
Assignees
@VitaliiMaltsev
Labels
area/connectors Connector related issues autoteam cdc connectors/source/mysql needs-triage type/enhancement New feature or request

Comments

@ThaliaBarrera
Copy link
Contributor

ThaliaBarrera commented May 17, 2022 •
edited by ChristopheDuong
Loading

In the analytics team we are looking for a way to track CDC vs. non-CDC source statistics. For more information, refer to this issue.

When trying to expose the fields in the warehouse, we realized there's a mismatch in spec.json files between Postgres and MySQL, for example. The difference in specs makes it difficult to track the fields, so we would like to standardize them.

image

The spec.json of connectors such as MySql should use a oneOf called "replication_method" where the method can be selected via a const value of CDC instead (follow postgres connector)
@VitaliiMaltsev VitaliiMaltsev self-assigned this Jul 5, 2022
@VitaliiMaltsev
Copy link
Contributor

To avoid backward compatibility issue should be merged after #14196

@grishick grishick mentioned this issue Jul 8, 2022
Closed
4 tasks
@VitaliiMaltsev
Copy link
Contributor

#14574 was merged. Please proceed

@VitaliiMaltsev VitaliiMaltsev linked a pull request Aug 3, 2022 that will close this issue
Standardize spec.json for DB connectors that support log-based CDC replication #14910
Closed
37 tasks
This was linked to pull requests Sep 5, 2022
MySQL Source : Standardize spec.json for DB connectors that support log-based CDC replication #16216
Merged
MSSQL Source : Standardize spec.json for DB connectors that support log-based CDC replication #16215
Merged
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@VitaliiMaltsev VitaliiMaltsev

Labels
area/connectors Connector related issues autoteam cdc connectors/source/mysql needs-triage type/enhancement New feature or request
Projects
None yet
Milestone
No milestone
5 participants
@grishick @ChristopheDuong @ThaliaBarrera @VitaliiMaltsev @octavia-squidington-iii