Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Modify CDC to be inline with namespace changes. #2986

Merged
merged 3 commits into from
Apr 20, 2021

Conversation

davinchia
Copy link
Contributor

@davinchia davinchia commented Apr 20, 2021

What

As the title suggests.

How

Slight logic changes for make CDC in line with new namespacing expectations, namely:

  1. schema is no longer part of the stream name. it is now set as the namespace field.
  2. CDC needs to be able to read/write from different schemas. (modified the existing test to support this)

I will publish the connector in a follow up PR where I'm going to publish a bunch of connectors before we release anyway.

Pre-merge Checklist

  • Run integration tests

Recommended reading order

  1. DezebiumEventUtils.java
    2.DezebiumRecordPublisher.java
  2. All the test files. Mostly importantly, CdcPostgreSourceTest

@@ -88,8 +88,10 @@
private static final Logger LOGGER = LoggerFactory.getLogger(CdcPostgresSourceTest.class);

private static final String SLOT_NAME_BASE = "debezium_slot";
private static final String MAKES_STREAM_NAME = "public.makes";
private static final String MODELS_STREAM_NAME = "public.models";
private static final String MAKES_SCHEMA = "public";
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Modified one of the tables to be in a separate schema to confirm data is propagated correctly. I think this is correct, but please sanity check my work.

@davinchia
Copy link
Contributor Author

davinchia commented Apr 20, 2021

/test connector=source-postgres

🕑 source-postgres https://github.com/airbytehq/airbyte/actions/runs/766289684
✅ source-postgres https://github.com/airbytehq/airbyte/actions/runs/766289684

Copy link
Contributor

@cgardens cgardens left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

lgtm

@davinchia davinchia merged commit 7e55f3a into master Apr 20, 2021
@davinchia davinchia deleted the davinchia/fix-postgres-cdc-tests branch April 20, 2021 23:39
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

None yet

2 participants