Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[DBZ-PGYB][#21591] Support to process update records with CHANGE #106

Open
wants to merge 23 commits into
base: ybdb-debezium-2.5.2
Choose a base branch
from

Conversation

vaibhav-yb
Copy link
Collaborator

@vaibhav-yb vaibhav-yb commented Mar 20, 2024

These changes add the support to the PG connector to stream UPDATE records from the YugabyteDB server.

Summary:

Suppose we execute the following statements:

CREATE TABLE test (pk INT PRIMARY KEY, aa INT, bb TEXT);
INSERT INTO test VALUES (1, 1, 'hello-world'); # Event 1
UPDATE test SET aa = 2 WHERE pk = 1; # Event 2
UPDATE test SET aa = NULL WHERE pk = 1; # Event 3

The payload structure for the above events would be as follows:

  1. Event 1: pk:{"value":1,"set":true},aa:{"value":1,"set":true},bb:{"value":"hello-world","set":true}
  2. Event 2: pk:{"value":1,"set":true},aa:{"value":2,"set":true},bb:null
  3. Event 3: pk:{"value":1,"set":true},aa:{"value":null,"set":true},bb:null

Changes in PR:

  1. Added PGTableSchemaBuilder to facilitate custom schema when replica identity is CHANGE.
  2. Added transformer to be used in sink connector YBExtractNewRecordState.
  3. Streaming and snapshot change event sources now use a modified emitter which also takes the replica identity as a parameter.

@vaibhav-yb vaibhav-yb self-assigned this Mar 20, 2024
@vaibhav-yb vaibhav-yb changed the title [DBZ-PGYB][#21591][WIP] Support to process update records with CHANGE [DBZ-PGYB][#21591] Support to process update records with CHANGE Mar 26, 2024
@vaibhav-yb vaibhav-yb added enhancement New feature or request bug Something isn't working labels Mar 26, 2024
@dr0pdb dr0pdb self-requested a review March 28, 2024 05:30
@dr0pdb
Copy link

dr0pdb commented Mar 28, 2024

A basic question to confirm.

  • bb: null means that the value is not being sent. It doesn't mean that the value of column bb is NULL
  • bb:{"value":null,"set":true} means that the value is explicitly NULL.

Is this correct?

@vaibhav-yb
Copy link
Collaborator Author

A basic question to confirm.

  • bb: null means that the value is not being sent. It doesn't mean that the value of column bb is NULL
  • bb:{"value":null,"set":true} means that the value is explicitly NULL.

Is this correct?

@dr0pdb Yes, your understanding is correct.

@yugabyte yugabyte deleted a comment from asrinivasanyb Mar 28, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
bug Something isn't working enhancement New feature or request
Projects
None yet
4 participants