Skip to content

2.27.0.0-b464

@fourpointfour fourpointfour tagged this 19 Aug 09:30
Summary:
After generating unique record IDs, for comparing 2 records, we have checks to compare them using commit_time, record_time, write_id, etc to ascertain which record comes before in a sorted order. The check is performed in the method `CDCSDKUniqueRecordID::GreaterThanDistributedLSN`.

Now when the GUC `yb_disable_transactional_writes` is set, we have multiple records inserted in a single `WRITE_OP` batch for a single shard transaction, there's a possibility that the records end up having the same `commit_time`, `record_time`, `write_id` and `table_id` so we fallback to comparing the primary keys. The core issue was that the VWAL expects each individual tablet to send records in order as per the order determined by `CDCSDKUniqueRecordID::GreaterThanDistributedLSN`. This was being violated and the situation leads us to a data loss scenario where we end up losing multiple records inserted in the same batch of a single shard transaction.

The fix to this requires a mechanism to reliably have a fixed sorting order when all the other parameters end up with the same value. This PR adds the same mechanism by assigning a `write_id` to every single shard record based on the index of the wal records batch we are processing. By doing this, we'll be breaking the tie using `write_id` and we can reliably sort the records without filtering them.
Jira: DB-17813

Test Plan:
The tests to reproduce the error and validate the fix has been added as a part of the logical replication connector's test suite in the following PR:
https://github.com/yugabyte/debezium/pull/182

Additionally, even though the issue will not be applicable to the gRPC connector, we are also adding a test prudently to gRPC connector as well:
https://github.com/yugabyte/debezium-connector-yugabytedb/pull/379

Reviewers: asrinivasan, sumukh.phalgaonkar, skumar

Reviewed By: skumar

Subscribers: ycdcxcluster

Differential Revision: https://phorge.dev.yugabyte.com/D45857
Assets 2
Loading