[xCluster] Investigate if the consumer can GC away packing schemas still needed by incoming data from the producer #17308
Assignees
Labels
area/docdb
YugabyteDB core features
kind/enhancement
This is an enhancement of an existing feature
priority/medium
Medium priority issue
Comments
mdbridge
added
area/docdb
yugabyte-ci
added
kind/bug
|
This issue was created as part of the investigation done in [xCluster] Handle any issues due to replication repeating changes #15547 |
yugabyte-ci
added
kind/enhancement
karan-yb
added a commit
that referenced
this issue
Jun 7, 2023
…nal records Summary: Prior to this change, there was no tracking of smallest/largest used schema packings at memtable/SST file level. This change adds this tracking which ensures that we don't GC schema packings aggressively. Jira: DB-6532 Test Plan: ./yb_build.sh --sj ./build/latest/tests-integration-tests/xcluster_ysql-test Reviewers: slingam, xCluster Reviewed By: slingam Subscribers: ybase Differential Revision: https://phorge.dev.yugabyte.com/D26034
karan-yb
added a commit
that referenced
this issue
Jun 8, 2023
…ntier for external records Summary: Original commit: fb6b943 / D26034 Prior to this change, there was no tracking of smallest/largest used schema packings at memtable/SST file level. This change adds this tracking which ensures that we don't GC schema packings aggressively. Jira: DB-6532 Test Plan: ./yb_build.sh --sj ./build/latest/tests-integration-tests/xcluster_ysql-test Reviewers: slingam, xCluster Reviewed By: slingam Subscribers: ybase Differential Revision: https://phorge.dev.yugabyte.com/D26055
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Jira Link: DB-6532
Description
When we replicate DML operations from the producer to the consumer, we rewrite the packing schema version in the payloads to reference an equivalent packing schema.
This task is to figure out if it is possible for the consumer to GC away the old packing schema that corresponds to incoming producer DML's packing schemas.
There is an interlock so the consumer should not GC packing schemas currently present in its RocksDB's, but it's not clear there is an interlock versus yet to arrive producer DML's.
I assume if we can't find an equivalent packing schema we have to drop the DML's or pause replication, so this is a possible data loss problem so likely quite serious if it occurs.
Warning: Please confirm that this issue does not contain any sensitive information
The text was updated successfully, but these errors were encountered: