Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

File expiration metadata not being included in frontiers for tablet replicas #10504

Closed
jmeehan16 opened this issue Nov 5, 2021 · 0 comments
Closed

File expiration metadata not being included in frontiers for tablet replicas #10504

jmeehan16 opened this issue Nov 5, 2021 · 0 comments
Assignees
@jmeehan16
Labels
area/docdb YugabyteDB core features
Projects
TTL file expiry YBase features

Comments

@jmeehan16
Copy link
Contributor

In clusters with RF > 1, files are being generated without max_value_level_ttl_expiration_time metadata.
@jmeehan16 jmeehan16 created this issue from a note in TTL file expiry (To do) Nov 5, 2021
@jmeehan16 jmeehan16 self-assigned this Nov 5, 2021
@jmeehan16 jmeehan16 moved this from To do to In progress in TTL file expiry Nov 5, 2021
@bmatican bmatican added the area/docdb YugabyteDB core features label Nov 6, 2021
@bmatican bmatican added this to Backlog in YBase features via automation Nov 6, 2021
jmeehan16 added a commit that referenced this issue Nov 12, 2021
@jmeehan16
[#10504] docdb - Move value TTL metadata collection to DocWriteBatch
bd2626d 
Summary:
Request TTL metadata is currently missing from tablet replicas, leading to replica SST files
missing value TTL frontier metadata necessary for expiration. This is happening because the metadata
was being stored in operations (which are rebuilt on replicas) rather than being stored in the doc
batch write requests themselves.

This commit moves the value TTL metadata collection into DocWriteBatch and adds a field for ttl in the
KeyValueWriteBatchPB. It also modifies the logic on the tablet side to calculate the expiration timestamp
based on the KeyValueWriteBatchPB metadata.

Also adds the CompactionTestWithFileExpirationRF3 test suite.

Test Plan:
Jenkins: hot
ybd --cxx-test integration-tests_compaction-test --gtest_filter CompactionTestWithFileExpiration.ReplicatedMetadataCanExpireFile
ybd --cxx-test integration-tests_compaction-test --gtest_filter CompactionTestWithFileExpiration.ReplicatedNoMetadataUsesTableTTL
ybd --cxx-test docdb_docdb-test --gtest_filter DocDBTests/DocDBTestWrapper.TestUpdateDocWriteBatchTTL/0

Reviewers: timur, rsami, sergei

Reviewed By: sergei

Subscribers: ybase, anallan, bogdan

Differential Revision: https://phabricator.dev.yugabyte.com/D13855
jmeehan16 added a commit that referenced this issue Nov 14, 2021
@jmeehan16
[BACKPORT 2.6][#10504] docdb - Move value TTL metadata collection to …
8200c41 
…DocWriteBatch

Summary:
Request TTL metadata is currently missing from tablet replicas, leading to replica SST files
missing value TTL frontier metadata necessary for expiration. This is happening because the metadata
was being stored in operations (which are rebuilt on replicas) rather than being stored in the doc
batch write requests themselves.

This commit moves the value TTL metadata collection into DocWriteBatch and adds a field for ttl in the
KeyValueWriteBatchPB. It also modifies the logic on the tablet side to calculate the expiration timestamp
based on the KeyValueWriteBatchPB metadata.

Also adds the CompactionTestWithFileExpirationRF3 test suite.

Adjusted compaction-test.cc for 2.6 constructors

Original commit: rYBDBbd2626d2b50f / D13855

Test Plan:
Jenkins: hot
Jenkins: rebase: 2.6
ybd --cxx-test integration-tests_compaction-test --gtest_filter CompactionTestWithFileExpiration.ReplicatedMetadataCanExpireFile
ybd --cxx-test integration-tests_compaction-test --gtest_filter CompactionTestWithFileExpiration.ReplicatedNoMetadataUsesTableTTL
ybd --cxx-test docdb_docdb-test --gtest_filter DocDBTests/DocDBTestWrapper.TestUpdateDocWriteBatchTTL/0

Reviewers: timur, rsami, sergei

Reviewed By: sergei

Subscribers: bogdan, anallan, ybase

Differential Revision: https://phabricator.dev.yugabyte.com/D13927
jmeehan16 added a commit that referenced this issue Nov 19, 2021
@jmeehan16
[BACKPORT 2.8][#10504] docdb - Move value TTL metadata collection to …
1da5ae4 
…DocWriteBatch

Summary:
Request TTL metadata is currently missing from tablet replicas, leading to replica SST files
missing value TTL frontier metadata necessary for expiration. This is happening because the metadata
was being stored in operations (which are rebuilt on replicas) rather than being stored in the doc
batch write requests themselves.

This commit moves the value TTL metadata collection into DocWriteBatch and adds a field for ttl in the
KeyValueWriteBatchPB. It also modifies the logic on the tablet side to calculate the expiration timestamp
based on the KeyValueWriteBatchPB metadata.

Also adds the CompactionTestWithFileExpirationRF3 test suite.

Original commit: rYBDBbd2626d2b50f / D13855

Test Plan:
Jenkins: rebase: 2.8
ybd --cxx-test integration-tests_compaction-test --gtest_filter CompactionTestWithFileExpiration.ReplicatedMetadataCanExpireFile
ybd --cxx-test integration-tests_compaction-test --gtest_filter CompactionTestWithFileExpiration.ReplicatedNoMetadataUsesTableTTL
ybd --cxx-test docdb_docdb-test --gtest_filter DocDBTests/DocDBTestWrapper.TestUpdateDocWriteBatchTTL/0

Reviewers: timur, rsami, sergei

Reviewed By: sergei

Subscribers: bogdan, anallan, ybase

Differential Revision: https://phabricator.dev.yugabyte.com/D13926
YBase features automation moved this from Backlog to Done Nov 23, 2021
TTL file expiry automation moved this from In progress to Done Nov 23, 2021
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@jmeehan16 jmeehan16

Labels
area/docdb YugabyteDB core features
Projects
TTL file expiry
Done
YBase features
  
Done
Milestone
No milestone
Development

No branches or pull requests
2 participants
@bmatican @jmeehan16