Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[DocDB] Add SST files zero-corruption check after they've been written #19691

Closed
1 task done
ttyusupov opened this issue Oct 26, 2023 · 0 comments
Closed
1 task done

[DocDB] Add SST files zero-corruption check after they've been written #19691

ttyusupov opened this issue Oct 26, 2023 · 0 comments
Assignees
@ttyusupov
Labels
2.14 Backport Required 2.16 Backport Required 2.18 Backport Required 2.20 Backport Required area/docdb YugabyteDB core features kind/enhancement This is an enhancement of an existing feature priority/highest Highest priority issue
Projects
YBase features

Comments

@ttyusupov
Copy link
Contributor

ttyusupov commented Oct 26, 2023 •
edited by jira bot

Jira Link: DB-8492

Description

We need to check SST files for configurable number of trailing zeros in order to detect potential specific corruption type after we've written the file and capture enough info to log in this case for further analysis.

Issue Type

kind/enhancement

Warning: Please confirm that this issue does not contain any sensitive information

  • I confirm this issue does not contain any sensitive information.
@ttyusupov ttyusupov added kind/enhancement This is an enhancement of an existing feature area/docdb YugabyteDB core features status/awaiting-triage Issue awaiting triage labels Oct 26, 2023
@ttyusupov ttyusupov self-assigned this Oct 26, 2023
@ttyusupov ttyusupov added this to Backlog in YBase features via automation Oct 26, 2023
@yugabyte-ci yugabyte-ci added the priority/medium Medium priority issue label Oct 26, 2023
@ttyusupov ttyusupov moved this from Backlog to In progress in YBase features Oct 26, 2023
@yugabyte-ci yugabyte-ci removed the status/awaiting-triage Issue awaiting triage label Oct 27, 2023
ttyusupov added a commit that referenced this issue Oct 30, 2023
@ttyusupov
[#19691] docdb: added SST files tail check for zero bytes
a18721c 
Summary:
Added support for `rocksdb_check_sst_file_tail_for_zeros` flag: size of just written SST data file tail to be checked for being zeros.
If this flag is set to positive value and we've detected that SST data file ends with at least `FLAGS_rocksdb_check_sst_file_tail_for_zeros` zeros we report an error and exact number of zero bytes at the end of the file.

Implemented the following code changes to support that behavior:
- Added rocksdb::WritableFile::filename() function
- rocksdb/util/file_reader_writer.cc: added CheckFileTailForZeros function
- rocksdb/db/builder.cc/h: added CheckSstTailForZeros function
- Used CheckSstTailForZeros during Flush and Compaction to verify for zeroed end of the data file just written
- Added more debug information to be logged when we open/close SST files if FLAGS_rocksdb_check_sst_file_tail_for_zeros > 0
- Added logging errors in case fallocate or ftruncate fails at callsites when their result was just ignored
- Added DBTest.SstTailZerosCheck* tests
- Added FileReaderWriterTest.CheckFileTailForZeros and included FileReaderWriterTest into CMakeLists.txt
Jira: DB-8492

Test Plan:
FileReaderWriterTest.CheckFileTailForZeros
DBTest.SstTailZerosCheck*

Reviewers: bogdan, arybochkin

Reviewed By: arybochkin

Subscribers: amitanand, bogdan, sergei, arybochkin, ybase

Tags: #jenkins-ready

Differential Revision: https://phorge.dev.yugabyte.com/D29532
@ttyusupov ttyusupov mentioned this issue Oct 30, 2023
Closed
1 task
ttyusupov added a commit that referenced this issue Nov 2, 2023
@ttyusupov
[BACKPORT 2.20][#19691] docdb: added SST files tail check for zero bytes
835a40f 
Summary:
Added support for `rocksdb_check_sst_file_tail_for_zeros` flag: size of just written SST data file tail to be checked for being zeros.
If this flag is set to positive value and we've detected that SST data file ends with at least `FLAGS_rocksdb_check_sst_file_tail_for_zeros` zeros we report an error and exact number of zero bytes at the end of the file.

Implemented the following code changes to support that behavior:
- Added rocksdb::WritableFile::filename() function
- rocksdb/util/file_reader_writer.cc: added CheckFileTailForZeros function
- rocksdb/db/builder.cc/h: added CheckSstTailForZeros function
- Used CheckSstTailForZeros during Flush and Compaction to verify for zeroed end of the data file just written
- Added more debug information to be logged when we open/close SST files if FLAGS_rocksdb_check_sst_file_tail_for_zeros > 0
- Added logging errors in case fallocate or ftruncate fails at callsites when their result was just ignored
- Added DBTest.SstTailZerosCheck* tests
- Added FileReaderWriterTest.CheckFileTailForZeros and included FileReaderWriterTest into CMakeLists.txt
Jira: DB-8492

Original commit: a18721c / D29532

Test Plan:
FileReaderWriterTest.CheckFileTailForZeros
DBTest.SstTailZerosCheck*

Reviewers: arybochkin

Reviewed By: arybochkin

Subscribers: ybase, arybochkin, sergei, bogdan, amitanand

Tags: #jenkins-ready

Differential Revision: https://phorge.dev.yugabyte.com/D29842
ttyusupov added a commit that referenced this issue Nov 2, 2023
@ttyusupov
[BACKPORT 2.18][#19691] docdb: added SST files tail check for zero bytes
a49b8e6 
Summary:
Added support for `rocksdb_check_sst_file_tail_for_zeros` flag: size of just written SST data file tail to be checked for being zeros.
If this flag is set to positive value and we've detected that SST data file ends with at least `FLAGS_rocksdb_check_sst_file_tail_for_zeros` zeros we report an error and exact number of zero bytes at the end of the file.

Implemented the following code changes to support that behavior:
- Added rocksdb::WritableFile::filename() function
- rocksdb/util/file_reader_writer.cc: added CheckFileTailForZeros function
- rocksdb/db/builder.cc/h: added CheckSstTailForZeros function
- Used CheckSstTailForZeros during Flush and Compaction to verify for zeroed end of the data file just written
- Added more debug information to be logged when we open/close SST files if FLAGS_rocksdb_check_sst_file_tail_for_zeros > 0
- Added logging errors in case fallocate or ftruncate fails at callsites when their result was just ignored
- Added DBTest.SstTailZerosCheck* tests
- Added FileReaderWriterTest.CheckFileTailForZeros and included FileReaderWriterTest into CMakeLists.txt
Jira: DB-8492

Original commit: a18721c / D29532

Test Plan:
FileReaderWriterTest.CheckFileTailForZeros
DBTest.SstTailZerosCheck*

Reviewers: arybochkin

Reviewed By: arybochkin

Subscribers: amitanand, bogdan, sergei, arybochkin, ybase

Tags: #jenkins-ready

Differential Revision: https://phorge.dev.yugabyte.com/D29849
ttyusupov added a commit that referenced this issue Nov 2, 2023
@ttyusupov
[BACKPORT 2.16][#19691] docdb: added SST files tail check for zero bytes
d186791 
Summary:
Added support for `rocksdb_check_sst_file_tail_for_zeros` flag: size of just written SST data file tail to be checked for being zeros.
If this flag is set to positive value and we've detected that SST data file ends with at least `FLAGS_rocksdb_check_sst_file_tail_for_zeros` zeros we report an error and exact number of zero bytes at the end of the file.

Implemented the following code changes to support that behavior:
- Added rocksdb::WritableFile::filename() function
- rocksdb/util/file_reader_writer.cc: added CheckFileTailForZeros function
- rocksdb/db/builder.cc/h: added CheckSstTailForZeros function
- Used CheckSstTailForZeros during Flush and Compaction to verify for zeroed end of the data file just written
- Added more debug information to be logged when we open/close SST files if FLAGS_rocksdb_check_sst_file_tail_for_zeros > 0
- Added logging errors in case fallocate or ftruncate fails at callsites when their result was just ignored
- Added DBTest.SstTailZerosCheck* tests
- Added FileReaderWriterTest.CheckFileTailForZeros and included FileReaderWriterTest into CMakeLists.txt
Jira: DB-8492

Original commit: a18721c / D29532

Test Plan:
FileReaderWriterTest.CheckFileTailForZeros
DBTest.SstTailZerosCheck*

Reviewers: arybochkin

Reviewed By: arybochkin

Subscribers: ybase, arybochkin, sergei, bogdan, amitanand

Tags: #jenkins-ready

Differential Revision: https://phorge.dev.yugabyte.com/D29871
ttyusupov added a commit that referenced this issue Nov 2, 2023
@ttyusupov
[BACKPORT 2.14][#19691] docdb: added SST files tail check for zero bytes
9242eeb 
Summary:
Added support for `rocksdb_check_sst_file_tail_for_zeros` flag: size of just written SST data file tail to be checked for being zeros.
If this flag is set to positive value and we've detected that SST data file ends with at least `FLAGS_rocksdb_check_sst_file_tail_for_zeros` zeros we report an error and exact number of zero bytes at the end of the file.

Implemented the following code changes to support that behavior:
- Added rocksdb::WritableFile::filename() function
- rocksdb/util/file_reader_writer.cc: added CheckFileTailForZeros function
- rocksdb/db/builder.cc/h: added CheckSstTailForZeros function
- Used CheckSstTailForZeros during Flush and Compaction to verify for zeroed end of the data file just written
- Added more debug information to be logged when we open/close SST files if FLAGS_rocksdb_check_sst_file_tail_for_zeros > 0
- Added logging errors in case fallocate or ftruncate fails at callsites when their result was just ignored
- Added DBTest.SstTailZerosCheck* tests
- Added FileReaderWriterTest.CheckFileTailForZeros and included FileReaderWriterTest into CMakeLists.txt
Jira: DB-8492

Original commit: a18721c / D29532

Test Plan:
FileReaderWriterTest.CheckFileTailForZeros
DBTest.SstTailZerosCheck*

Reviewers: arybochkin

Reviewed By: arybochkin

Subscribers: amitanand, bogdan, sergei, arybochkin, ybase

Tags: #jenkins-ready

Differential Revision: https://phorge.dev.yugabyte.com/D29874
YBase features automation moved this from In progress to Done Nov 2, 2023
@yugabyte-ci yugabyte-ci reopened this Nov 3, 2023
YBase features automation moved this from Done to In progress Nov 3, 2023
@yugabyte-ci yugabyte-ci assigned arybochkin and unassigned ttyusupov Nov 3, 2023
arybochkin pushed a commit that referenced this issue Nov 3, 2023
@ttyusupov @arybochkin
[BACKPORT 2.14.10][#19691] docdb: added SST files tail check for zero…
a717259 
… bytes

Summary:
Added support for `rocksdb_check_sst_file_tail_for_zeros` flag: size of just written SST data file tail to be checked for being zeros.
If this flag is set to positive value and we've detected that SST data file ends with at least `FLAGS_rocksdb_check_sst_file_tail_for_zeros` zeros we report an error and exact number of zero bytes at the end of the file.

Implemented the following code changes to support that behavior:
- Added rocksdb::WritableFile::filename() function
- rocksdb/util/file_reader_writer.cc: added CheckFileTailForZeros function
- rocksdb/db/builder.cc/h: added CheckSstTailForZeros function
- Used CheckSstTailForZeros during Flush and Compaction to verify for zeroed end of the data file just written
- Added more debug information to be logged when we open/close SST files if FLAGS_rocksdb_check_sst_file_tail_for_zeros > 0
- Added logging errors in case fallocate or ftruncate fails at callsites when their result was just ignored
- Added DBTest.SstTailZerosCheck* tests
- Added FileReaderWriterTest.CheckFileTailForZeros and included FileReaderWriterTest into CMakeLists.txt
Jira: DB-8492

Original commit: a18721c / D29532
Backported from 2.14 commit: 9242eeb / D29874

Test Plan:
Jenkins: urgent

FileReaderWriterTest.CheckFileTailForZeros
DBTest.SstTailZerosCheck*

Reviewers: bogdan, timur, rthallam

Reviewed By: bogdan

Subscribers: ybase, sergei, amitanand

Tags: #jenkins-ready

Differential Revision: https://phorge.dev.yugabyte.com/D29950
YBase features automation moved this from In progress to Done Nov 3, 2023
@yugabyte-ci yugabyte-ci assigned ttyusupov and unassigned arybochkin Nov 4, 2023
@yugabyte-ci yugabyte-ci removed the priority/medium Medium priority issue label Nov 15, 2023
@yugabyte-ci yugabyte-ci added the priority/highest Highest priority issue label Nov 15, 2023
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@ttyusupov ttyusupov

Labels
2.14 Backport Required 2.16 Backport Required 2.18 Backport Required 2.20 Backport Required area/docdb YugabyteDB core features kind/enhancement This is an enhancement of an existing feature priority/highest Highest priority issue
Projects
YBase features
  
Done
Milestone
No milestone
Development

No branches or pull requests
4 participants
@ttyusupov @rthallamko3 @yugabyte-ci @arybochkin