From 800cfae9872f6ec58224e720d8276b1ca1ac042a Mon Sep 17 00:00:00 2001 From: Peter Dillinger Date: Sat, 20 Jan 2024 08:38:20 -0800 Subject: [PATCH] Start 9.0.0 release (#12256) Summary: with release notes for 8.11.fb, format_compatible test update, and version.h update. Pull Request resolved: https://github.com/facebook/rocksdb/pull/12256 Test Plan: CI Reviewed By: cbi42 Differential Revision: D52926051 Pulled By: pdillinger fbshipit-source-id: adcf7119b065758599e904c16cbdf1d28811e0b4 --- HISTORY.md | 24 +++++++++++++++++++ include/rocksdb/version.h | 4 ++-- tools/check_format_compatible.sh | 2 +- .../blob_file_write_micros.md | 1 - ...hange_behavior_of_compact_to_next_level.md | 1 - .../more_debt_based_speedup.md | 1 - .../pri_file_marked_for_compaction | 1 - .../bug_fixes/auto_readahead_size.md | 1 - .../bug_fixes/backup_verify_gap.md | 1 - unreleased_history/bug_fixes/blob_tickers.md | 1 - unreleased_history/bug_fixes/cfd_leak.md | 1 - .../bug_fixes/fix_fileprefetchbuffer_mmap.md | 1 - .../bug_fixes/seqnototimemapping.md | 1 - .../bug_fixes/unreferenced_blob_files.md | 1 - ...rite_micros_file_write_stats_break_down.md | 1 - .../comp_cache_compression_level.md | 1 - .../deprecate_unsafe_or_outdated_options.md | 1 - .../expose_options_ttl_via_c_api.md | 1 - 18 files changed, 27 insertions(+), 18 deletions(-) delete mode 100644 unreleased_history/behavior_changes/blob_file_write_micros.md delete mode 100644 unreleased_history/behavior_changes/change_behavior_of_compact_to_next_level.md delete mode 100644 unreleased_history/behavior_changes/more_debt_based_speedup.md delete mode 100644 unreleased_history/behavior_changes/pri_file_marked_for_compaction delete mode 100644 unreleased_history/bug_fixes/auto_readahead_size.md delete mode 100644 unreleased_history/bug_fixes/backup_verify_gap.md delete mode 100644 unreleased_history/bug_fixes/blob_tickers.md delete mode 100644 unreleased_history/bug_fixes/cfd_leak.md delete mode 100644 unreleased_history/bug_fixes/fix_fileprefetchbuffer_mmap.md delete mode 100644 unreleased_history/bug_fixes/seqnototimemapping.md delete mode 100644 unreleased_history/bug_fixes/unreferenced_blob_files.md delete mode 100644 unreleased_history/new_features/sst_write_micros_file_write_stats_break_down.md delete mode 100644 unreleased_history/public_api_changes/comp_cache_compression_level.md delete mode 100644 unreleased_history/public_api_changes/deprecate_unsafe_or_outdated_options.md delete mode 100644 unreleased_history/public_api_changes/expose_options_ttl_via_c_api.md diff --git a/HISTORY.md b/HISTORY.md index 0d7c284b2f0..37f42892fe8 100644 --- a/HISTORY.md +++ b/HISTORY.md @@ -1,6 +1,30 @@ # Rocksdb Change Log > NOTE: Entries for next release do not go here. Follow instructions in `unreleased_history/README.txt` +## 8.11.0 (01/19/2024) +### New Features +* Add new statistics: `rocksdb.sst.write.micros` measures time of each write to SST file; `rocksdb.file.write.{flush|compaction|db.open}.micros` measure time of each write to SST table (currently only block-based table format) and blob file for flush, compaction and db open. + +### Public API Changes +* Add CompressionOptions to the CompressedSecondaryCacheOptions structure to allow users to specify library specific options when creating the compressed secondary cache. +* Deprecated several options: `level_compaction_dynamic_file_size`, `ignore_max_compaction_bytes_for_input`, `check_flush_compaction_key_order`, `flush_verify_memtable_count`, `compaction_verify_record_count`, `fail_if_options_file_error`, and `enforce_single_del_contracts` +* Exposed options ttl via c api. + +### Behavior Changes +* `rocksdb.blobdb.blob.file.write.micros` expands to also measure time writing the header and footer. Therefore the COUNT may be higher and values may be smaller than before. For stacked BlobDB, it no longer measures the time of explictly flushing blob file. +* Files will be compacted to the next level if the data age exceeds periodic_compaction_seconds except for the last level. +* Reduced the compaction debt ratio trigger for scheduling parallel compactions +* For leveled compaction with default compaction pri (kMinOverlappingRatio), files marked for compaction will be prioritized over files not marked when picking a file from a level for compaction. + +### Bug Fixes +* Fix bug in auto_readahead_size that combined with IndexType::kBinarySearchWithFirstKey + fails or iterator lands at a wrong key +* Fixed some cases in which DB file corruption was detected but ignored on creating a backup with BackupEngine. +* Fix bugs where `rocksdb.blobdb.blob.file.synced` includes blob files failed to get synced and `rocksdb.blobdb.blob.file.bytes.written` includes blob bytes failed to get written. +* Fixed a possible memory leak or crash on a failure (such as I/O error) in automatic atomic flush of multiple column families. +* Fixed some cases of in-memory data corruption using mmap reads with `BackupEngine`, `sst_dump`, or `ldb`. +* Fixed issues with experimental `preclude_last_level_data_seconds` option that could interfere with expected data tiering. +* Fixed the handling of the edge case when all existing blob files become unreferenced. Such files are now correctly deleted. + ## 8.10.0 (12/15/2023) ### New Features * Provide support for async_io to trim readahead_size by doing block cache lookup diff --git a/include/rocksdb/version.h b/include/rocksdb/version.h index 4ff6ab51fcc..88af8ce2101 100644 --- a/include/rocksdb/version.h +++ b/include/rocksdb/version.h @@ -11,8 +11,8 @@ // NOTE: in 'main' development branch, this should be the *next* // minor or major version number planned for release. -#define ROCKSDB_MAJOR 8 -#define ROCKSDB_MINOR 11 +#define ROCKSDB_MAJOR 9 +#define ROCKSDB_MINOR 0 #define ROCKSDB_PATCH 0 // Do not use these. We made the mistake of declaring macros starting with diff --git a/tools/check_format_compatible.sh b/tools/check_format_compatible.sh index 21c17feae04..e4343a626c5 100755 --- a/tools/check_format_compatible.sh +++ b/tools/check_format_compatible.sh @@ -125,7 +125,7 @@ EOF # To check for DB forward compatibility with loading options (old version # reading data from new), as well as backward compatibility -declare -a db_forward_with_options_refs=("6.27.fb" "6.28.fb" "6.29.fb" "7.0.fb" "7.1.fb" "7.2.fb" "7.3.fb" "7.4.fb" "7.5.fb" "7.6.fb" "7.7.fb" "7.8.fb" "7.9.fb" "7.10.fb" "8.0.fb" "8.1.fb" "8.2.fb" "8.3.fb" "8.4.fb" "8.5.fb" "8.6.fb" "8.7.fb" "8.8.fb" "8.9.fb" "8.10.fb") +declare -a db_forward_with_options_refs=("6.27.fb" "6.28.fb" "6.29.fb" "7.0.fb" "7.1.fb" "7.2.fb" "7.3.fb" "7.4.fb" "7.5.fb" "7.6.fb" "7.7.fb" "7.8.fb" "7.9.fb" "7.10.fb" "8.0.fb" "8.1.fb" "8.2.fb" "8.3.fb" "8.4.fb" "8.5.fb" "8.6.fb" "8.7.fb" "8.8.fb" "8.9.fb" "8.10.fb" "8.11.fb") # To check for DB forward compatibility without loading options (in addition # to the "with loading options" set), as well as backward compatibility declare -a db_forward_no_options_refs=() # N/A at the moment diff --git a/unreleased_history/behavior_changes/blob_file_write_micros.md b/unreleased_history/behavior_changes/blob_file_write_micros.md deleted file mode 100644 index aceb0592008..00000000000 --- a/unreleased_history/behavior_changes/blob_file_write_micros.md +++ /dev/null @@ -1 +0,0 @@ -`rocksdb.blobdb.blob.file.write.micros` expands to also measure time writing the header and footer. Therefore the COUNT may be higher and values may be smaller than before. For stacked BlobDB, it no longer measures the time of explictly flushing blob file. diff --git a/unreleased_history/behavior_changes/change_behavior_of_compact_to_next_level.md b/unreleased_history/behavior_changes/change_behavior_of_compact_to_next_level.md deleted file mode 100644 index 8faf130d4fe..00000000000 --- a/unreleased_history/behavior_changes/change_behavior_of_compact_to_next_level.md +++ /dev/null @@ -1 +0,0 @@ -Files will be compacted to the next level if the data age exceeds periodic_compaction_seconds except for the last level. \ No newline at end of file diff --git a/unreleased_history/behavior_changes/more_debt_based_speedup.md b/unreleased_history/behavior_changes/more_debt_based_speedup.md deleted file mode 100644 index c10f0d7b361..00000000000 --- a/unreleased_history/behavior_changes/more_debt_based_speedup.md +++ /dev/null @@ -1 +0,0 @@ -Reduced the compaction debt ratio trigger for scheduling parallel compactions diff --git a/unreleased_history/behavior_changes/pri_file_marked_for_compaction b/unreleased_history/behavior_changes/pri_file_marked_for_compaction deleted file mode 100644 index e0720160294..00000000000 --- a/unreleased_history/behavior_changes/pri_file_marked_for_compaction +++ /dev/null @@ -1 +0,0 @@ -* For leveled compaction with default compaction pri (kMinOverlappingRatio), files marked for compaction will be prioritized over files not marked when picking a file from a level for compaction. \ No newline at end of file diff --git a/unreleased_history/bug_fixes/auto_readahead_size.md b/unreleased_history/bug_fixes/auto_readahead_size.md deleted file mode 100644 index 7591dc53614..00000000000 --- a/unreleased_history/bug_fixes/auto_readahead_size.md +++ /dev/null @@ -1 +0,0 @@ -Fix bug in auto_readahead_size that combined with IndexType::kBinarySearchWithFirstKey + fails or iterator lands at a wrong key diff --git a/unreleased_history/bug_fixes/backup_verify_gap.md b/unreleased_history/bug_fixes/backup_verify_gap.md deleted file mode 100644 index 075dec6865f..00000000000 --- a/unreleased_history/bug_fixes/backup_verify_gap.md +++ /dev/null @@ -1 +0,0 @@ -Fixed some cases in which DB file corruption was detected but ignored on creating a backup with BackupEngine. diff --git a/unreleased_history/bug_fixes/blob_tickers.md b/unreleased_history/bug_fixes/blob_tickers.md deleted file mode 100644 index 6858959443e..00000000000 --- a/unreleased_history/bug_fixes/blob_tickers.md +++ /dev/null @@ -1 +0,0 @@ -Fix bugs where `rocksdb.blobdb.blob.file.synced` includes blob files failed to get synced and `rocksdb.blobdb.blob.file.bytes.written` includes blob bytes failed to get written. diff --git a/unreleased_history/bug_fixes/cfd_leak.md b/unreleased_history/bug_fixes/cfd_leak.md deleted file mode 100644 index e9586ec8308..00000000000 --- a/unreleased_history/bug_fixes/cfd_leak.md +++ /dev/null @@ -1 +0,0 @@ -Fixed a possible memory leak or crash on a failure (such as I/O error) in automatic atomic flush of multiple column families. diff --git a/unreleased_history/bug_fixes/fix_fileprefetchbuffer_mmap.md b/unreleased_history/bug_fixes/fix_fileprefetchbuffer_mmap.md deleted file mode 100644 index e5cd8be3562..00000000000 --- a/unreleased_history/bug_fixes/fix_fileprefetchbuffer_mmap.md +++ /dev/null @@ -1 +0,0 @@ -Fixed some cases of in-memory data corruption using mmap reads with `BackupEngine`, `sst_dump`, or `ldb`. diff --git a/unreleased_history/bug_fixes/seqnototimemapping.md b/unreleased_history/bug_fixes/seqnototimemapping.md deleted file mode 100644 index b1bd5cc4e96..00000000000 --- a/unreleased_history/bug_fixes/seqnototimemapping.md +++ /dev/null @@ -1 +0,0 @@ -Fixed issues with experimental `preclude_last_level_data_seconds` option that could interfere with expected data tiering. diff --git a/unreleased_history/bug_fixes/unreferenced_blob_files.md b/unreleased_history/bug_fixes/unreferenced_blob_files.md deleted file mode 100644 index 2f680372ebe..00000000000 --- a/unreleased_history/bug_fixes/unreferenced_blob_files.md +++ /dev/null @@ -1 +0,0 @@ -Fixed the handling of the edge case when all existing blob files become unreferenced. Such files are now correctly deleted. diff --git a/unreleased_history/new_features/sst_write_micros_file_write_stats_break_down.md b/unreleased_history/new_features/sst_write_micros_file_write_stats_break_down.md deleted file mode 100644 index 6e869986754..00000000000 --- a/unreleased_history/new_features/sst_write_micros_file_write_stats_break_down.md +++ /dev/null @@ -1 +0,0 @@ -Add new statistics: `rocksdb.sst.write.micros` measures time of each write to SST file; `rocksdb.file.write.{flush|compaction|db.open}.micros` measure time of each write to SST table (currently only block-based table format) and blob file for flush, compaction and db open. diff --git a/unreleased_history/public_api_changes/comp_cache_compression_level.md b/unreleased_history/public_api_changes/comp_cache_compression_level.md deleted file mode 100644 index 29843c95243..00000000000 --- a/unreleased_history/public_api_changes/comp_cache_compression_level.md +++ /dev/null @@ -1 +0,0 @@ -Add CompressionOptions to the CompressedSecondaryCacheOptions structure to allow users to specify library specific options when creating the compressed secondary cache. diff --git a/unreleased_history/public_api_changes/deprecate_unsafe_or_outdated_options.md b/unreleased_history/public_api_changes/deprecate_unsafe_or_outdated_options.md deleted file mode 100644 index 93e359462df..00000000000 --- a/unreleased_history/public_api_changes/deprecate_unsafe_or_outdated_options.md +++ /dev/null @@ -1 +0,0 @@ -Deprecated several options: `level_compaction_dynamic_file_size`, `ignore_max_compaction_bytes_for_input`, `check_flush_compaction_key_order`, `flush_verify_memtable_count`, `compaction_verify_record_count`, `fail_if_options_file_error`, and `enforce_single_del_contracts` diff --git a/unreleased_history/public_api_changes/expose_options_ttl_via_c_api.md b/unreleased_history/public_api_changes/expose_options_ttl_via_c_api.md deleted file mode 100644 index 5576f93b3c2..00000000000 --- a/unreleased_history/public_api_changes/expose_options_ttl_via_c_api.md +++ /dev/null @@ -1 +0,0 @@ -Exposed options ttl via c api. \ No newline at end of file