You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
This commit was created on GitHub.com and signed with GitHub’s verified signature.
New Features
Added FullExternalTableFactory and related full-mode external table APIs that preserve RocksDB internal keys, enabling external tables to support nonzero sequence numbers, point and single deletions, merge operands, blob indexes, wide-column entities, snapshots, live DB writes, and multi-level compactions.
Public API Changes
Added experimental coroutine capability interfaces for basic and full ExternalTableReader implementations. Coroutine-capable external readers can initiate file reads through FSRandomAccessFile::SubmitReadAsync().
ExternalTableFactory::NewTableReader() now receives an already-opened FSRandomAccessFile and its file size instead of requiring implementations to reopen file_path. The supplied file preserves RocksDB's direct-I/O alignment handling, rate limiting, read statistics and listener notifications, prefetching, and synchronous, asynchronous, and batched reads.
Added PutRangeDeletionBlock(), GetRangeDeletionBlock(), and IsDeleteRangeSupported() to the external table interfaces so formats can opt in to range deletions using RocksDB's raw range-deletion metadata block.
Opaque PreparedFileInfo metadata can now be serialized to a checksummed representation with SerializePreparedFileInfo() and restored with DeserializePreparedFileInfo() for reuse across processes.
Removed the deprecated IngestExternalFileOptions::write_global_seqno option and its C, Java, and ldb APIs, TableProperties::external_sst_file_global_seqno_offset, and the file_offset output from ExternalTableReader::GetPropertiesBlock(). Existing external SST files whose global sequence number was rewritten without updating the properties-block checksum remain readable.
TablePropertiesNames has no apparent use in the public API so has been deprecated there.
TableProperties::creation_time is renamed to oldest_ancestor_time, which better describes its meaning, along with TablePropertiesNames::kCreationTime to kOldestAncestorTime. The misspelled SstFileMetaData::oldest_ancester_time is renamed to oldest_ancestor_time. The old names remain available as deprecated aliases and will be removed in a future release.
Added WriteBufferFlushPolicy and WriteBufferManager policy selection for flushing the oldest or largest column family, including configurable batched flushing of the largest column families across DBs sharing a manager, with C and Java bindings.
Behavior Changes
When ExternalTableBuilderBase::PutPropertiesBlock() returns NotSupported, RocksDB now overrides num_entries, raw_key_size, and raw_value_size from ExternalTableBuilderBase::GetTableProperties() with the logical totals tracked by the external-table adapter. Format-specific properties such as data_size remain unchanged.
The FSWritableFile passed to ExternalTableFactory::NewTableBuilder() now reports logical file size including RocksDB-buffered data and exposes the underlying file's direct-I/O mode and required buffer alignment.
Iterator::Refresh() now returns NotSupported on iterators from a secondary DB's DB::NewIterators() only, because those iterators are pinned to one cross-column-family read view. Iterators from DB::NewIterator() are unaffected; to advance a batch, call DB::NewIterators() again after TryCatchUpWithPrimary().
SstFileReader now reports standard SST read statistics and file I/O listener notifications, and honors the configured read rate limiter.
Bug Fixes
Fixed full recovery from retryable background errors to atomically advance WAL replay boundaries for all column families, preventing partial recovery of multi-column-family write batches after a failed recovery flush.
Fixed RocksDB builds on big-endian architectures such as s390x by disabling the optional persistent fault-injection log, whose binary format is little-endian-only.
Fixed DB::ClipColumnFamily() deleting a non-L0 SST file whose largest key equals begin_key, which lost keys at the start of the kept range.
Fixed swapped oldest_key_time and newest_key_time values in the table_file_creation event log entry.
Fixed ExternalTableReader::VerifyChecksum() results not being honored by the external table adapter.
Fixed a bug in manifest handling that could lead to a hang or memory corruption in the case of LSM sanity checks being fatally violated (exceptional; see force_consistency_checks).
Fixed MultiGet() with ReadOptions::read_tier == kBlockCacheTier reading data blocks from disk when allow_mmap_reads is false. Keys whose data block is not in the block cache now return Status::Incomplete(), the same as Get().
Fixed trace replay (DB::NewDefaultReplayer(), db_bench replay) failing every read record (Get, MultiGet, iterator seek) on the default column family with Corruption: Invalid Column Family ID when the default column family handle was not passed, which is always the case in db_bench with --num_column_families=1.
Fixed multi-threaded Replayer::Replay() ignoring errors from executing trace records and returning OK. It now stops and returns the error, including Status::NotSupported() from executing a record, as single-threaded replay does.
Fixed secondary DB NewIterators() returning iterators from different database states or blocking for TryCatchUpWithPrimary() by serving the last completed catch-up view.
Performance Improvements
Single Get coroutine reads using non-mmap, non-direct I/O on a kFSBuffer-capable FileSystem now use FSRandomAccessFile::SubmitReadAsync() instead of falling back to synchronous Read(). When rate-limited, these reads acquire their full token budget before submission to preserve a single filesystem-owned buffer, which can make physical I/O burstier than reads using caller-provided buffers.