Add Lucene Partitioning Metadata #2390
Assignees
Comments
wshaib-apple
added a commit
to wshaib-apple/fdb-record-layer
that referenced
this issue
Dec 5, 2023
WIP of the Graceful Degradation Feature This initial update adds support for partitioning a logical Lucene index into multiple physical indexes spanning a time range. Each physical Lucene index (hereafter referred to as `partition`) will store document whose timestamp falls within the time range it covers. Partitions do *not* have overlapping time ranges. A document's timestamp, for the purposes of partitioning, is indicated by a record's field name in the index's options. The following options are added to the Lucene index: - `textDocumentPartitionTimestamp`: the name of field to use as timestamp for partitioning. If this option isn't specified, the index is not partitioned. - `textDocumentParititionHiWatermark`: high watermark size for a lucene partition, beyond which the partition would be split, or a new partition would be created - `textDocumentParititionLoWatermark`: low watermark size for a lucene partition, below which the partition would be a candidate for merge with another adjacent partition. The new partitions will be laid out in subspaces exactly like the current implementation, however they will be scoped as follows: - `indexSubspace.subspace(Tuple.from(groupingKey, 1, partitionId))` where `partitionId` is an integer starting at 0 Partition metadata will be stored in FDB as kv pairs at: - `indexSubspace.subspace(Tuple.from(groupingKey, 0)` the key is the `partitionId`, and the value is a protobuf message containing the `partitionId`, `from` timestamp, `to` timestamp, and document `count` This update *does not* (yet) support - splitting or merging partitions - index merging across multiple partitions - query continuation - optimized query based on timestamp predicate (if provided) This update sends all queries (including spellchecks) to the most recent partition
alecgrieser
pushed a commit
to alecgrieser/fdb-record-layer
that referenced
this issue
Dec 19, 2023
…ionDB#2391) * RESOLVES Add Lucene Partitioning Metadata FoundationDB#2390 WIP of the Graceful Degradation Feature This initial update adds support for partitioning a logical Lucene index into multiple physical indexes spanning a time range. Each physical Lucene index (hereafter referred to as `partition`) will store document whose timestamp falls within the time range it covers. Partitions do *not* have overlapping time ranges. A document's timestamp, for the purposes of partitioning, is indicated by a record's field name in the index's options. The following options are added to the Lucene index: - `textDocumentPartitionTimestamp`: the name of field to use as timestamp for partitioning. If this option isn't specified, the index is not partitioned. - `textDocumentParititionHiWatermark`: high watermark size for a lucene partition, beyond which the partition would be split, or a new partition would be created - `textDocumentParititionLoWatermark`: low watermark size for a lucene partition, below which the partition would be a candidate for merge with another adjacent partition. The new partitions will be laid out in subspaces exactly like the current implementation, however they will be scoped as follows: - `indexSubspace.subspace(Tuple.from(groupingKey, 1, partitionId))` where `partitionId` is an integer starting at 0 Partition metadata will be stored in FDB as kv pairs at: - `indexSubspace.subspace(Tuple.from(groupingKey, 0)` the key is the `partitionId`, and the value is a protobuf message containing the `partitionId`, `from` timestamp, `to` timestamp, and document `count` This update *does not* (yet) support - splitting or merging partitions - index merging across multiple partitions - query continuation - optimized query based on timestamp predicate (if provided) This update sends all queries (including spellchecks) to the most recent partition * - typo * - fix inverted condition * - javadoc fixes * - checkstyle fixes * - fix bug with partition subspace passed to `FDBDirectory` * - added support for nested fields (in joined/synthetic records) as the partition timestamp - added unit test for the above * Update FDBDirectoryManager.java previous git conflict resolution inadvertently kept `mapKey` final. * - merge conflict from upstream - remove sonar warnings (especially code complexity warning, by extracting some partitioning code from `LuceneIndexMaintainer.update()` and putting it in private functions. * - checkstyle * - addressed code review comments, especially change to partition metadata primary key which required an almost complete change of `LucenePartitioner` and removal of previously added code to `LuceneIndexMaintainer` * - missed a couple `@Nonnull` annotations * - added setting up initial `from` value in a new partition's metadata * - javadoc warning * - checkstyle fixes in tests Please enter the commit message for your changes. Lines starting * - sonar "smell" fixes * - more code review comments * - restored precision to doubles * Review comments: - changed index option name from `INDEX_PARTITION_HI_WATERMARK` to `INDEX_PARTITION_HIGH_WATERMARK` to be more consistent - changed the implementation of `getPartitioningTimestampValue()` to use a `KeyExpression` (which is also cached) - added release notes entry
alecgrieser
pushed a commit
to alecgrieser/fdb-record-layer
that referenced
this issue
Dec 19, 2023
…ionDB#2391) * RESOLVES Add Lucene Partitioning Metadata FoundationDB#2390 WIP of the Graceful Degradation Feature This initial update adds support for partitioning a logical Lucene index into multiple physical indexes spanning a time range. Each physical Lucene index (hereafter referred to as `partition`) will store document whose timestamp falls within the time range it covers. Partitions do *not* have overlapping time ranges. A document's timestamp, for the purposes of partitioning, is indicated by a record's field name in the index's options. The following options are added to the Lucene index: - `textDocumentPartitionTimestamp`: the name of field to use as timestamp for partitioning. If this option isn't specified, the index is not partitioned. - `textDocumentParititionHiWatermark`: high watermark size for a lucene partition, beyond which the partition would be split, or a new partition would be created - `textDocumentParititionLoWatermark`: low watermark size for a lucene partition, below which the partition would be a candidate for merge with another adjacent partition. The new partitions will be laid out in subspaces exactly like the current implementation, however they will be scoped as follows: - `indexSubspace.subspace(Tuple.from(groupingKey, 1, partitionId))` where `partitionId` is an integer starting at 0 Partition metadata will be stored in FDB as kv pairs at: - `indexSubspace.subspace(Tuple.from(groupingKey, 0)` the key is the `partitionId`, and the value is a protobuf message containing the `partitionId`, `from` timestamp, `to` timestamp, and document `count` This update *does not* (yet) support - splitting or merging partitions - index merging across multiple partitions - query continuation - optimized query based on timestamp predicate (if provided) This update sends all queries (including spellchecks) to the most recent partition * - typo * - fix inverted condition * - javadoc fixes * - checkstyle fixes * - fix bug with partition subspace passed to `FDBDirectory` * - added support for nested fields (in joined/synthetic records) as the partition timestamp - added unit test for the above * Update FDBDirectoryManager.java previous git conflict resolution inadvertently kept `mapKey` final. * - merge conflict from upstream - remove sonar warnings (especially code complexity warning, by extracting some partitioning code from `LuceneIndexMaintainer.update()` and putting it in private functions. * - checkstyle * - addressed code review comments, especially change to partition metadata primary key which required an almost complete change of `LucenePartitioner` and removal of previously added code to `LuceneIndexMaintainer` * - missed a couple `@Nonnull` annotations * - added setting up initial `from` value in a new partition's metadata * - javadoc warning * - checkstyle fixes in tests Please enter the commit message for your changes. Lines starting * - sonar "smell" fixes * - more code review comments * - restored precision to doubles * Review comments: - changed index option name from `INDEX_PARTITION_HI_WATERMARK` to `INDEX_PARTITION_HIGH_WATERMARK` to be more consistent - changed the implementation of `getPartitioningTimestampValue()` to use a `KeyExpression` (which is also cached) - added release notes entry
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Certain datasets may grow into hundreds of millions of records. A single Lucene index would not support such a large number of documents. A new feature creates multiple physical Lucene indexes (partitions) and spreads the documents across them, based on a timestamp associated with each document.
This first phase of this feature introduces the metadata needed to support a partitioned Lucene index.
The text was updated successfully, but these errors were encountered: