STAR-1812 Stop collection guardrails spamming logs#588
Merged
mike-tr-adamson merged 2 commits intods-trunkfrom Nov 28, 2022
Merged
STAR-1812 Stop collection guardrails spamming logs#588mike-tr-adamson merged 2 commits intods-trunkfrom
mike-tr-adamson merged 2 commits intods-trunkfrom
Conversation
mfleming
approved these changes
Nov 24, 2022
|
Kudos, SonarCloud Quality Gate passed! |
mfleming
pushed a commit
that referenced
this pull request
Jul 10, 2023
* STAR-1812 Stop collection guardrails spamming logs * Fix failing tests (cherry picked from commit e953dc6)
djatnieks
pushed a commit
that referenced
this pull request
Jul 24, 2023
* STAR-1812 Stop collection guardrails spamming logs * Fix failing tests (cherry picked from commit e953dc6)
djatnieks
pushed a commit
that referenced
this pull request
Aug 22, 2023
djatnieks
pushed a commit
that referenced
this pull request
Sep 12, 2023
djatnieks
pushed a commit
that referenced
this pull request
Feb 9, 2024
djatnieks
pushed a commit
that referenced
this pull request
Mar 29, 2024
* STAR-1812 Stop collection guardrails spamming logs * Fix failing tests (cherry picked from commit e953dc6) (cherry picked from commit 8aa875b) (cherry picked from commit 0d25080) STAR-1812 Fix guardrail tests that broke after row info was removed from logging; In GuardrailCollectionSizeOnSSTableWriteTest.testMapSizeAfterCompaction make sure insert/update statements are unique to get expected logging; Move testGuardrailRespectsMinimumNotificationInterval from distributed to unit test class GuardrailCollectionSizeTest to avoid effects of NoSpamLogger STAR-1812 Fix GuardrailItemsPerCollectionOnSSTableWriteTest; Update expected warn and fail messages to not expect "in row" Split some tests to avoid NoSpamLogger from suppressing expected messages
djatnieks
pushed a commit
that referenced
this pull request
Apr 1, 2024
* STAR-1812 Stop collection guardrails spamming logs * Fix failing tests (cherry picked from commit e953dc6) (cherry picked from commit 8aa875b) (cherry picked from commit 0d25080) STAR-1812 Fix guardrail tests that broke after row info was removed from logging; In GuardrailCollectionSizeOnSSTableWriteTest.testMapSizeAfterCompaction make sure insert/update statements are unique to get expected logging; Move testGuardrailRespectsMinimumNotificationInterval from distributed to unit test class GuardrailCollectionSizeTest to avoid effects of NoSpamLogger STAR-1812 Fix GuardrailItemsPerCollectionOnSSTableWriteTest; Update expected warn and fail messages to not expect "in row" Split some tests to avoid NoSpamLogger from suppressing expected messages
djatnieks
pushed a commit
that referenced
this pull request
Apr 16, 2024
* STAR-1812 Stop collection guardrails spamming logs * Fix failing tests (cherry picked from commit e953dc6) (cherry picked from commit 8aa875b) (cherry picked from commit 0d25080) STAR-1812 Fix guardrail tests that broke after row info was removed from logging; In GuardrailCollectionSizeOnSSTableWriteTest.testMapSizeAfterCompaction make sure insert/update statements are unique to get expected logging; Move testGuardrailRespectsMinimumNotificationInterval from distributed to unit test class GuardrailCollectionSizeTest to avoid effects of NoSpamLogger STAR-1812 Fix GuardrailItemsPerCollectionOnSSTableWriteTest; Update expected warn and fail messages to not expect "in row" Split some tests to avoid NoSpamLogger from suppressing expected messages
djatnieks
pushed a commit
that referenced
this pull request
May 17, 2024
* STAR-1812 Stop collection guardrails spamming logs * Fix failing tests (cherry picked from commit e953dc6) (cherry picked from commit 8aa875b) (cherry picked from commit 0d25080) STAR-1812 Fix guardrail tests that broke after row info was removed from logging; In GuardrailCollectionSizeOnSSTableWriteTest.testMapSizeAfterCompaction make sure insert/update statements are unique to get expected logging; Move testGuardrailRespectsMinimumNotificationInterval from distributed to unit test class GuardrailCollectionSizeTest to avoid effects of NoSpamLogger STAR-1812 Fix GuardrailItemsPerCollectionOnSSTableWriteTest; Update expected warn and fail messages to not expect "in row" Split some tests to avoid NoSpamLogger from suppressing expected messages
jacek-lewandowski
pushed a commit
that referenced
this pull request
Jul 17, 2024
* STAR-1812 Stop collection guardrails spamming logs * Fix failing tests (cherry picked from commit e953dc6) (cherry picked from commit 8aa875b) (cherry picked from commit 0d25080) STAR-1812 Fix guardrail tests that broke after row info was removed from logging; In GuardrailCollectionSizeOnSSTableWriteTest.testMapSizeAfterCompaction make sure insert/update statements are unique to get expected logging; Move testGuardrailRespectsMinimumNotificationInterval from distributed to unit test class GuardrailCollectionSizeTest to avoid effects of NoSpamLogger STAR-1812 Fix GuardrailItemsPerCollectionOnSSTableWriteTest; Update expected warn and fail messages to not expect "in row" Split some tests to avoid NoSpamLogger from suppressing expected messages
djatnieks
pushed a commit
that referenced
this pull request
Jan 30, 2025
* STAR-1812 Stop collection guardrails spamming logs * Fix failing tests (cherry picked from commit e953dc6) (cherry picked from commit 8aa875b) (cherry picked from commit 0d25080) STAR-1812 Fix guardrail tests that broke after row info was removed from logging; In GuardrailCollectionSizeOnSSTableWriteTest.testMapSizeAfterCompaction make sure insert/update statements are unique to get expected logging; Move testGuardrailRespectsMinimumNotificationInterval from distributed to unit test class GuardrailCollectionSizeTest to avoid effects of NoSpamLogger STAR-1812 Fix GuardrailItemsPerCollectionOnSSTableWriteTest; Update expected warn and fail messages to not expect "in row" Split some tests to avoid NoSpamLogger from suppressing expected messages
djatnieks
pushed a commit
that referenced
this pull request
May 18, 2025
* STAR-1812 Stop collection guardrails spamming logs * Fix failing tests (cherry picked from commit e953dc6) (cherry picked from commit 8aa875b) (cherry picked from commit 0d25080) STAR-1812 Fix guardrail tests that broke after row info was removed from logging; In GuardrailCollectionSizeOnSSTableWriteTest.testMapSizeAfterCompaction make sure insert/update statements are unique to get expected logging; Move testGuardrailRespectsMinimumNotificationInterval from distributed to unit test class GuardrailCollectionSizeTest to avoid effects of NoSpamLogger STAR-1812 Fix GuardrailItemsPerCollectionOnSSTableWriteTest; Update expected warn and fail messages to not expect "in row" Split some tests to avoid NoSpamLogger from suppressing expected messages
michaelsembwever
pushed a commit
that referenced
this pull request
Feb 6, 2026
* STAR-1812 Stop collection guardrails spamming logs * Fix failing tests (cherry picked from commit e953dc6) (cherry picked from commit 8aa875b) (cherry picked from commit 0d25080) STAR-1812 Fix guardrail tests that broke after row info was removed from logging; In GuardrailCollectionSizeOnSSTableWriteTest.testMapSizeAfterCompaction make sure insert/update statements are unique to get expected logging; Move testGuardrailRespectsMinimumNotificationInterval from distributed to unit test class GuardrailCollectionSizeTest to avoid effects of NoSpamLogger STAR-1812 Fix GuardrailItemsPerCollectionOnSSTableWriteTest; Update expected warn and fail messages to not expect "in row" Split some tests to avoid NoSpamLogger from suppressing expected messages
michaelsembwever
pushed a commit
that referenced
this pull request
Feb 10, 2026
* STAR-1812 Stop collection guardrails spamming logs * Fix failing tests (cherry picked from commit e953dc6) (cherry picked from commit 8aa875b) (cherry picked from commit 0d25080) STAR-1812 Fix guardrail tests that broke after row info was removed from logging; In GuardrailCollectionSizeOnSSTableWriteTest.testMapSizeAfterCompaction make sure insert/update statements are unique to get expected logging; Move testGuardrailRespectsMinimumNotificationInterval from distributed to unit test class GuardrailCollectionSizeTest to avoid effects of NoSpamLogger STAR-1812 Fix GuardrailItemsPerCollectionOnSSTableWriteTest; Update expected warn and fail messages to not expect "in row" Split some tests to avoid NoSpamLogger from suppressing expected messages (Rebase of commit d2af13f)
michaelsembwever
pushed a commit
that referenced
this pull request
Feb 11, 2026
* STAR-1812 Stop collection guardrails spamming logs * Fix failing tests (cherry picked from commit e953dc6) (cherry picked from commit 8aa875b) (cherry picked from commit 0d25080) STAR-1812 Fix guardrail tests that broke after row info was removed from logging; In GuardrailCollectionSizeOnSSTableWriteTest.testMapSizeAfterCompaction make sure insert/update statements are unique to get expected logging; Move testGuardrailRespectsMinimumNotificationInterval from distributed to unit test class GuardrailCollectionSizeTest to avoid effects of NoSpamLogger STAR-1812 Fix GuardrailItemsPerCollectionOnSSTableWriteTest; Update expected warn and fail messages to not expect "in row" Split some tests to avoid NoSpamLogger from suppressing expected messages (Rebase of commit d2af13f)
michaelsembwever
pushed a commit
that referenced
this pull request
Feb 12, 2026
* STAR-1812 Stop collection guardrails spamming logs * Fix failing tests (cherry picked from commit e953dc6) (cherry picked from commit 8aa875b) (cherry picked from commit 0d25080) STAR-1812 Fix guardrail tests that broke after row info was removed from logging; In GuardrailCollectionSizeOnSSTableWriteTest.testMapSizeAfterCompaction make sure insert/update statements are unique to get expected logging; Move testGuardrailRespectsMinimumNotificationInterval from distributed to unit test class GuardrailCollectionSizeTest to avoid effects of NoSpamLogger STAR-1812 Fix GuardrailItemsPerCollectionOnSSTableWriteTest; Update expected warn and fail messages to not expect "in row" Split some tests to avoid NoSpamLogger from suppressing expected messages (Rebase of commit d2af13f)
michaelsembwever
pushed a commit
that referenced
this pull request
Feb 14, 2026
* STAR-1812 Stop collection guardrails spamming logs * Fix failing tests (cherry picked from commit e953dc6) (cherry picked from commit 8aa875b) (cherry picked from commit 0d25080) STAR-1812 Fix guardrail tests that broke after row info was removed from logging; In GuardrailCollectionSizeOnSSTableWriteTest.testMapSizeAfterCompaction make sure insert/update statements are unique to get expected logging; Move testGuardrailRespectsMinimumNotificationInterval from distributed to unit test class GuardrailCollectionSizeTest to avoid effects of NoSpamLogger STAR-1812 Fix GuardrailItemsPerCollectionOnSSTableWriteTest; Update expected warn and fail messages to not expect "in row" Split some tests to avoid NoSpamLogger from suppressing expected messages (Rebase of commit d2af13f)
michaelsembwever
pushed a commit
that referenced
this pull request
Feb 16, 2026
* STAR-1812 Stop collection guardrails spamming logs * Fix failing tests (cherry picked from commit e953dc6) (cherry picked from commit 8aa875b) (cherry picked from commit 0d25080) STAR-1812 Fix guardrail tests that broke after row info was removed from logging; In GuardrailCollectionSizeOnSSTableWriteTest.testMapSizeAfterCompaction make sure insert/update statements are unique to get expected logging; Move testGuardrailRespectsMinimumNotificationInterval from distributed to unit test class GuardrailCollectionSizeTest to avoid effects of NoSpamLogger STAR-1812 Fix GuardrailItemsPerCollectionOnSSTableWriteTest; Update expected warn and fail messages to not expect "in row" Split some tests to avoid NoSpamLogger from suppressing expected messages (Rebase of commit d2af13f)
michaeljmarshall
added a commit
that referenced
this pull request
Feb 20, 2026
…2042) ### What is the issue Fixes: https://github.com/riptano/cndb/issues/15527 CNDB test PR: https://github.com/riptano/cndb/pull/16797 ### What does this PR fix and why was it fixed This PR upgrades jvector, which brings several improvements. Here are the git commits brought in: ``` 8b3e93cf (tag: 4.0.0-rc.8) chore: update changelog for 4.0.0-rc.8 (#627) 9d0488e5 release 4.0.0-rc.8 (#626) 570bd118 Refactor parallel writer (#608) 20c348ec Move buffer position in ByteBufferIndexWriter#writeFloats (#607) d9ddce51 Ensure extractTrainingVectors return a list of at most MAX_PQ_TRAINING_SET_SIZE (#610) d663b4f7 add config options for regression testing (#609) 7e493eee On-disk index cache for the Grid benchmark harness (#612) e263cc80 Improved dataset loading; fixes, safeties, diagnostics, and better feedback (#613) 6b235ce7 bump to next SNAPSHOT (#605) 84bf5708 (tag: 4.0.0-rc.7) chore: update changelog for 4.0.0-rc.7 (#604) fceeb885 release 4.0.0-rc.7 (#603) 51807cba add protection against bad ordinal mappings (#602) 6ca3b5e2 adding memory and disk usage stats to bench tests (#591) a66fd914 Fix OnDiskGraphIndex#ramBytesUsed NPE (#588) 0ca5a392 Move float bulk-write into IndexWriter to enforce endianness (#577) a6c6c09b Add diversityScoreFunctionFor to avoid creation of wrapper object (#592) 977c21d4 Relax the threshold of a flaky test related to an experimental feature (#598) fa808d69 adding average nodes visited to benchmark tests (#552) 3bd15e70 Virtualize and Modularize DataSetLoader logic (#593) 42259e9f Speed up ivec reads by buffering (#584) f967f1c9 virtualize DataSet (#589) 55f902f4 turn off parallel writes in grid (#582) 019a241d Parallelize graph writes (#542) 02fea879 Save allocation of a large array in PQVectors.encodeAndBuild (#574) 32a51821 javadoc for base [graph] (#548) 4eb607f8 javadoc for base [disk,exceptions] (#547) 30e8932c Enable the fused graph index (#561) d8848fc6 Start development on 4.0.0-rc.7-SNAPSHOT (#573) c57f3a62 (tag: 4.0.0-rc.6) chore: update changelog for 4.0.0-rc.6 (#572) 214b7c20 release 4.0.0-rc.6 (#571) e3686999 fix javadoc error (#570) 88669887 Ignoring testIncrementalInsertionFromOnDiskIndex_withNonIdentityOrdinalMapping and adding a TODO in buildAndMergeNewNodes (#569) 29a943e1 Computation of reconstruction errors for vector compressors (#567) d8e9cb16 Add NVQ paper in README (#560) d5cbe658 Add ImmutableGraphIndex.isHierarchical (#563) b484dae2 Harden tests for heap graph reconstruction (#543) 9471c57d Make the thresholds in TestLowCardinalityFiltering tighter (#559) 21e4a226 Begin development on 4.0.0-rc.6 (#558) 4f661d99 Revert "Start development on 4.0.0-rc.6-SNAPSHOT" fdee5779 Start development on 4.0.0-rc.6-SNAPSHOT ``` ### SAI Version Bump Adds a new sai on disk version: `fa` ### Fused PQ With this version, we are adding a new, experimental feature to write PQ vectors fused into the graph. In doing so, we are able to skip writing the PQ vectors to the PQ file, which results in significant memory savings since the PQ vectors in the `CassandraDiskAnn` graph searcher consumers `O(n)` memory based on the number of vectors and their quantized size. The fused pq vectors mostly fit within the page cache as we read the node and its neighbors from disk, so we see minimal latency reduction due to this change, though further testing is required to see the real impact. In order to enable fused pq, the runtime needs `cassandra.sai.latest.version=fa` or greater and `cassandra.sai.vector.enable_fused=true`. Note that because this feature is still experimental, `cassandra.sai.vector.enable_fused` defaults to `false`. Another experimental feature introduced in this commit via the jvector upgrade is parallel graph encoding and writing to disk. Writing the fused graph requires increased CPU time to encode the graph node and we write more bytes to disk, so this parallelism is likely necessary to keep vector index creation/compaction times down. The key configurations available with their associated defaults: ```java // When building a compaction graph, encode layer 0 nodes in parallel and subsequently use async io for writes. // This feature is experimental, so defaults to false. SAI_ENCODE_AND_WRITE_VECTOR_GRAPH_IN_PARALLEL_ENABLED("cassandra.sai.vector.encode_and_write_graph_in_parallel.enabled", "false"), // When parallel graph encoding is enabled, the number of threads to use for encoding. Defaults to 0, meaning // use all available processors as reported by the JVM. SAI_ENCODE_AND_WRITE_VECTOR_GRAPH_IN_PARALLEL_NUM_THREADS("cassandra.sai.vector.encode_and_write_graph_in_parallel.num_threads", "0"), // When parallel graph encoding is enabled, whether to use director buffers. Defaults to false, meaning heap // buffers are used. A buffer will be allocated per encoding thread. The size of each buffer is the size // of the encoded graph node at layer 0, which varies based on graph feature settings. SAI_ENCODE_AND_WRITE_VECTOR_GRAPH_IN_PARALLEL_USE_DIRECT_BUFFERS("cassandra.sai.vector.encode_and_write_graph_in_parallel.use_direct_buffers", "false"), ``` ### OnDiskVectorValues and OnDiskVectorValuesWriter `OnDiskVectorValues` is now in its own file and is now thread safe in order to account for some necessary implementation details within jvector. Added `OnDiskVectorValuesWriter` to improve test coverage and to abstract away the flush issues associated with `BufferedRandomAccessWriter` as described in datastax/jvector#562. ### Verification This PR also introduces new benchmarks as well as improved unit testing. The new benchmarks verify the performance of the `OnDiskVectorValues` and `OnDiskVectorValuesWriter` to confirm (at least directionally) the time associated with read and write operations. New tests have been added to verify that when we iterate over an sstable's rows, we are able to assert that the sstable's vector value's similarity to the one stored in the vector graph is ~1. This testing is valuable in that it confirms the row id to ordinal mapping is correct at every node. Previously, we relied on recall results to verify this for us. This new pattern allows us to confirm _every_ node, which is more thorough and removes most edge cases that might have led to partially correct graphs that may have achieved acceptable recall.
michaelsembwever
pushed a commit
that referenced
this pull request
Feb 27, 2026
* STAR-1812 Stop collection guardrails spamming logs * Fix failing tests (cherry picked from commit e953dc6) (cherry picked from commit 8aa875b) (cherry picked from commit 0d25080) STAR-1812 Fix guardrail tests that broke after row info was removed from logging; In GuardrailCollectionSizeOnSSTableWriteTest.testMapSizeAfterCompaction make sure insert/update statements are unique to get expected logging; Move testGuardrailRespectsMinimumNotificationInterval from distributed to unit test class GuardrailCollectionSizeTest to avoid effects of NoSpamLogger STAR-1812 Fix GuardrailItemsPerCollectionOnSSTableWriteTest; Update expected warn and fail messages to not expect "in row" Split some tests to avoid NoSpamLogger from suppressing expected messages (Rebase of commit d2af13f)
michaelsembwever
pushed a commit
that referenced
this pull request
Mar 2, 2026
* STAR-1812 Stop collection guardrails spamming logs * Fix failing tests (cherry picked from commit e953dc6) (cherry picked from commit 8aa875b) (cherry picked from commit 0d25080) STAR-1812 Fix guardrail tests that broke after row info was removed from logging; In GuardrailCollectionSizeOnSSTableWriteTest.testMapSizeAfterCompaction make sure insert/update statements are unique to get expected logging; Move testGuardrailRespectsMinimumNotificationInterval from distributed to unit test class GuardrailCollectionSizeTest to avoid effects of NoSpamLogger STAR-1812 Fix GuardrailItemsPerCollectionOnSSTableWriteTest; Update expected warn and fail messages to not expect "in row" Split some tests to avoid NoSpamLogger from suppressing expected messages (Rebase of commit d2af13f)
michaelsembwever
pushed a commit
that referenced
this pull request
Mar 4, 2026
* STAR-1812 Stop collection guardrails spamming logs * Fix failing tests (cherry picked from commit e953dc6) (cherry picked from commit 8aa875b) (cherry picked from commit 0d25080) STAR-1812 Fix guardrail tests that broke after row info was removed from logging; In GuardrailCollectionSizeOnSSTableWriteTest.testMapSizeAfterCompaction make sure insert/update statements are unique to get expected logging; Move testGuardrailRespectsMinimumNotificationInterval from distributed to unit test class GuardrailCollectionSizeTest to avoid effects of NoSpamLogger STAR-1812 Fix GuardrailItemsPerCollectionOnSSTableWriteTest; Update expected warn and fail messages to not expect "in row" Split some tests to avoid NoSpamLogger from suppressing expected messages (Rebase of commit d2af13f)
michaelsembwever
pushed a commit
that referenced
this pull request
Mar 5, 2026
* STAR-1812 Stop collection guardrails spamming logs * Fix failing tests (cherry picked from commit e953dc6) (cherry picked from commit 8aa875b) (cherry picked from commit 0d25080) STAR-1812 Fix guardrail tests that broke after row info was removed from logging; In GuardrailCollectionSizeOnSSTableWriteTest.testMapSizeAfterCompaction make sure insert/update statements are unique to get expected logging; Move testGuardrailRespectsMinimumNotificationInterval from distributed to unit test class GuardrailCollectionSizeTest to avoid effects of NoSpamLogger STAR-1812 Fix GuardrailItemsPerCollectionOnSSTableWriteTest; Update expected warn and fail messages to not expect "in row" Split some tests to avoid NoSpamLogger from suppressing expected messages (Rebase of commit d2af13f)
michaelsembwever
pushed a commit
that referenced
this pull request
Mar 25, 2026
* STAR-1812 Stop collection guardrails spamming logs * Fix failing tests (cherry picked from commit e953dc6) (cherry picked from commit 8aa875b) (cherry picked from commit 0d25080) STAR-1812 Fix guardrail tests that broke after row info was removed from logging; In GuardrailCollectionSizeOnSSTableWriteTest.testMapSizeAfterCompaction make sure insert/update statements are unique to get expected logging; Move testGuardrailRespectsMinimumNotificationInterval from distributed to unit test class GuardrailCollectionSizeTest to avoid effects of NoSpamLogger STAR-1812 Fix GuardrailItemsPerCollectionOnSSTableWriteTest; Update expected warn and fail messages to not expect "in row" Split some tests to avoid NoSpamLogger from suppressing expected messages (Rebase of commit d2af13f)
michaelsembwever
pushed a commit
that referenced
this pull request
Mar 27, 2026
* STAR-1812 Stop collection guardrails spamming logs * Fix failing tests (cherry picked from commit e953dc6) (cherry picked from commit 8aa875b) (cherry picked from commit 0d25080) STAR-1812 Fix guardrail tests that broke after row info was removed from logging; In GuardrailCollectionSizeOnSSTableWriteTest.testMapSizeAfterCompaction make sure insert/update statements are unique to get expected logging; Move testGuardrailRespectsMinimumNotificationInterval from distributed to unit test class GuardrailCollectionSizeTest to avoid effects of NoSpamLogger STAR-1812 Fix GuardrailItemsPerCollectionOnSSTableWriteTest; Update expected warn and fail messages to not expect "in row" Split some tests to avoid NoSpamLogger from suppressing expected messages (Rebase of commit d2af13f)
michaelsembwever
pushed a commit
that referenced
this pull request
Apr 14, 2026
* STAR-1812 Stop collection guardrails spamming logs * Fix failing tests (cherry picked from commit e953dc6) (cherry picked from commit 8aa875b) (cherry picked from commit 0d25080) STAR-1812 Fix guardrail tests that broke after row info was removed from logging; In GuardrailCollectionSizeOnSSTableWriteTest.testMapSizeAfterCompaction make sure insert/update statements are unique to get expected logging; Move testGuardrailRespectsMinimumNotificationInterval from distributed to unit test class GuardrailCollectionSizeTest to avoid effects of NoSpamLogger STAR-1812 Fix GuardrailItemsPerCollectionOnSSTableWriteTest; Update expected warn and fail messages to not expect "in row" Split some tests to avoid NoSpamLogger from suppressing expected messages (Rebase of commit d2af13f)
driftx
pushed a commit
that referenced
this pull request
Apr 27, 2026
…2042) Fixes: riptano/cndb#15527 CNDB test PR: riptano/cndb#16797 This PR upgrades jvector, which brings several improvements. Here are the git commits brought in: ``` 8b3e93cf (tag: 4.0.0-rc.8) chore: update changelog for 4.0.0-rc.8 (#627) 9d0488e5 release 4.0.0-rc.8 (#626) 570bd118 Refactor parallel writer (#608) 20c348ec Move buffer position in ByteBufferIndexWriter#writeFloats (#607) d9ddce51 Ensure extractTrainingVectors return a list of at most MAX_PQ_TRAINING_SET_SIZE (#610) d663b4f7 add config options for regression testing (#609) 7e493eee On-disk index cache for the Grid benchmark harness (#612) e263cc80 Improved dataset loading; fixes, safeties, diagnostics, and better feedback (#613) 6b235ce7 bump to next SNAPSHOT (#605) 84bf5708 (tag: 4.0.0-rc.7) chore: update changelog for 4.0.0-rc.7 (#604) fceeb885 release 4.0.0-rc.7 (#603) 51807cba add protection against bad ordinal mappings (#602) 6ca3b5e2 adding memory and disk usage stats to bench tests (#591) a66fd914 Fix OnDiskGraphIndex#ramBytesUsed NPE (#588) 0ca5a392 Move float bulk-write into IndexWriter to enforce endianness (#577) a6c6c09b Add diversityScoreFunctionFor to avoid creation of wrapper object (#592) 977c21d4 Relax the threshold of a flaky test related to an experimental feature (#598) fa808d69 adding average nodes visited to benchmark tests (#552) 3bd15e70 Virtualize and Modularize DataSetLoader logic (#593) 42259e9f Speed up ivec reads by buffering (#584) f967f1c9 virtualize DataSet (#589) 55f902f4 turn off parallel writes in grid (#582) 019a241d Parallelize graph writes (#542) 02fea879 Save allocation of a large array in PQVectors.encodeAndBuild (#574) 32a51821 javadoc for base [graph] (#548) 4eb607f8 javadoc for base [disk,exceptions] (#547) 30e8932c Enable the fused graph index (#561) d8848fc6 Start development on 4.0.0-rc.7-SNAPSHOT (#573) c57f3a62 (tag: 4.0.0-rc.6) chore: update changelog for 4.0.0-rc.6 (#572) 214b7c20 release 4.0.0-rc.6 (#571) e3686999 fix javadoc error (#570) 88669887 Ignoring testIncrementalInsertionFromOnDiskIndex_withNonIdentityOrdinalMapping and adding a TODO in buildAndMergeNewNodes (#569) 29a943e1 Computation of reconstruction errors for vector compressors (#567) d8e9cb16 Add NVQ paper in README (#560) d5cbe658 Add ImmutableGraphIndex.isHierarchical (#563) b484dae2 Harden tests for heap graph reconstruction (#543) 9471c57d Make the thresholds in TestLowCardinalityFiltering tighter (#559) 21e4a226 Begin development on 4.0.0-rc.6 (#558) 4f661d99 Revert "Start development on 4.0.0-rc.6-SNAPSHOT" fdee5779 Start development on 4.0.0-rc.6-SNAPSHOT ``` Adds a new sai on disk version: `fa` With this version, we are adding a new, experimental feature to write PQ vectors fused into the graph. In doing so, we are able to skip writing the PQ vectors to the PQ file, which results in significant memory savings since the PQ vectors in the `CassandraDiskAnn` graph searcher consumers `O(n)` memory based on the number of vectors and their quantized size. The fused pq vectors mostly fit within the page cache as we read the node and its neighbors from disk, so we see minimal latency reduction due to this change, though further testing is required to see the real impact. In order to enable fused pq, the runtime needs `cassandra.sai.latest.version=fa` or greater and `cassandra.sai.vector.enable_fused=true`. Note that because this feature is still experimental, `cassandra.sai.vector.enable_fused` defaults to `false`. Another experimental feature introduced in this commit via the jvector upgrade is parallel graph encoding and writing to disk. Writing the fused graph requires increased CPU time to encode the graph node and we write more bytes to disk, so this parallelism is likely necessary to keep vector index creation/compaction times down. The key configurations available with their associated defaults: ```java // When building a compaction graph, encode layer 0 nodes in parallel and subsequently use async io for writes. // This feature is experimental, so defaults to false. SAI_ENCODE_AND_WRITE_VECTOR_GRAPH_IN_PARALLEL_ENABLED("cassandra.sai.vector.encode_and_write_graph_in_parallel.enabled", "false"), // When parallel graph encoding is enabled, the number of threads to use for encoding. Defaults to 0, meaning // use all available processors as reported by the JVM. SAI_ENCODE_AND_WRITE_VECTOR_GRAPH_IN_PARALLEL_NUM_THREADS("cassandra.sai.vector.encode_and_write_graph_in_parallel.num_threads", "0"), // When parallel graph encoding is enabled, whether to use director buffers. Defaults to false, meaning heap // buffers are used. A buffer will be allocated per encoding thread. The size of each buffer is the size // of the encoded graph node at layer 0, which varies based on graph feature settings. SAI_ENCODE_AND_WRITE_VECTOR_GRAPH_IN_PARALLEL_USE_DIRECT_BUFFERS("cassandra.sai.vector.encode_and_write_graph_in_parallel.use_direct_buffers", "false"), ``` `OnDiskVectorValues` is now in its own file and is now thread safe in order to account for some necessary implementation details within jvector. Added `OnDiskVectorValuesWriter` to improve test coverage and to abstract away the flush issues associated with `BufferedRandomAccessWriter` as described in datastax/jvector#562. This PR also introduces new benchmarks as well as improved unit testing. The new benchmarks verify the performance of the `OnDiskVectorValues` and `OnDiskVectorValuesWriter` to confirm (at least directionally) the time associated with read and write operations. New tests have been added to verify that when we iterate over an sstable's rows, we are able to assert that the sstable's vector value's similarity to the one stored in the vector graph is ~1. This testing is valuable in that it confirms the row id to ordinal mapping is correct at every node. Previously, we relied on recall results to verify this for us. This new pattern allows us to confirm _every_ node, which is more thorough and removes most edge cases that might have led to partially correct graphs that may have achieved acceptable recall.
driftx
pushed a commit
that referenced
this pull request
Apr 28, 2026
…2042) Fixes: riptano/cndb#15527 CNDB test PR: riptano/cndb#16797 This PR upgrades jvector, which brings several improvements. Here are the git commits brought in: ``` 8b3e93cf (tag: 4.0.0-rc.8) chore: update changelog for 4.0.0-rc.8 (#627) 9d0488e5 release 4.0.0-rc.8 (#626) 570bd118 Refactor parallel writer (#608) 20c348ec Move buffer position in ByteBufferIndexWriter#writeFloats (#607) d9ddce51 Ensure extractTrainingVectors return a list of at most MAX_PQ_TRAINING_SET_SIZE (#610) d663b4f7 add config options for regression testing (#609) 7e493eee On-disk index cache for the Grid benchmark harness (#612) e263cc80 Improved dataset loading; fixes, safeties, diagnostics, and better feedback (#613) 6b235ce7 bump to next SNAPSHOT (#605) 84bf5708 (tag: 4.0.0-rc.7) chore: update changelog for 4.0.0-rc.7 (#604) fceeb885 release 4.0.0-rc.7 (#603) 51807cba add protection against bad ordinal mappings (#602) 6ca3b5e2 adding memory and disk usage stats to bench tests (#591) a66fd914 Fix OnDiskGraphIndex#ramBytesUsed NPE (#588) 0ca5a392 Move float bulk-write into IndexWriter to enforce endianness (#577) a6c6c09b Add diversityScoreFunctionFor to avoid creation of wrapper object (#592) 977c21d4 Relax the threshold of a flaky test related to an experimental feature (#598) fa808d69 adding average nodes visited to benchmark tests (#552) 3bd15e70 Virtualize and Modularize DataSetLoader logic (#593) 42259e9f Speed up ivec reads by buffering (#584) f967f1c9 virtualize DataSet (#589) 55f902f4 turn off parallel writes in grid (#582) 019a241d Parallelize graph writes (#542) 02fea879 Save allocation of a large array in PQVectors.encodeAndBuild (#574) 32a51821 javadoc for base [graph] (#548) 4eb607f8 javadoc for base [disk,exceptions] (#547) 30e8932c Enable the fused graph index (#561) d8848fc6 Start development on 4.0.0-rc.7-SNAPSHOT (#573) c57f3a62 (tag: 4.0.0-rc.6) chore: update changelog for 4.0.0-rc.6 (#572) 214b7c20 release 4.0.0-rc.6 (#571) e3686999 fix javadoc error (#570) 88669887 Ignoring testIncrementalInsertionFromOnDiskIndex_withNonIdentityOrdinalMapping and adding a TODO in buildAndMergeNewNodes (#569) 29a943e1 Computation of reconstruction errors for vector compressors (#567) d8e9cb16 Add NVQ paper in README (#560) d5cbe658 Add ImmutableGraphIndex.isHierarchical (#563) b484dae2 Harden tests for heap graph reconstruction (#543) 9471c57d Make the thresholds in TestLowCardinalityFiltering tighter (#559) 21e4a226 Begin development on 4.0.0-rc.6 (#558) 4f661d99 Revert "Start development on 4.0.0-rc.6-SNAPSHOT" fdee5779 Start development on 4.0.0-rc.6-SNAPSHOT ``` Adds a new sai on disk version: `fa` With this version, we are adding a new, experimental feature to write PQ vectors fused into the graph. In doing so, we are able to skip writing the PQ vectors to the PQ file, which results in significant memory savings since the PQ vectors in the `CassandraDiskAnn` graph searcher consumers `O(n)` memory based on the number of vectors and their quantized size. The fused pq vectors mostly fit within the page cache as we read the node and its neighbors from disk, so we see minimal latency reduction due to this change, though further testing is required to see the real impact. In order to enable fused pq, the runtime needs `cassandra.sai.latest.version=fa` or greater and `cassandra.sai.vector.enable_fused=true`. Note that because this feature is still experimental, `cassandra.sai.vector.enable_fused` defaults to `false`. Another experimental feature introduced in this commit via the jvector upgrade is parallel graph encoding and writing to disk. Writing the fused graph requires increased CPU time to encode the graph node and we write more bytes to disk, so this parallelism is likely necessary to keep vector index creation/compaction times down. The key configurations available with their associated defaults: ```java // When building a compaction graph, encode layer 0 nodes in parallel and subsequently use async io for writes. // This feature is experimental, so defaults to false. SAI_ENCODE_AND_WRITE_VECTOR_GRAPH_IN_PARALLEL_ENABLED("cassandra.sai.vector.encode_and_write_graph_in_parallel.enabled", "false"), // When parallel graph encoding is enabled, the number of threads to use for encoding. Defaults to 0, meaning // use all available processors as reported by the JVM. SAI_ENCODE_AND_WRITE_VECTOR_GRAPH_IN_PARALLEL_NUM_THREADS("cassandra.sai.vector.encode_and_write_graph_in_parallel.num_threads", "0"), // When parallel graph encoding is enabled, whether to use director buffers. Defaults to false, meaning heap // buffers are used. A buffer will be allocated per encoding thread. The size of each buffer is the size // of the encoded graph node at layer 0, which varies based on graph feature settings. SAI_ENCODE_AND_WRITE_VECTOR_GRAPH_IN_PARALLEL_USE_DIRECT_BUFFERS("cassandra.sai.vector.encode_and_write_graph_in_parallel.use_direct_buffers", "false"), ``` `OnDiskVectorValues` is now in its own file and is now thread safe in order to account for some necessary implementation details within jvector. Added `OnDiskVectorValuesWriter` to improve test coverage and to abstract away the flush issues associated with `BufferedRandomAccessWriter` as described in datastax/jvector#562. This PR also introduces new benchmarks as well as improved unit testing. The new benchmarks verify the performance of the `OnDiskVectorValues` and `OnDiskVectorValuesWriter` to confirm (at least directionally) the time associated with read and write operations. New tests have been added to verify that when we iterate over an sstable's rows, we are able to assert that the sstable's vector value's similarity to the one stored in the vector graph is ~1. This testing is valuable in that it confirms the row id to ordinal mapping is correct at every node. Previously, we relied on recall results to verify this for us. This new pattern allows us to confirm _every_ node, which is more thorough and removes most edge cases that might have led to partially correct graphs that may have achieved acceptable recall.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
3 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
This is a port of https://github.com/riptano/bdp/pull/20124 for CNDB-3630 from
master