Fix BinaryIndexerString collision on NUL-containing keys - #689
Merged
Conversation
BinaryIndexerString packs UTF-8 bytes into longs without encoding the byte length, so trailing 0x00 (NUL) bytes vanish during packing: "alpha" and "alpha\u0000", or NUL-only strings of different length, collided the same index key. An exact-match query then returned entities that an in-memory Condition#test scan rejects. Reject keys containing the NUL character (U+0000) with an IllegalArgumentException in fillCarrier, which both the add and query paths go through. This mirrors how BinaryIndexerLong rejects the reserved Long.MAX_VALUE and turns a silent false-positive into a loud failure, with no change to the persistent format. NUL-free strings and the empty string (#688 sentinel) are unaffected.
- BinaryIndexerString: collapse the stray trailing blank lines before the interface's closing brace to a single one, matching neighboring types. - types.adoc: clarify that NUL rejection applies to every operation that derives the index key (add, query, update, remove — the latter two re-index), and add an upgrade WARNING for storages that already contain NUL-valued keys.
zdenek-jonas
approved these changes
Jun 2, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
BinaryIndexerStringpacks a string's UTF-8 bytes intolongs without encoding the byte length. Trailing0x00(NUL) bytes occupy the high bytes of the lastlongandcontribute nothing, so they vanish during packing — and a
longthat ends up fully0Lis additionally remapped to theLong.MAX_VALUE"all-null" sentinel. As a result:"alpha"and"alpha\u0000"pack to the same index key, and[Long.MAX_VALUE].An exact-match query for one therefore returns the other. This contradicts the index's own equality notion —
Condition#test(entity)(the in-memory predicate the same condition uses) reports no match — so the index disagrees with a linear scan. It's a false-positive only (no missed results, no data loss), but a genuine exact-match correctness gap, and it survives the empty-string-sentinel fix from #688 (the empty string has its own distinct sentinel; NUL strings still don't).Fix
Reject keys containing the NUL character (
U+0000) with anIllegalArgumentExceptioninfillCarrier. Both the add path (index → indexValue → fillCarrier) and the query path (is → isValue → indexValue → fillCarrier) converge there, so NUL is rejected on both adding an entity and forming a query condition.This mirrors the existing family pattern where
BinaryIndexerLongrejects the reservedLong.MAX_VALUE. It turns a silent wrong-result into a loud failure, requires no change to the persistent format (no migration of existing GigaMaps), and leaves NUL-free strings and the empty string fully supported. Lenient "trim trailing garbage" behavior, if desired, belongs at the input — not as a silent, partial, NUL-only normalization inside the index.Changes
BinaryIndexerString.java— guard at the top offillCarrierthrowingIllegalArgumentExceptionfor NUL keys; updated Javadoc.BinaryIndexerStringNulRejectionTest.java(new) — verifies NUL-only, trailing-NUL, and embedded-NUL keys are rejected on both add and query, and that normal keys / the empty string still work. NUL strings are built at runtime so the source contains no raw NUL bytes.BinaryIndexerStringTest.java— the existing test stored an 8-NUL string (now rejected); replaced that scenario with a normal value.docs/.../indexing/bitmap/types.adoc— documented the limitation alongside theBinaryIndexerLong/Long.MAX_VALUEnote, with a code example showing how to normalize NUL out (cutAtNul/stripNul) on both the indexed value and the query key.Trade-off
Breaks callers that currently store NUL-containing strings (rare). They must strip or cut at the terminator before indexing — see the new docs example.