Support Java-compatible bitmap global index

[JingsongLi]

Summary

Adds Java-compatible bitmap global index support alongside the existing BTree global index path. The PR also wires sorted global-index reads, build/drop procedures, fallback scans, and table options so BTree and Bitmap behavior stay aligned with Java Paimon.

Changes

• Add a bitmap global index reader/writer using the Java-compatible on-disk format and BTree metadata for pruning.
• Reuse the sorted global-index scanner for BTree and Bitmap files, including equality, range, null, and string fallback scan reads.
• Align fallback scan budgeting with Java: the limit is the total size of selected files per index kind, applies to range/between and suffix/contains/complex LIKE predicates, and over-budget fallback predicates become unsupported instead of returning full indexed coverage.
• Add Java-written bitmap global index golden files (.index and .meta) generated by Java Paimon's BitmapGlobalIndexer, and use them in scanner compatibility tests.
• Extend global index build/drop procedures and DataFusion tests to support index_type => 'bitmap'.
• Align fallback scan options with Java names and defaults: btree-index.fallback-scan-max-size and bitmap-index.fallback-scan-max-size, both defaulting to 256mb.
• Update BTree serialization/query handling and docs for Java-compatible global-index reads.

Testing

• cargo test -p paimon spec::core_options::tests
• cargo test -p paimon table::global_index_scanner::tests
• cargo test -p paimon btree::tests
• cargo test -p paimon-datafusion --test procedures
• cargo fmt --check
• git diff --check

Notes

No migration is required. The new fallback scan table options use Java-compatible names rather than Rust-only aliases.

[leaves12138]

I checked the bitmap global index changes against Java's BitmapGlobalIndexFormat, BitmapIndexReader, and SortedFileGlobalIndexReader. The default no-compression file layout looks aligned: magic/version/footer length and big-endian footer fields, null/non-null bitmap placement, dictionary/index block varint encoding, serialized-key ordering, raw roaring bitmap blocks, and the 5-byte compressed-block trailer/CRC all match the Java format. The Java-generated golden bitmap fixture is a useful addition.

No blocker from my side. Two non-blocking compatibility boundaries are worth documenting/tracking:

1. Java can write bitmap dictionary/index blocks with lz4 or lzo via bitmap-index.compression, while the Rust reader currently supports only none and zstd.
2. Java's fallback dictionary scan limit is based on the total selected index file size and applies to range/between fallback scans as well. Rust currently checks the limit per index file, applies it only to ends_with/contains/residual like, and returns full indexed coverage when a fallback scan is skipped. This is safe with residual row filters, but it is not identical to Java's pruning/performance behavior.

[leaves12138]

Thanks for the update. I re-checked the bitmap global index format and fallback behavior against the Java implementation. The fallback-budget change looks aligned now, but I found one remaining Java compatibility issue in decimal key serialization.

Java's KeySerializer.DecimalSerializer decides whether a decimal is compact from the column DecimalType precision captured by KeySerializer.create(dataField.type()), not from the literal object's own precision. The Rust implementation currently uses Datum::Decimal.precision in serialize_datum, so a predicate literal whose datum precision differs from the column precision can be encoded differently from Java. For example, for a DECIMAL(20, 0) column with a literal datum carrying precision 10, Java writes the non-compact BigInteger.toByteArray() form, while Rust currently writes the compact little-endian i64 form.

Suggested fix:

diff --git a/crates/paimon/src/btree/key_serde.rs b/crates/paimon/src/btree/key_serde.rs
@@
         Datum::Decimal {
             unscaled,
             precision,
             ..
         } => {
-            if *precision <= DECIMAL_COMPACT_PRECISION {
+            let key_precision = match data_type {
+                DataType::Decimal(decimal_type) => decimal_type.precision(),
+                _ => *precision,
+            };
+            if key_precision <= DECIMAL_COMPACT_PRECISION {
                 (*unscaled as i64).to_le_bytes().to_vec()
             } else {
                 encode_java_big_integer_i128(*unscaled)
             }
         }

Please also add a regression test where the literal datum precision differs from the column decimal precision, covering both compact and non-compact column types.

I applied this fix locally and verified it with:

• cargo test -p paimon key_serde --lib
• cargo test -p paimon bitmap --lib
• cargo test -p paimon fallback_scan --lib
• cargo test -p paimon-datafusion --test procedures global_index -- --nocapture

I could not push the fix directly to this PR branch from the current token because the head branch is in the author's fork and this token has no push permission there.

[leaves12138]

Looks good to me. I re-checked the Java-compatible bitmap global index implementation against the Java format and query behavior, including the fallback-budget behavior and the decimal key serialization fix.

Verified locally with:

• cargo fmt --all -- --check
• cargo test -p paimon key_serde --lib
• cargo test -p paimon bitmap --lib
• cargo test -p paimon fallback_scan --lib
• cargo test -p paimon-datafusion --test procedures global_index -- --nocapture