fix(scan): reduce large-table global-index planning memory

[XiaoHongbo-Hope]

Summary

• align Rust large-table planning with Java's two-pass manifest scan: collect DELETE identifiers first, then retain live ADD entries
• prune files against exact global-index row ranges while each manifest is consumed, instead of retaining every live file until the end
• load the BTree null bitmap lazily; equality and prefix queries no longer deserialize it during reader construction
• recognize escaped SQL LIKE wildcards, so a literal-underscore prefix becomes an exact StartsWith index predicate

This version removes the rejected DataFusion-specific streaming path and does not include #672's planner-time LIMIT pushdown, candidate early stop, or hard query-memory fallback.

Why

The Shanghai table currently has roughly 23.6 million manifest entries. Rust previously accumulated all live DataFileMeta values before applying global-index row ranges, which amplified scan-planning memory. Java performs a two-pass manifest scan and avoids eagerly loading the BTree null bitmap.

Validation

Shanghai DLF/OSS table, under an 8 GiB memory limit:

• equality planning: 14.2 s, 533 MiB peak RSS, 24,749 row IDs, 29 files, 4 splits
• equivalent StartsWith planning: 13.7 s, 546 MiB peak RSS
• escaped prefix LIKE planning: 22.3 s, 570 MiB peak RSS, 24,749 row IDs, 29 files

Local tests:

• table scan tests: 81 passed
• BTree tests: 28 passed, including a new lazy-null-bitmap invariant test
• LIKE predicate tests: 5 passed
• Python escaped-LIKE test: passed
• DataFusion escaped-LIKE translation test: passed
• cargo fmt --all -- --check and git diff --check: passed

SQL semantics

In SQL LIKE, an unescaped _ matches any single character. Therefore the original pattern with several underscores is a broad residual LIKE and cannot safely become a prefix index lookup. Literal underscores must be escaped, for example:

WHERE clip_id LIKE 'lowprec\_78b6c46abebe83dd\_20260803\_173424\_c002%' ESCAPE '\'
LIMIT 1000

That escaped form is recognized as an exact prefix and completed within the memory limit above.

[XiaoHongbo-Hope]

Addressed the two review concerns in 483555a:

• Streaming execution now prepares the fixed snapshot once and reuses its decoded manifest-list metadata across row-id batches. A bounded 64 MiB query-local manifest-content cache avoids reopening the same overlapping manifest file while keeping metadata memory capped.
• DETAIL mode now uses exact live data-file row ranges. It does not switch to FAST because that would skip unindexed live data and change query correctness. Direct callers that omit DETAIL ranges also resolve them exactly instead of widening to [0, next_row_id).
• The combined Paimon residual predicate is built once per execution rather than reconstructed for every batch.

Validation: table-scan tests 82 passed; streaming Top-K integration passed; cargo clippy for paimon and paimon-datafusion passed with -D warnings.

[JingsongLi]

See comments in #672

[JingsongLi]

Please re-create this optimization without #672

[JingsongLi]

This PR introduces a large, DataFusion-specific execution path instead of addressing the underlying RowID representation problem. The main risks are:

1. Incorrect results with mixed BTree and Bitmap coverage
   In FAST mode, the streaming path reads only BTree indexes. If Bitmap indexes cover additional RowID ranges for the same field, matching rows from those ranges can be silently omitted.

2. Valid large indexes can fail queries
   The streaming reader enforces a hard-coded 256 MiB block limit. A valid hotspot key may legitimately contain more RowIDs than this because all RowIDs for one key are stored in a single BTree value. The query then fails instead of falling back to a normal data scan.

3. Existing fallback controls are bypassed
   Wide LIKE predicates without a literal prefix may scan every BTree shard while ignoring btree-index.fallback-scan-max-size. This can cause extremely long-running queries and excessive index I/O.

4. Repeated planning overhead
   Every 250K-RowID batch repeats manifest decoding, pruning, split planning, and file setup. The cache avoids some object-store reads but does not eliminate most of the repeated CPU and planning work.

5. Reduced parallelism
   The new path processes BTree shards serially and forces the DataFusion scan into a single execution partition. global-index.thread-number is effectively ignored for this path.

6. Incomplete and inconsistent coverage
   The optimization only applies to DataFusion, BTree indexes, and single-leaf Eq, StartsWith, and Like predicates. Bitmap indexes, compound predicates, range predicates, and other readers/connectors retain the original query-wide materialization problem.

7. High maintenance and regression risk
   The PR adds a second scan-planning and execution pipeline, including snapshot preparation, manifest caching, ownership assignment, batching, and fallback logic. This duplicates existing behavior and makes correctness across search modes, index types, and connectors harder to maintain.

First, we need to investigate the differences between Rust and Java to understand why Rust's memory usage is exceeding expectations.

If Java also fails to handle the load, we should consider falling back to a full table scan instead of using a global index. Global indexes are designed for small datasets or range queries; attempting to force them to serve other use cases—as discussed in the feedback above—carries significant risk.

[XiaoHongbo-Hope]

@JingsongLi Sorry, I did not have a time to review the PR diff carefully myself, mark it as draft. Back it to open again once ready.