feat(tantivy): Add Tantivy full-text search with on-demand archive reading

[JingsongLi]

Purpose

Subtask of #227

Introduce a complete Tantivy-based full-text search pipeline for global indexes, with on-demand I/O throughout:

• ArchiveDirectory: reads only the archive header eagerly; file data is loaded via async FileRead when Tantivy requests it (sync-to-async bridge using std::thread::scope).
• TantivyFullTextWriter: streams the packed archive directly to an OutputFile instead of buffering in memory.
• TantivyFullTextReader: opens from InputFile/FileRead, never loads the full archive into memory.
• FullTextSearchBuilder: self-contained builder on Table that reads the index manifest, evaluates searches against multiple Tantivy indexes in parallel (try_join_all), and returns ScoredGlobalIndexResult.
• ScoredGlobalIndexResult + bitmap_to_ranges moved to table/source.rs (alongside RowRange) so vector search can reuse them later.
• TableScan.with_row_ranges(): generic row-range filtering, decoupled from full-text specifics.
• DataFusion full_text_search UDTF integration with test data.

Brief change log

Tests

API and Format

Documentation

[jerry-024]

1. Unsafe integer casts in archive parsing (crates/paimon/src/tantivy/directory.rs)

i32::from_be_bytes(...) as usize for file_count — if the archive is corrupt or malicious and file_count is negative, this wraps to a very large usize, causing OOM or panic in HashMap::with_capacity.

Same issue with i64::from_be_bytes(...) as u64 for data_len — a negative value wraps to a huge offset.

Suggest validating both are non-negative before casting:

let file_count = i32::from_be_bytes([buf[0], buf[1], buf[2], buf[3]]);
if file_count < 0 {
    return Err(io::Error::new(io::ErrorKind::InvalidData, "Negative file count in archive"));
}
let file_count = file_count as usize;

(and similarly for data_len)

---

2. UInt64 as i64 can silently overflow (crates/integrations/datafusion/src/full_text_search.rs)

In extract_int_literal: ScalarValue::UInt64(Some(v)) => Ok(*v as i64) — values greater than i64::MAX silently wrap to negative, which then fails the limit <= 0 check with a confusing error, or worse passes through.

Suggest:

ScalarValue::UInt64(Some(v)) => i64::try_from(*v).map_err(|_| {
    datafusion::error::DataFusionError::Plan(
        format!("full_text_search: {name} value {v} exceeds i64 range")
    )
}),

[jerry-024]

+1