feat: add native Delta Lake scan contrib module (page/row-group pruning)

[dwsmith1983]

Which issue does this PR close?

Part of #174 (Explore integration with Delta Lake). It does not close #174, that issue also tracks writes, CDF, and broader integration; this PR delivers the native read path.

Supersedes two earlier efforts, and deliberately builds on both (both given co-authored by since ideas were learned and borrowed):

• [Tracking] feat(contrib): Native Delta Lake scan via delta-kernel-rs (Iceberg-style contrib) #4366 (@schenksj) — the delta-kernel-rs-based contrib. This PR keeps its contrib module shape, build gating, decline-gate philosophy, and test scenario catalog, but replaces the kernel data-read path: kernel's parquet reader has no page-index pruning, and its planning duplicates work delta-spark has already done by the time the DSv1 scan reaches Comet.
• feat: support native Comet scan of plain Delta Lake tables #4669 (@adityavaish) — the minimal plain-table native scan. This PR takes the same core insight (reuse Comet's existing native parquet scan; no kernel needed) and extends it to deletion vectors, column mapping, DPP, and the full Spark matrix.

Rationale for this change

Comet currently falls back to Spark's reader for all Delta tables (isFileFormatSupported requires exact ParquetFileFormat, and DeltaParquetFileFormat is a subclass). That forfeits native execution and all of Comet's parquet pruning on one of the most common table formats.

Key observation: delta-spark has already done log replay, snapshot resolution, time travel, and partition pruning by the time CometScanRule sees the FileSourceScanExec. So no Delta planning is needed on the native side at all, the scan can route through the exact same DataFusion ParquetSource path as CometNativeScanExec, inheriting row-group stats pruning, page-index pruning (#5142), and filter pushdown (#4722) for free. The only genuinely Delta-specific native code is deletion-vector decoding: DV bitmaps are decoded into per-file ParquetAccessPlans, which DataFusion intersects with page-index pruning, so deleted rows are skipped in-scan and DV skips compose with page skips.

Local benchmark (20M rows, selective predicate): 1.44x faster than stock Spark 9.4% of bytes read; DV tables at time parity with in-scan DV application.

What changes are included in this PR?

• contrib/delta/ new Maven module behind a -Pdelta profile: scan rule, decline gates, serde, CometDeltaNativeScanExec (split-mode partition serialization, DPP via derived scan helper), ServiceLoader registrations, differential test suites, Delta own-suite regression harness, benchmark script.
• Core (Delta-free, generic) new CometScanRuleExtension SPI + ServiceLoader hook at the top of transformV1Scan; CometNativeScan.convert body extracted into reusable buildNativeScanCommon`.
• Native delta cargo feature: DeltaScan proto + planner arm delegati the shared parquet scan builder; delta_dv.rs for DV blob unframing (CRC verified), roaring decode (portable + native magic), and access-plan construction.
• Supported plain/partitioned tables, deletion vectors (inline + on-disk), column mapping name mode, time travel, checkpoints, OPTIMIZE'd tables, schema evolution + defaults, DPP, INT96, special-character paths.
• Declines (fall back to Spark, tagged for EXPLAIN) CDC reads, column mapping id mode, row-index-consuming plans, unknown reader features, generated columns, encryption, input_file_name().
• Spark matrix: 3.5 / 4.0 / 4.1 (Delta 3.3.2 / 4.0.1 / 4.3.1 — Delta 4.1.0/4.2.0 are binary-incompatible with Spark 4.1.3). Spark 3.4 declines (Delta 2.4 needs shims); spark-4.2 profile is dormant until Delta ships support.

How are these changes tested?

• 36-test differential suite (comet-on, comet-off) covering the DV, column mapping, DPP, schema evolution, and decline-gate matrices green on Spark 3.5/4.0/4.1 cells. Pruning is asserted as hard metrics checks (page_index_rows_pruned > 0, row_groups_pruned_statistics > 0), not benchmark notes.
• Delta's own test suites run with Comet injected (`dev/run-delta-regression.s DeletionVectorsSuite 29/29, TimeTravel/ColumnMapping/DeleteSQL/UpdateSQL 197/197, MergeIntoSQLSuite 664/665 (the one failure is a scan-telemetry count assertio plan-shape artifact; data assertions pass).
• Rust unit tests for DV unframing/decode/access-plan edge cases and the feature-off error path.
• A DML repro suite proving DELETE writes DVs (not rewrites) under the claimed scan, in both useMetadataRowIndex modes.

---

Co-authored-by: Scott Schenkein schenksj@yahoo.com
Co-authored-by: Aditya Vaish adivaish@microsoft.com

[dwsmith1983]

Update: pushed two follow-up commits extending the scan's pruning and object-store behavior.

perf: fetch Delta deletion vectors and footers concurrently DV blob and footer reads were sequential: two serial round-trips per DV'd file before the scan could start, which scales badly on object stores. They now fetch with a bounded fan-out of 8, preserving file order and fail-fast error semantics. Covered by a new end-to-end unit test (inline DVs, on-disk DVs, pass-through files, exact row selections, output ordering).

feat: push resolved scalar-subquery filters into the native Delta scan predicates like id >= (SELECT max(ts) FROM checkpoint) previously contributed nothing to the native scan: subquery results don't exist at planning, so the scan
decoded the full table and Spark's covering FilterExec did all the filtering. They are now resolved at execution time and appended as pushed filters, so row-group and page-index pruning fire the same as for literal bounds. Three version-specific traps handled:

1. Spark 3.x strips subquery predicates from a scan's dataFilters (FileSourceStrategy); Spark 4.x keeps them. The contrib harvests them from the covering FilterExec at claim time and dedups, so both behaviors converge.
2. The DV plan shape interposes nodes between the filter and the scan, so the harvest matches the nearest filter above the scan, guarded by references scan output.
3. MergeScalarSubqueries fuses multiple scalar subqueries into one struct-returning subquery accessed via GetStructField; that subtree is folded to a literal before serialization.

[dwsmith1983]

HI @andygrove,

Can you review this as it adds Delta functionality?