feat: enable page-level Parquet stats + add rg_partition_prefix_len marker

[g-talbot]

Summary

PR-1 of a 7-PR stack producing a memory-bounded, streaming, column-major Parquet pipeline (the reformulated Phase 4a). This PR lays the on-disk format foundation and verifies it works end-to-end through the production query path. Subsequent PRs add the streaming reader, writer, merge engine, and pipeline wiring.

Three changes:

1. Page-level statistics by default. Writer flips from EnabledStatistics::Chunk to EnabledStatistics::Page, so every new file carries Parquet Column Index + Offset Index in the footer. The level is now a config knob — ParquetWriterConfig::with_page_statistics(false) falls back to chunk-level for callers that want a smaller footer. A second knob (with_data_page_row_count_limit) exposes Parquet's per-page row count rollover threshold, useful for tests and for future writers that want to control page granularity directly.

2. qh.rg_partition_prefix_len marker. A numeric KV in the Parquet footer (and a matching u32 field on ParquetSplitMetadata) recording how many leading sort schema columns row group boundaries align with. 0 (or absent) = no claim, the legacy default. N = aligned with the first N sort columns. The marker is read by the merge engine and added to the compaction scope so files with different prefix values stay in separate buckets. The merge engine preserves the inputs' prefix on a single-RG output (vacuously aligned) and demotes to 0 only when the output is genuinely multi-RG, so post-PR-3 ingest splits stay in their bucket through small merges.

3. Developer tooling. inspect_parquet_page_stats(...) library + inspect_parquet CLI binary in quickwit-parquet-engine. Reads the footer (including the page indexes), pretty-prints a per-RG / per-column / per-page report, and supports --json, --all-pages, and --verify-prefix (which enforces the strong form of the prefix claim: every column in the prefix must be constant within each RG).

End-to-end verification (Gate-A)

A new integration test (test_page_index_pruning_via_query in quickwit-datafusion/tests/metrics.rs) confirms page-level pruning fires through the production query path:

• Builds a single split with two metric_names interleaved in one row group, forced into ~16 pages.
• Runs WHERE metric_name = 'cpu.usage' via the regular SQL session builder (no shortcuts; same path the REST API uses).
• Walks the executed ExecutionPlan for the PruningMetrics::page_index_rows_pruned counter exposed by DataFusion's ParquetSource.
• Asserts the counter ≥ 4096 (the rows from the other metric) AND that the query still returns exactly the cpu.usage rows.

This proves that:

• The new EnabledStatistics::Page writer config makes the column index + offset index land in the footer (already covered by inspect.rs unit tests).
• The reader path (MetricsParquetTableProvider::scan) loads the page index — ParquetSource::with_enable_page_index(true) was already wired at line 207, no change needed.
• DataFusion's predicate pushdown evaluates against the page-level stats and skips matching pages.

This is the gate that had to pass before PR-3 cuts ingest over to single-RG: without page-level pruning, single-RG would collapse query pruning to one min/max per file. The gate is now passed.

Why now

Single-row-group files — coming in PR-3 — have only one RG worth of chunk-level statistics, which collapses query pruning to one min/max per file unless page-level data is in the footer. Page indexes have to land in the footer before the writer cuts over to single-RG, which is what this PR sets up.

The marker is the contract between the streaming reader and the merge engine: only files that claim the same prefix length can be merged together. Defining and threading it through compaction scope + validation now means later PRs in the stack just turn on writes — no retroactive plumbing.

Footer overhead

Measured on production-sized files (zstd-3, default writer config) by writing the same data twice — once with EnabledStatistics::Chunk, once with Page — and comparing total file size:

Shape	Compressed size	Page-index overhead
1.5M rows × 11 cols (realistic)	22.6 MiB	+0.033% (7.6 KiB)
3.5M rows × 11 cols (realistic)	52.2 MiB	+0.033% (17.6 KiB)
3.5M rows × 50 cols (wide schema worst case)	52.9 MiB	+0.162% (87.5 KiB)

The page index scales with column count × page count, not row count, so on production-sized splits it's a rounding error. Pinned by an integration test (test_footer_size_delta_for_page_level_stats) that fails if the delta exceeds 30% on a small synthetic file — a generous bound that catches regressions without flapping on the absolute size.

Test plan

• cargo nextest run -p quickwit-parquet-engine --all-features — 384 / 384 (17 new for this PR)
• cargo nextest run -p quickwit-datafusion --test metrics — 14 / 14 (1 new: end-to-end page-index pruning)
• cargo clippy -p quickwit-parquet-engine --tests — clean
• cargo clippy -p quickwit-datafusion --tests — clean
• cargo doc -p quickwit-parquet-engine --no-deps — clean
• cargo machete — clean
• bash quickwit/scripts/check_license_headers.sh — clean
• bash quickwit/scripts/check_log_format.sh — clean
• Manual: cargo run -p quickwit-parquet-engine --bin inspect_parquet -- <file>.parquet produces a useful human report; --json round-trips through serde; --verify-prefix errors with a per-RG diagnostic when the prefix claim is violated
• Workspace cargo check --workspace --all-features — no breakage from the new ParquetSplitMetadata field (default 0, additive JSON Serde) or the new ParquetWriterConfig field (default 0 = unbounded, behavior unchanged)

What's not in this PR (deferred to later in the stack)

• Any writer or reader changes beyond the config flip and the marker plumbing.
• Setting rg_partition_prefix_len > 0 from any production code path. PR-3 (single-RG ingest) is the first writer that will set it; PR-6 (streaming column-major merge engine) is where multi-RG-by-metric_name output lands.

🤖 Generated with Claude Code