Restore dropped Spice patch families (metadata columns, object versioning, version-cache invalidation, hash-join accumulator hooks)

[lukekim]

Summary

Restores Spice patch families that existed on the old spiceai-52.5 line (539e6963) but were functionally deleted by the 53.1.0 reconcile merge 289b74a64 ("Carry Spice patches forward to DataFusion 53.1.0"). The commits remained reachable by SHA, but their wiring was gone from the tip tree of spiceai-53-patches (0c648eb06).

Three of the four audited families are restored here. The fourth (HashJoin custom left-side accumulator) is not restored because 53.1 deliberately replaced the underlying mechanism upstream — details below.

Built and tested against the spiceai/arrow-rs fork (rev 0bb8862, arrow/parquet 58.3.0), which already carries ObjectVersionType + ParquetObjectReader::new_with_meta.

Families restored

1. Metadata columns (_location / _last_modified / _size)

datasource/src/metadata.rs (MetadataColumn) survived the reconcile but was orphaned — with_metadata_cols / ExtendedColumnProjector had 0 references. Upstream 53.1 has no MetadataColumn, and it replaced the 52.5 ExtendedColumnProjector (a single chokepoint in file_stream.rs) with a per-source projection model: TableSchema + SplitProjection + ProjectionOpener, plus the bespoke ParquetOpener literal-injection path.

Rather than port ExtendedColumnProjector verbatim (its host code no longer exists), metadata columns are threaded the DF53-native way:

• TableSchema carries metadata_cols, appended to table_schema() after the partition columns.
• SplitProjection::new_with_table_schema classifies trailing metadata-column indices separately from partition columns; ProjectionOpener substitutes them with per-file literals from PartitionedFile.object_meta (inject_metadata_columns_into_projection). The legacy SplitProjection::new is unchanged.
• ParquetOpener (which does not use ProjectionOpener) folds metadata columns into its literal_columns map from the file's ObjectMeta.
• FileSource gains with_metadata_cols() (default None; implemented for parquet/csv/json/avro/arrow).
• FileScanConfigBuilder::with_metadata_cols() folds them into the FileSource.
• ListingOptions gains metadata_cols + with_metadata_cols + validate_metadata_cols; ListingTable wires them into the scan.

Public API names/signatures match the 52.5 surface so spice2 call sites (cayenne provider/table.rs, runtime metadata tests) compile unchanged. ExtendedColumnProjector itself is not reintroduced — it is purely internal and has no spice2 call sites; its behavior is provided by the DF53-native injection above.

2. Object versioning (ObjectVersionType)

Restores object_versioning_type end-to-end (origin f32727fbb / #113):

• DefaultParquetFileReaderFactory / CachedParquetFileReaderFactory gain object_versioning_type + with_object_versioning_type(), and build the reader via ParquetObjectReader::new_with_meta(...).with_object_versioning_type(...).
• FileScanConfig / FileScanConfigBuilder gain object_versioning_type (cfg parquet); ParquetFormat::create_physical_plan wires it into the cached factory.
• ListingOptions gains object_versioning_type + with_object_versioning_type, wired into ListingTable's scan.
• datasource's parquet feature now enables parquet/object_store so the store-level types are in scope.
• Cargo.lock refreshed so the spiceai/arrow-rs fork patches (arrow + parquet 58.3.0) are actually adopted. The base lock left these [patch] entries unused in the crate graph (parquet resolved to crates.io 58.0.0), so ObjectVersionType was not visible at all.

3. Object-version cache invalidation (is_same_file_version)

Upstream 53.1 rewrote the metadata/statistics cache to validate purely on size + last_modified (CachedFileMetadata::is_valid_for / CachedFileMetadataEntry::is_valid_for). Restores the etag/version-aware is_same_file_version (origin d8a28cf63 / #113) in cache_unit.rs and consults it from both is_valid_for checks, so versioned objects sharing size + mtime are correctly invalidated.

Family NOT restored: HashJoin custom left-side accumulator

The 52.5 patch (origin 38b6c042c / 463193add / #117) made HashJoinExec generic over a CollectLeftAccumulator trait (HashJoinExec<A = MinMaxLeftAccumulator>), added recreate_with_accumulator::<T>(), and a ColumnBounds trait whose physical_expr() produced a custom per-partition dynamic-filter expression (spice2's ExactLeftAccumulator produces an exact InList with a byte-budget fallback to range + bloom).

DF53.1 deliberately replaced this mechanism upstream: exec.rs was rewritten (+1643/-424 vs the 52.5 base), ColumnBounds is now a concrete pub(crate) min/max struct (not a trait), and the dynamic filter is built centrally in shared_bounds.rs::create_membership_predicate via a PushdownStrategy (InList for small build sides, hash-table lookup for large) driven by SharedBuildAccumulator + inlist_builder.rs. There is no accumulator-injection seam, no generic parameter, and no physical_expr() delegation.

Restoring the spice2 API contract would mean reverting that upstream redesign across a near-rewritten module with incompatible ColumnBounds semantics (struct→trait, generic-param threading through HashJoinExecBuilder / Debug / DisplayAs / ExecutionPlan + ~43 HashJoinExec reference sites), which is a re-architecture rather than a restore. Flagging explicitly rather than silently skipping. Re-enabling this needs a separate decision: either (a) fork 53.1's centralized dynamic-filter pipeline back to the trait-delegated model, or (b) adapt cayenne to 53.1's native InList/hash-table pushdown + a byte-budget config (spice2-side, out of scope here).

Build / test

• cargo check -p datafusion -p datafusion-datasource -p datafusion-catalog-listing -p datafusion-execution -p datafusion-physical-plan — clean (EXIT0).
• cargo test -p datafusion-datasource --lib — 107 passed, 0 failed (covers projection / table_schema).
• cargo test -p datafusion-execution --lib cache — (see PR thread).

Audit context

• Old line (all 4 families wired): spiceai-52.5 @ 539e6963.
• Reconcile that dropped them: 289b74a64 ("Carry Spice patches forward to DataFusion 53.1.0").
• New base: spiceai-53-patches @ 0c648eb06.

[Copilot]

Pull request overview

This PR reintroduces several “Spice” extension patch families that were dropped during the DataFusion 53.1.0 reconcile, focusing on (1) per-file metadata columns, (2) parquet/object-store object versioning propagation, and (3) version/etag-aware cache invalidation. It also refreshes Cargo.lock to ensure the spiceai/arrow-rs forked Arrow/Parquet crates are actually used in the dependency graph.

Changes:

• Thread metadata columns (_location, _size, _last_modified) through TableSchema/SplitProjection/ProjectionOpener and into file sources (parquet/csv/json/avro/arrow), including parquet literal injection in ParquetOpener.
• Add ObjectVersionType plumbing from ListingOptions → FileScanConfig → parquet reader factories (ParquetObjectReader::new_with_meta(...).with_object_versioning_type(...)).
• Restore version/etag-aware cache validation by introducing is_same_file_version and consulting it from cache validity checks.

Reviewed changes

Copilot reviewed 17 out of 18 changed files in this pull request and generated 4 comments.

Show a summary per file

File	Description
datafusion/execution/src/cache/cache_unit.rs	Adds is_same_file_version helper for version/etag-aware cache invalidation.
datafusion/execution/src/cache/cache_manager.rs	Incorporates is_same_file_version into cached-metadata validity checks.
datafusion/datasource/src/table_schema.rs	Extends TableSchema to append metadata columns after partition columns.
datafusion/datasource/src/projection.rs	Adds metadata-column classification/mapping and literal injection in ProjectionOpener / SplitProjection.
datafusion/datasource/src/file.rs	Extends FileSource with with_metadata_cols hook.
datafusion/datasource/src/file_scan_config.rs	Adds builder support for metadata columns and object_versioning_type plumbing.
datafusion/datasource/Cargo.toml	Enables parquet/object_store under the datasource parquet feature.
datafusion/datasource-parquet/src/source.rs	Implements FileSource::with_metadata_cols for parquet and recomputes default projection.
datafusion/datasource-parquet/src/reader.rs	Adds object_versioning_type to parquet reader factories and uses new_with_meta.
datafusion/datasource-parquet/src/opener.rs	Injects metadata columns into parquet literal substitution path.
datafusion/datasource-parquet/src/file_format.rs	Wires scan config object_versioning_type into cached parquet reader factory.
datafusion/datasource-json/src/source.rs	Implements metadata columns and updates projection pushdown to use table-schema-aware split projection.
datafusion/datasource-csv/src/source.rs	Implements metadata columns and updates projection pushdown to use table-schema-aware split projection.
datafusion/datasource-avro/src/source.rs	Implements metadata columns and updates projection pushdown to use table-schema-aware split projection.
datafusion/datasource-arrow/src/source.rs	Implements metadata columns and updates projection pushdown to use table-schema-aware split projection.
datafusion/catalog-listing/src/table.rs	Wires listing options into scan config and appends metadata columns to the scan TableSchema.
datafusion/catalog-listing/src/options.rs	Adds metadata_cols, object_versioning_type, and metadata validation helper APIs.
Cargo.lock	Refreshes lockfile so Arrow/Parquet resolve to the intended fork versions and updates transitive deps.

Comments suppressed due to low confidence (1)

datafusion/datasource/src/table_schema.rs:233

• TableSchema::table_schema doc comment is now outdated: the returned schema includes metadata columns as well (appended after partition columns), not just file schema + partition columns.

    /// Get the full table schema (file schema + partition columns).
    ///
    /// This is the complete schema that will be seen by queries, combining
    /// both the columns from the files and the partition columns.

---

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

[lukekim]

Decision on the un-ported family 4 (hash-join custom accumulator): we are adopting DF53.1's native dynamic-filter pushdown (PushdownStrategy / create_membership_predicate) on the spice2 side instead of re-architecting this fork back to the trait seam — per the principle of taking full advantage of upstream improvements rather than carrying forked mechanisms. The cayenne probe-join optimization is being re-implemented against the native API on the upgrade branch (spiceai/spiceai#11118). This PR's 3 restored families stand as-is.