deps: bump to datafusion 55.1.0

[comphead]

Which issue does this PR close?

N/A. Routine dependency bump, no issue filed.

Rationale for this change

Pick up the bugfixes in DataFusion 55.1.0. It is a patch release over 55.0.0
(10 commits, all fixes, no API changes). The ones most relevant to Comet:

• fix: make UnnestExec respect datafusion.execution.batch_size (#24529)
• fix: apply struct field filters when the file schema needs adaptation (#24530)
• fix: adapt input batches with stricter nested nullability to planned schema in aggregation (#24699)
• Fix panic on RightMark hash joins when propagating ordering (#24759)
• fix(common): support empty struct in ScalarValue::compact and new_default (#24876)
• Align metadata propagation through Physical and Logical casts (#24875)
• fix: preserve projection metadata during optimization (#24992)

Full changelog: https://github.com/apache/datafusion/blob/55.1.0/dev/changelog/55.1.0.md

What changes are included in this PR?

• native/Cargo.toml: datafusion, datafusion-datasource, datafusion-physical-expr-adapter
  and datafusion-spark bumped 55.0.0 -> 55.1.0.
• native/core/Cargo.toml: datafusion-functions-nested bumped 55.0.0 -> 55.1.0.
• contrib/delta/native/Cargo.toml: datafusion bumped 55.0.0 -> 55.1.0.
• Both lockfiles refreshed with cargo update.

No Comet source changes were needed. arrow, parquet and object_store are
unchanged in native/Cargo.lock (59.3.0 / 59.3.0 / 0.13.2); the remaining lock
churn there is patch-level bumps of transitive crates (aws-sdk-*, reqwest,
rustls, wasm-bindgen, zstd, ...).

One thing worth a reviewer's eye: contrib/delta/native/Cargo.lock had drifted
out of sync with its own manifest. It still pinned datafusion 54.0.0 (and
arrow/parquet 58.3.0) while contrib/delta/native/Cargo.toml already asked
for 55.0.0, so a --locked build of that crate could not have succeeded.
Regenerating it moves that crate to datafusion 55.1.0 and arrow/parquet
59.3.0, matching what the native/ workspace already resolves for it. That
accounts for most of the diff in that file.

How are these changes tested?

Existing CI. Locally verified on macOS/aarch64:

• cargo check --workspace --all-targets --locked — clean
• cargo clippy --workspace --all-targets --locked -- -D warnings — clean
• cargo fmt --all -- --check — clean
• cargo check -p datafusion-comet --features contrib-delta --locked — clean
• cargo check --locked --all-targets in contrib/delta/native — clean

No JVM or Spark SQL suites were run locally; relying on CI for those.

[sunchao]

Correctness

Reviewed c31138028baf against f29a236128b3. I found no verified P1/P2 regression in this dependency upgrade. All three manifests change only DataFusion version requirements; their feature selections and build profiles stay the same. Both lockfiles consistently resolve DataFusion 55.1.0. The main workspace retains Arrow/Parquet 59.3.0 and object_store 0.13.2; the standalone Delta lock moves from its stale 54.0.0/58.3.0 resolution to the versions already required by the updated manifests.

I checked the upstream release diff, including cast metadata, projection metadata, schema adaptation, nested-nullability handling and empty structs. Comet's Spark schema adapter still replaces general DataFusion casts with Spark-compatible conversions and retains only approved structural narrowing. The new upstream struct-filter rewrite specifically matches GetFieldFunc; Comet's ordinary struct access uses its own ordinal-based GetStructField, so that fix should not be counted as blanket coverage of Comet's nested filters.

CI evidence is partial as of September 11, 16:40 UTC: the Delta build gate passed both default and contrib-enabled native builds and symbol checks. The Spark 4.1 build passed with tests explicitly skipped. The subsequently completed native unit job passed 1,362 tests with 5 skipped, including the struct-field nullability tests. All three logs identify merge fa7511c3dbbc, whose complete tree equals this PR head. Spark SQL 3.5/4.1 and Iceberg runtime coverage remain pending; the Spark SQL 3.4/4.0 workflows were skipped. I ran source/lock consistency checks locally, not product builds or Spark suites. The author's local --locked checks remain author-reported. Maintained Spark 3.4/4.1 source branches were unavailable, so this review does not establish their full compatibility.

Performance

The upstream unnest batching fix does not itself change Comet's explode path: the planner still constructs its specialized ExplodeExec. Its existing batching, outer/null handling and contiguous-run optimizations remain in place. The upstream nested batch adaptation is guarded by schema mismatch/containment in MemoryStream; this is not a new conversion inserted into every Comet aggregate input. There are no upstream hash-join spill or memory-pool implementation changes in the release diff.

The lock refresh includes minor-version transitive changes, not exclusively patch increments: examples include aws-config, aws-types, serde_with and an additional zstd 0.14 dependency through compression-codecs. Comet shuffle, Arrow IPC and Parquet still select zstd 0.13.3, with both wrapper families sharing one zstd-sys package carrying Zstandard 1.5.7. I found no demonstrated compression/link regression, but no matched runtime benchmark was run, so this review makes no throughput or memory improvement claim.

Design

Keeping the DataFusion requirements aligned across the workspace, nested functions and the separately resolved Delta crate is the right boundary for this change. Refreshing the standalone lock also repairs its existing mismatch with its manifest. The Delta crate remains an inert build-gate stub; the successful feature-enabled build does not establish native Delta query support.

The release does contain API adjustments—for example, cast_with_target_field changes its field argument from an owned reference to a borrowed reference—so “no API changes” is broader than the source supports. Comet has no callsite of that changed helper, and its existing CastExpr constructors remain supported. The metadata changes preserve explicit target field information and prevent projection optimization from discarding observable metadata; no additional compatibility wrapper was needed here.

Abstraction & complexity

The PR introduces no new Comet abstraction, execution branch or configuration switch. The larger standalone lock diff follows from bringing an excluded crate's independent resolution back into alignment, rather than from a second execution-engine design. DataFusion's FFI crate is absent from both resolved dependency graphs, so its release fixes do not imply changes to Comet's JNI bridge.

The local explode fork now has an upstream batching counterpart, but retiring it is separate work: its documented Comet-specific fast paths need equivalent correctness and performance validation before replacement. Keeping that implementation unchanged makes the dependency upgrade easier to assess.