chore(deps): bump DataFusion to 54.1.0

[andygrove]

Which issue does this PR close?

N/A — routine dependency bump; no tracking issue was filed.

Rationale for this change

Keeps the binding current with upstream DataFusion. 54.1.0 is the latest release
line, and staying close to it keeps the next bump small and makes upstream fixes
available to Java callers.

What changes are included in this PR?

Bumps datafusion, datafusion-proto, datafusion-spark and
datafusion-substrait from 53.1.0 to 54.1.0. The datafusion.version Maven
property moves in lock step, since it selects the upstream tag the
datafusion.proto / datafusion_common.proto definitions are downloaded from —
the generated Java protobuf classes must match what datafusion-proto 54.1.0
decodes. The pinned sha512 digests for both protos are updated accordingly; each
was verified to match the copy vendored in the published datafusion-proto
/ datafusion-proto-common 54.1.0 crates, independently of the GitHub download.

arrow (58) and object_store (0.13) are unchanged — 54.1.0 resolves to the
same majors, so the object_store pin comment still holds.

Adapting to the upstream API changes the bump requires:

• TableProvider, ExecutionPlan and ScalarUDFImpl now take Any as a
  supertrait, so the manual as_any overrides are no longer trait members and
  are removed.
• MemoryPool gained a name() method and a Display supertrait.
  TrackingMemoryPool implements both, following upstream's wrapper convention:
  name the wrapper, add the counters it exists to expose, and defer to the inner
  pool for the usage detail.
• CacheManagerConfig::table_files_statistics_cache is renamed to
  file_statistics_cache, and the accompanying limit is now the on/off switch —
  CacheManager::try_new installs a default statistics cache whenever
  file_statistics_cache_limit > 0, even when the cache slot is None, and the
  default limit is non-zero. An explicit fileStatisticsCache(false) from the
  Java surface therefore has to zero the limit as well; otherwise upstream would
  install the very cache the caller asked us to skip. This is the one place the
  bump would have silently changed observable Java behavior.
• The cache_unit module is gone; the default cache impls now live in
  cache::file_statistics_cache and cache.
• DataFusionError::AvroError is gone — DataFusion 54 reads Avro through
  arrow-avro rather than apache-avro. Avro decode failures now arrive as
  ArrowError::AvroError, which the exception classifier already routed to
  ExecutionException alongside the CsvError / JsonError decoder variants,
  so the mapping stays coherent. The dead arm is dropped, along with the avro
  feature on datafusion-jni-common that existed only to gate it.

One test fixture is also corrected. SessionContextSubstraitTest built plans
whose base schema declared both columns NULLABILITY_REQUIRED, while the tests
register a CSV — whose inferred schema is always nullable. DataFusion 54's
Substrait consumer now validates that a field a plan declares non-nullable
really is non-nullable in the table, and rejects the mismatch. That check is
correct: a plan built around a "never null" assumption must not run against data
that can contain nulls. The fixture is fixed to declare nullable columns; it
only passed before because 53 did not check.

Are these changes tested?

Covered by the existing suites — this is a dependency bump, so the value is in
the current tests continuing to pass against the new version rather than in new
assertions.

• ./mvnw test — 349 tests, 0 failures. Run with the substrait Cargo feature
  enabled (cargo build -p datafusion-jni --features substrait) so the
  Substrait tests execute rather than skip.
• cargo test --workspace — all passing.
• cargo clippy --workspace --all-targets -- -D warnings — clean.
• cargo fmt --all -- --check and ./mvnw -q spotless:check — clean.
• cargo build --workspace --all-features with RUSTFLAGS="--cfg tokio_unstable",
  to cover the optional substrait and runtime-metrics features that the
  default build does not compile.

Are there any user-facing changes?

No API changes. Two behavioral notes, both inherited from upstream:

• Avro decode failures now surface as ExecutionException rather than
  IoException, following the move to arrow-avro. The IoException javadoc
  is updated to match.
• DataFusion 54 enables the file statistics and list files caches by default.
  Callers who never configured CacheManagerOptions pick up upstream's new
  defaults; an explicit fileStatisticsCache(false) continues to disable the
  cache, as described above.

[milenkovicm]

LGNM, thanks @andygrove

[andygrove]

Merged. Thanks @milenkovicm!