Merge DataFusion 54.1.0 into spiceai-54#187
Merged
Merged
Conversation
…trip (backport apache#22104) (apache#22785) ## Which issue does this PR close? - Backport of apache#22104 to `branch-54` (for 54.1.0, tracked in apache#22547). This PR: - Backports apache#22104 to the `branch-54` line so the `null_aware` proto round-trip fix ships in 54.1.0, as requested in apache#22065 (comment) Clean cherry-pick; `datafusion-proto` builds and both round-trip regression tests pass on `branch-54`.
…ark's width_bucket to i64) (apache#23087) backport apache#22811 to `branch-54` in anticipation of 54.1.0 release. DataFusion Comet encountered this issue while trying to upgrade to DF 54.0.0, so it would ease that upgrade for Comet if this were backported. Co-authored-by: Eduardo Aguilar <contact@eduaguilar.com>
…en row-group statistics already prove it cannot prune) (apache#23088) backport apache#22857 to `branch-54` in anticipation of 54.1.0 release. DataFusion Comet encountered this issue while trying to upgrade to DF 54.0.0, so it would ease that upgrade for Comet if this were backported. This one had conflicts on cherry-pick, so I could use a sanity check on the review by comparing to the original PR. --------- Co-authored-by: Ratul Dawar <ratuldawar11@gmail.com>
…th NULLs (apache#23196) ## Which issue does this PR close? <!-- We generally require a GitHub issue to be filed for all bug fixes and enhancements and this helps us generate change logs for our releases. You can link an issue to this PR using the GitHub syntax. For example `Closes #123` indicates that this PR will close issue #123. --> - Closes #. ## Rationale for this change Backport `array_compact` handle edge case with NULLs. The issue causing correctness issues and has no breaking API, therefore should fall into backport criteria <!-- Why are you proposing this change? If this is already explained clearly in the issue then this section is not needed. Explaining clearly why changes are proposed helps reviewers understand your changes and offer better suggestions for fixes. --> ## What changes are included in this PR? <!-- There is no need to duplicate the description in the issue here but it is sometimes worth providing a summary of the individual changes in this PR. --> ## Are these changes tested? <!-- We typically require tests for all PRs in order to: 1. Prevent the code from being accidentally broken by subsequent changes 2. Serve as another way to document the expected behavior of the code If tests are not included in your PR, please explain why (for example, are they covered by existing tests)? --> ## Are there any user-facing changes? <!-- If there are user-facing changes then we may require documentation to be updated before approving the PR. --> <!-- If there are any breaking changes to public APIs, please add the `api change` label. -->
… config (apache#23296) ## Which issue does this PR close? - Backport of apache#23294 to `branch-54`. - Closes apache#23293 for the 54 release line. ## Rationale for this change `FileStream` sibling work-stealing (`WorkSource::Shared`) seeds one shared work queue from every file group and lets whichever output partition goes idle first steal the next unopened file (or byte-range morsel). This assumes all output partitions of a scan are polled concurrently in one process. Executors that run each output partition as an isolated task in a separate process — Ballista and datafusion-distributed — never poll the sibling partitions. The single polled partition drains the whole shared queue and reads files belonging to other partitions, so every isolated task reads the entire input and the scan output is inflated by the partition count. This is a correctness bug for those executors, not just a performance one. The existing escape hatches (`preserve_order`, `partitioned_by_file_group`) are plan-level flags on `FileScanConfig`, not something a distributed executor can set centrally through the session config, and a plain repartitioned scan does not set `partitioned_by_file_group`. There is no session-level off switch, unlike `datafusion.optimizer.enable_dynamic_filter_pushdown`, which exists precisely so consumers that cannot support runtime cross-partition state can disable it. ## What changes are included in this PR? This is a backport of apache#23294. The changes are identical in intent; two conflicts were resolved for `branch-54`, which does not yet carry the `output_partitioning` field on `FileScanConfig` or the `enable_migration_aggregate` config that exist on `main`: - Add `datafusion.execution.enable_file_stream_work_stealing` (default `true`). When `false`, `FileScanConfig::create_sibling_state` returns `None`, so each partition falls back to `WorkSource::Local` and reads only its own file group. - Thread `&ConfigOptions` into `DataSource::create_sibling_state` so the flag is read from the session config at `execute` time. As a session config value it round-trips through `datafusion-proto` with no proto schema change. - Regenerate `configs.md` and add the setting to `information_schema.slt`. - Add the regression test `morsel_disabled_work_stealing_keeps_files_local` using the file's standard morsel snapshot harness. ## Are these changes tested? Yes. The `file_stream` test suite (including the new `morsel_disabled_work_stealing_keeps_files_local`) passes, and `cargo clippy` is clean. The existing sibling work-stealing tests continue to pass with the default. ## Are there any user-facing changes? A new session config, `datafusion.execution.enable_file_stream_work_stealing` (default `true`), so existing behavior is unchanged. `DataSource::create_sibling_state` gains a `&ConfigOptions` parameter (an API change for anyone implementing the `DataSource` trait directly).
…le group split (backport apache#23277) (apache#23340) ## Which issue does this PR close? - Backport of apache#23277 to `branch-54` (for 54.1.0, tracked in apache#22547). Clean cherry-pick, all tests pass.
…pache#22815, adapted) (apache#23576) ## Which issue does this PR close? - Backport of apache#22815 to `branch-54` (for 54.1.0, tracked in apache#22547). - Fixes wrong results apache#22796 (regression introduced by the inline-u128 hashing fast path in apache#21064, which is present on `branch-54` as commit 1cb4de4). ## Rationale for this change `approx_distinct` over-counts `Utf8View` values when the same short (<= 12 byte) string appears across batches with different layouts: an all-inline batch hashes it as a raw `u128` view, while a mixed batch (with a data buffer) hashed it as a `&str`, so the same value is counted twice. ## What changes are included in this PR? This is an **adapted** backport, not a clean cherry-pick. `branch-54` only has the scalar `StringViewHLLAccumulator` path; the group `HllGroupsAccumulator` / `Utf8ViewHasher` code the upstream fix also touched comes from apache#22768, which is **not** on `branch-54`. - Applied verbatim from apache#22815: the scalar `StringViewHLLAccumulator::update_batch` mixed-batch fix (per-row length check) and its regression test `utf8view_acc_split_batches_match_single_mixed_batch`. - Omitted as not applicable: the `Utf8ViewHasher` group-accumulator hunk and its group test. - Added: a `#[cfg(test)] mod tests` scaffold and imports, since `branch-54`'s `approx_distinct.rs` had no test module. ## Are these changes tested? Yes. The scalar regression test passes with the fix and fails without it (over-count), confirming it exercises the regression. ## Are there any user-facing changes? `approx_distinct` on `Utf8View` / `VARCHAR VIEW` columns now returns correct (lower) counts. Results may differ from the previously incorrect values. No API changes. Co-authored-by: Huaijin <haohuaijin@gmail.com> Co-authored-by: Yongting You <2010youy01@gmail.com>
…ache#22950, adapted) (apache#23573) ## Which issue does this PR close? - Backport of apache#22950 to `branch-54` (for 54.1.0, tracked in apache#22547). - Fixes panic apache#22935 (reporter bisected to a regression between 53.1.0 and 54.0.0; introduced by apache#20047). ## Rationale for this change Anonymous file reads can read the same path with different explicit schemas in one session. The shared file-statistics cache did not validate that cached statistics matched the read schema, so a narrower cached entry could be reused for a later wider-schema read and panic during statistics projection (`index out of bounds` in projection). No API change, so it fits the backport criteria. ## What changes are included in this PR? This is an **adapted** backport. The core fix (`table.rs`, `options.rs`, `context/mod.rs`) is applied; two conflicts were resolved for `branch-54`'s API shape: - `ListingTable::statistics_cache()` returns `Option<&Arc<dyn FileStatisticsCache>>` (on `branch-54`, `FileStatisticsCache` is a trait, not a concrete type as on `main`). - Kept `branch-54`'s import paths for `TableScopedPath` / `FileStatisticsCache`. - The regression test `anonymous_parquet_stats_cache_with_explicit_wider_schema` computes statistics via `plan.partition_statistics(None)` instead of `statistics_with_args`, which does not exist on `branch-54`. ## Are these changes tested? Yes. The regression test passes with the fix and, without it, reproduces the exact reported panic (`index out of bounds: the len is 2 but the index is 2`), confirming it exercises the regression. ## Are there any user-facing changes? No API change. Fixes a panic on anonymous multi-schema reads of the same path. Co-authored-by: Kumar Ujjawal <ujjawalpathak6@gmail.com>
…error (backport apache#22915) (apache#23575) ## Which issue does this PR close? - Backport of apache#22915 to `branch-54` (for 54.1.0, tracked in apache#22547). - Fixes planning failure apache#22477 (regression introduced in 54.0.0 by apache#21265). ## Rationale for this change `= ANY (SELECT ...)` and `<> ALL (SELECT ...)` fail to plan in 54.0.0 with `duplicate unqualified field name mark`. Empirically verified working on the published 53.1.0 crate. No API changes, so it fits the backport criteria. ## What changes are included in this PR? Clean cherry-pick of apache#22915. No adaptation required. ## Are these changes tested? Yes. Carries the original regression coverage (`subquery.slt`), all tests pass. ## Are there any user-facing changes? These subquery forms plan again. No API changes. Co-authored-by: H <25857835+HairstonE@users.noreply.github.com>
…s with multiple probe partitions (backport apache#22791) (apache#23577) ## Which issue does this PR close? - Backport of apache#22791 to `branch-54` (for 54.1.0, tracked in apache#22547). - Regression introduced in 54.0.0 by apache#21448. ## Rationale for this change `NestedLoopJoinExec` emits spurious NULL-padded unmatched-left rows when there are multiple probe partitions, producing wrong results in 54.0.0. No API changes, correctness-only fix, so it fits the backport criteria. ## What changes are included in this PR? Clean cherry-pick of apache#22791. No adaptation required. ## Are these changes tested? Yes. Carries the original regression coverage (`joins.slt`), all tests pass. ## Are there any user-facing changes? Nested loop joins with multiple probe partitions no longer emit spurious rows. No API changes. Co-authored-by: Nathan <56370526+nathanb9@users.noreply.github.com>
…rong results (backport apache#22727) (apache#23578) ## Which issue does this PR close? - Backport of apache#22727 to `branch-54` (for 54.1.0, tracked in apache#22547). - Fixes wrong results apache#22726 (regression introduced in 54.0.0 by apache#20581, which extended the rewrite to these types). ## Rationale for this change Anchored-regex simplification produces wrong results or errors on `Utf8View` and `LargeUtf8` columns in 54.0.0. No API changes, correctness-only fix, so it fits the backport criteria. ## What changes are included in this PR? Clean cherry-pick of apache#22727. No adaptation required. ## Are these changes tested? Yes. Carries the original regression coverage (`predicates.slt`), all tests pass. ## Are there any user-facing changes? Anchored regex predicates on `Utf8View`/`LargeUtf8` now return correct results. No API changes. Co-authored-by: linfeng <33561138+lyne7-sc@users.noreply.github.com>
…as (backport apache#22552) (apache#23579) ## Which issue does this PR close? - Backport of apache#22552 to `branch-54` (for 54.1.0, tracked in apache#22547). - Fixes execution failure apache#22034 (regression introduced in 54.0.0 by apache#21912, which made nullability checking strict without fixing schema derivation). ## Rationale for this change Recursive CTE queries with certain nullability patterns fail at execution in 54.0.0. This restores correct schema derivation for recursive queries. Reviewer note: this has a larger blast radius than the other 54.1.0 backports. It touches 12 files, adds a field to `LogicalPlan::RecursiveQuery`, and includes proto changes (`proto/src/logical_plan/mod.rs`). Worth a deliberate call on whether it clears the patch-release stability bar. ## What changes are included in this PR? Clean cherry-pick of apache#22552. No adaptation required. ## Are these changes tested? Yes. Carries the original regression coverage (`cte.slt`, `explain_tree.slt`, `explain_analyze.rs`), all tests pass. ## Are there any user-facing changes? Recursive CTEs with these nullability patterns execute again. Adds a field to `LogicalPlan::RecursiveQuery` and the corresponding proto messages (internal, but on public-facing structs). Co-authored-by: Neil Conway <neil.conway@gmail.com>
…backport apache#22444) (apache#23580) ## Which issue does this PR close? - Backport of apache#22444 to `branch-54` (for 54.1.0, tracked in apache#22547). - Fixes wrong results apache#22441 (regression introduced in 54.0.0 by apache#21549). ## Rationale for this change `IS TRUE` in `EliminateOuterJoin` wrongly converts outer joins to inner joins, so 54.0.0 drops rows that 53.1.0 returned. Empirically verified: 53.1.0 returns correct rows, 54.0.0 does not. No API changes, correctness-only fix, so it fits the backport criteria. ## What changes are included in this PR? Clean cherry-pick of apache#22444. No adaptation required. ## Are these changes tested? Yes. Carries the original regression coverage (`eliminate_outer_join.slt`), all tests pass. ## Are there any user-facing changes? Affected outer-join queries now return the correct rows. No API changes. Co-authored-by: Neil Conway <neil.conway@gmail.com> Co-authored-by: Kumar Ujjawal <ujjawalpathak6@gmail.com>
…batches (backport apache#23049) (apache#23574) ## Which issue does this PR close? - Backport of apache#23049 to `branch-54` (for 54.1.0, tracked in apache#22547). - Fixes wrong results apache#23048 (regression introduced in 54.0.0 by the SMJ bitwise-path rewrite apache#20806). ## Rationale for this change Sort-merge join semi/anti drops matches across pending outer batches, producing wrong results in 54.0.0. No API changes, correctness-only fix, so it fits the backport criteria. ## What changes are included in this PR? Clean cherry-pick of apache#23049. No adaptation required. ## Are these changes tested? Yes. Carries the original SMJ regression tests, all tests pass. ## Are there any user-facing changes? Semi/anti sort-merge joins now return the correct rows. No API changes. Co-authored-by: Neil Conway <neil.conway@gmail.com>
…function (backport apache#23481) (apache#23582) ## Which issue does this PR close? - Backport of apache#23481 to `branch-54` (for 54.1.0, tracked in apache#22547). ## Rationale for this change Performance improvement with minimal blast radius. ## What changes are included in this PR? Clean cherry-pick of apache#23481. No adaptation required. ## Are these changes tested? Yes, existing tests. ## Are there any user-facing changes? No. Co-authored-by: Andy Grove <agrove@apache.org>
…ojection into file scan (backport apache#23395, adapted) (apache#23585) ## Which issue does this PR close? - Backport of apache#23395 to branch-54 (apache#22547) - Closes apache#23220. ## Rationale for this change The reported bug showed that when volatile (non-deterministic) expressions are referenced multiple times in an outer function the query results can be incorrect. To produce correct results, a volatile expression (e.g. `random()`, `uuid()`) aliased once in a subquery and referenced multiple times must be evaluated once and reused. Since 52.0.0 this evaluation pattern for volatile functions has been broken; the physical projection-pushdown rule merges the outer projection into the file `DataSourceExec`, inlining the aliased volatile expression at each reference site. Instead of being evaluated once, the volatile expression is evaluated N times, so the references diverge: ```sql SELECT s.r AS x, s.r AS y FROM (SELECT random() AS r FROM t) AS s; -- x != y on >= 52.0.0 ``` This was correct in 51.0.0 and regressed in 52.0.0/53.0.0. It reproduces on file scans (Parquet/CSV) but not in-memory tables, and was surfaced downstream in Ibis. Worth noting, apache#10337 appears to report the same class of bug but a different cause (I'll try to look into that issue soon as well). ## What changes are included in this PR? `FileScanConfig::try_swapping_with_projection` now declines to merge a projection into the file source when the merge would inline a volatile expression that the incoming projection references. The check reuses the existing `is_volatile()` utility from `datafusion_physical_expr_common` — the same volatility gate the physical `ProjectionPushdown` and `FilterPushdown` rules already apply — so file-source projection merging is now consistent with them. Deterministic expressions still merge freely. The guard blocks when a volatile inner expression is referenced at least once (not only more than once), because a single outer expression can itself duplicate the value (e.g.`r + r`). ## Are these changes tested? Yes: - Unit tests in `file_scan_config.rs`: - volatile referenced ≥1× → blocked; - deterministic computed and column-only → allowed; - unreferenced volatile → allowed; - single-expression self-reference (`r + r`) → blocked; - volatile nested in arithmetic → blocked; - empty → allowed. - A regression test in `projection_pushdown.slt` asserting `x = y` for the aliased-`random()` pattern over a Parquet scan. - Full `datafusion-sqllogictest` suite passes. ## Are there any user-facing changes? No API changes! Query results for the affected pattern are corrected (a volatile expression aliased in a subquery is no longer duplicated by projection pushdown into a file scan). Physical plans for such queries now retain a `ProjectionExec` above the scan rather than inlining the volatile expression into the `DataSourceExec` projection. Co-authored-by: Matt Butrovich <mbutrovich@users.noreply.github.com>
## Which issue does this PR close? - Part of apache#22547 (54.1.0 release) ## Rationale for this change `cargo audit` is failing on `branch-54` with 7 vulnerabilities across 5 crates: - `crossbeam-epoch` 0.9.18: [RUSTSEC-2026-0204](https://rustsec.org/advisories/RUSTSEC-2026-0204) - `postgres-protocol` 0.6.11: [RUSTSEC-2026-0179](https://rustsec.org/advisories/RUSTSEC-2026-0179) (high), [RUSTSEC-2026-0180](https://rustsec.org/advisories/RUSTSEC-2026-0180) - `quinn-proto` 0.11.14: [RUSTSEC-2026-0185](https://rustsec.org/advisories/RUSTSEC-2026-0185) (high) - `tokio-postgres` 0.7.17: [RUSTSEC-2026-0178](https://rustsec.org/advisories/RUSTSEC-2026-0178) - `quick-xml` 0.39.2: [RUSTSEC-2026-0194](https://rustsec.org/advisories/RUSTSEC-2026-0194) (high), [RUSTSEC-2026-0195](https://rustsec.org/advisories/RUSTSEC-2026-0195) (high) This follows the model of similar fixes on earlier maintenance branches: - apache#21415 (branch-52) - apache#21587 (branch-53) ## What changes are included in this PR? - `cargo update` for crates that have dependencies - Add ignore for the two `quick-xml` advisories. The fix requires `quick-xml` 0.41, which is only used by `object_store` 0.14 — a breaking upgrade that can not be backported to this maintenance branch. The same advisories are ignored on `main` (added in apache#23298, tracked by apache#23297). ## Are these changes tested? Yes, by CI. `cargo audit` now passes locally with the ignore flags (only allowed warnings remain). ## Are there any user-facing changes? No, dependency version bumps in `Cargo.lock` and a CI change only. Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
apache#23654) ## Which issue does this PR close? - part of apache#22547 54.x branch backport of fix for apache#23447 ## Rationale for this change See main PR apache#23522 ## What changes are included in this PR? See main PR apache#23522 ## Are these changes tested? Additional test cases added to verify existing behaviour Fix manually tested with reproduction code from apache#23447 ## Are there any user-facing changes? No
…xtract, spark array_repeat (backport apache#23071) (apache#23629) ## Which issue does this PR close? - Backport of apache#23071 (clean cherry-pick of non-merge commits) - Closes apache#23068 . ## Rationale for this change Unresolved lambda variables and unspecified placeholders report a `DataType::Null` data type, which causes an error when used as argument of a function that doesn't handle nulls during type coercion ## What changes are included in this PR? Handle null in `coerce_single_list_arg` for higher-order functions, `map_extract` and in spark `array_repeat`. All other functions (scalar, window, agg and higher-order) that uses user defined type coercion have been checked to currently handle null args. ## Are these changes tested? Test passing `array_filter` with an unresolved lambda variable to `Dataframe::with_column` (which indirectly calls `value_fields_with_higher_order_udf_and_lambdas` coercion via `Projection::try_new`, `projection_schema`, `exprlist_to_fields`, `Expr::to_field` ), and sqllogictests ## Are there any user-facing changes? No
# Conflicts: # Cargo.lock
There was a problem hiding this comment.
Pull request overview
Merges upstream Apache DataFusion 54.1.0 into the spiceai-54 maintenance branch, updating version references and incorporating upstream bugfixes (planner/proto correctness, join behavior, spilling/concurrency fixes, Parquet metadata/pruning behavior), alongside expanded regression coverage in SQLLogicTests and Rust unit tests.
Changes:
- Bump crate versions and documentation examples from 54.0.0 → 54.1.0 and add the 54.1.0 changelog entry.
- Incorporate upstream engine fixes across SQL planning, optimizer rules, physical execution (joins/spill), functions, and Parquet reading/metrics.
- Add/adjust regression tests (sqllogictest + Rust tests) for fixed edge cases (recursive CTE nullability, mark joins, volatility pushdown, proto join fields, etc.).
Reviewed changes
Copilot reviewed 66 out of 69 changed files in this pull request and generated 2 comments.
Show a summary per file
| File | Description |
|---|---|
| docs/source/user-guide/example-usage.md | Update dependency example to 54.1.0 |
| docs/source/user-guide/crate-configuration.md | Update version in backtrace feature example |
| docs/source/user-guide/configs.md | Update created_by example; document new work-stealing config |
| docs/source/download.md | Update dependency example to 54.1.0 |
| dev/changelog/54.1.0.md | Add 54.1.0 changelog content |
| datafusion/sqllogictest/test_files/subquery.slt | Regression for = ANY / <> ALL on same table |
| datafusion/sqllogictest/test_files/spark/array/array_repeat.slt | Add NULL-count coverage |
| datafusion/sqllogictest/test_files/projection_pushdown.slt | Regression for volatile expr duplication under pushdown |
| datafusion/sqllogictest/test_files/predicates.slt | Expanded anchored-regex simplification coverage (Utf8View + cases) |
| datafusion/sqllogictest/test_files/map.slt | NULL input coverage for map_extract |
| datafusion/sqllogictest/test_files/joins.slt | Regression for NLJ unmatched-left emission with multi partitions |
| datafusion/sqllogictest/test_files/information_schema.slt | Surface new config in information_schema |
| datafusion/sqllogictest/test_files/explain_tree.slt | Snapshot update for CASTed literals in plan |
| datafusion/sqllogictest/test_files/eliminate_outer_join.slt | Coverage for NOT(… IS TRUE/FALSE/…) semantics |
| datafusion/sqllogictest/test_files/dynamic_filter_pushdown_config.slt | Snapshot update for new parquet metric field |
| datafusion/sqllogictest/test_files/cte.slt | Snapshot updates + recursive CTE nullability widening tests |
| datafusion/sqllogictest/test_files/array/array_transform.slt | NULL input coverage |
| datafusion/sqllogictest/test_files/array/array_filter.slt | NULL input coverage |
| datafusion/sqllogictest/test_files/array/array_distinct.slt | array_compact edge cases for NullArray / nested all-null lists |
| datafusion/sqllogictest/test_files/array/array_any_match.slt | NULL input coverage |
| datafusion/sql/src/cte.rs | Make recursive work-table schema conservative-nullable |
| datafusion/spark/src/function/math/width_bucket.rs | Spark width_bucket return type fix (i64) + test updates |
| datafusion/spark/src/function/array/slice.rs | Avoid allocation by using value_length |
| datafusion/spark/src/function/array/repeat.rs | Handle NULL in repeat-count coercion |
| datafusion/proto/tests/cases/roundtrip_logical_plan.rs | Regression tests for join null-awareness / null-equality roundtrip |
| datafusion/proto/src/logical_plan/mod.rs | Preserve join null flags on decode; validate key length; recompute recursive schema via constructor |
| datafusion/proto/src/generated/prost.rs | Add null_aware to JoinNode message |
| datafusion/proto/src/generated/pbjson.rs | JSON (de)serialization for null_aware |
| datafusion/proto/proto/datafusion.proto | Protobuf schema: add null_aware field |
| datafusion/physical-plan/src/spill/spill_pool.rs | Fix spill pool concurrency/wakeup; support concurrent writers; add regression test |
| datafusion/physical-plan/src/repartition/mod.rs | Regression test for preserve_order across spill file rotation |
| datafusion/physical-plan/src/recursive_query.rs | Thread output schema through RecursiveQueryExec constructor |
| datafusion/physical-plan/src/joins/sort_merge_join/tests.rs | Regression for Pending boundary re-entry (no-filter semi/anti joins) |
| datafusion/physical-plan/src/joins/sort_merge_join/bitwise_stream.rs | Clear stale outer batch before polling next batch |
| datafusion/physical-plan/src/joins/nested_loop_join.rs | Guard probe-completion reporting to prevent premature unmatched-left emission |
| datafusion/optimizer/src/simplify_expressions/regex.rs | Correct anchored-regex simplification for case-insensitive & Utf8View literals |
| datafusion/optimizer/src/optimize_projections/mod.rs | Ensure mark joins keep a qualifying column even when projection is empty |
| datafusion/optimizer/src/eliminate_outer_join.rs | Improve null-rejecting predicate analysis for outer-join elimination |
| datafusion/functions-nested/src/map_extract.rs | Make map_extract handle NULL arg types/values |
| datafusion/functions-nested/src/lambda_utils.rs | Coercion helper: treat NULL list arg consistently |
| datafusion/functions-nested/src/array_compact.rs | Use logical nulls to handle NullArray correctly |
| datafusion/functions-nested/src/array_any_match.rs | Reuse list-arg coercion helper |
| datafusion/functions-aggregate/src/approx_distinct.rs | Fix Utf8View hashing consistency; add regression test |
| datafusion/expr/src/planner.rs | Clarify recursive CTE work-table API docs |
| datafusion/expr/src/logical_plan/tree_node.rs | Preserve derived recursive-query schema under rewrites |
| datafusion/expr/src/logical_plan/plan.rs | Store recursive-query schema; recompute on schema reconciliation; drop FDs |
| datafusion/expr/src/logical_plan/builder.rs | Build recursive query via RecursiveQuery::try_new |
| datafusion/datasource/src/statistics.rs | Return explicit error when partition stats are missing + unit test |
| datafusion/datasource/src/source.rs | Pass session config into sibling-state creation |
| datafusion/datasource/src/file_stream/mod.rs | Tests + wiring for disabling file-stream work stealing |
| datafusion/datasource/src/file_scan_config/mod.rs | Volatility guard in projection pushdown; honor work-stealing config in sibling-state creation; add tests |
| datafusion/datasource/Cargo.toml | Add dev-dep used by new tests |
| datafusion/datasource-parquet/src/reader.rs | Forward page-index policy from ArrowReaderOptions into metadata fetch |
| datafusion/datasource-parquet/src/opener/mod.rs | Skip page-index I/O when stats fully match; add metric + tests |
| datafusion/datasource-parquet/src/metrics.rs | Add page_index_load_skipped metric |
| datafusion/datasource-parquet/src/metadata.rs | Page-index policy + cache-aware metadata fetching / incremental page-index load |
| datafusion/core/tests/sql/explain_analyze.rs | Update snapshot for CASTed literals in recursive plan |
| datafusion/core/tests/parquet/file_statistics.rs | Stats-cache correctness with anonymous specified-schema reads |
| datafusion/core/tests/dataframe/mod.rs | DataFrame regression for lambda variable resolution |
| datafusion/core/src/physical_planner.rs | Pass recursive-query schema into RecursiveQueryExec |
| datafusion/core/src/execution/context/mod.rs | Respect inferred vs specified schema source when reading |
| datafusion/core/src/datasource/file_format/options.rs | Track whether schema was inferred or specified |
| datafusion/common/src/config.rs | Add config: enable_file_stream_work_stealing |
| datafusion/catalog/src/cte_worktable.rs | Clarify work-table schema semantics for recursive CTEs |
| datafusion/catalog-listing/src/table.rs | Avoid stats-cache reuse for anonymous specified-schema reads |
| datafusion-cli/src/main.rs | Update metadata-cache hit-count snapshots |
| Cargo.toml | Bump workspace crate versions to 54.1.0 |
| Cargo.lock | Regenerated lockfile for 54.1.0 |
| .github/workflows/audit.yml | Ignore specific quick-xml advisories with rationale |
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
sgrebnov
reviewed
Jul 24, 2026
sgrebnov
approved these changes
Jul 24, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Advances
spiceai-54to upstream DataFusion 54.1.0 (minor bump) by merging the54.1.0tag into the branch. All Spice-specific patches are preserved as-is; the merge layers the 20 upstream bugfix backports on top.The only merge conflict was
Cargo.lock(mechanical54.0.0→54.1.0), resolved by regeneration. All source files auto-merged.Upstream 54.0.0 → 54.1.0
20 backported bugfixes, no breaking API changes. Notable:
map_extract, sparkarray_repeatSpillPoolReaderwith concurrent readersNestedLoopJoinExecspurious unmatched-left rows with multiple probe partitions= ANY (SELECT ...)/<> ALL (SELECT ...)schema errorapprox_distinctover-counting forutf8viewarray_compactNULL edge case;width_bucketreturn typeValidation
Built and tested within Spice against this commit:
[patch.crates-io]unchanged and compile against 54.154.0.0→54.1.0) in 4 NSQL context snapshots — updated accordingly in the Spice PR