Skip to content
Closed
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
157 commits
Select commit Hold shift + click to select a range
6d157cc
Feat: Make extract (date_part) timezone aware
codetyri0n Oct 31, 2025
3f2e378
Format files.
codetyri0n Oct 31, 2025
97b0524
Make extract() time aware and register it as an independent function …
codetyri0n Nov 1, 2025
1b7f8f5
cargo fmt
codetyri0n Nov 1, 2025
924e33f
add extract.rs
codetyri0n Nov 1, 2025
d3043dc
CI fixes.
codetyri0n Nov 1, 2025
5f2f5d7
CI fixes.
codetyri0n Nov 1, 2025
be7385c
Make extract timezone aware and part of the date part udf itself
codetyri0n Nov 10, 2025
8cd5d2e
CI Fix.
codetyri0n Nov 10, 2025
6ec6e46
CI Fix (i).
codetyri0n Nov 10, 2025
cd7c5e8
Push partition_statistics into DataSource (#18233)
adriangb Oct 27, 2025
e15b056
feat: Add `output_bytes` to baseline metrics (#18268)
2010YOUY01 Oct 27, 2025
b291f33
Fix: Error rather than silently ignore extra parameter passed to ceil…
toxicteddy00077 Oct 27, 2025
9c64644
fix: Support Dictionary[Int32, Binary] for bitmap count spark functio…
kazantsev-maksim Oct 27, 2025
1c6916b
chore(deps): Update `half` to 2.7.1, ignore `RUSTSEC-2025-0111` (#18287)
alamb Oct 27, 2025
f8d05e8
chore(deps): bump taiki-e/install-action from 2.62.36 to 2.62.38 (#18…
dependabot[bot] Oct 27, 2025
b76f9eb
"Gentle Introduction to Arrow / Record Batches" #11336 (#18051)
sm4rtm4art Oct 27, 2025
bdf346e
chore(deps): bump regex from 1.11.3 to 1.12.2 (#18294)
dependabot[bot] Oct 27, 2025
98b11c0
chore(deps): bump clap from 4.5.48 to 4.5.50 (#18292)
dependabot[bot] Oct 27, 2025
ac89545
Upgrade DataFusion to arrow/parquet 57.0.0 (#17888)
alamb Oct 27, 2025
561e00b
chore(deps): bump syn from 2.0.106 to 2.0.108 (#18291)
dependabot[bot] Oct 27, 2025
1cb226e
Let `FileScanConfig` own a list of `ProjectionExpr`s (#18253)
friendlymatthew Oct 27, 2025
fe54d87
Deduplicate range/gen_series nested functions code (#18198)
Jefffrey Oct 28, 2025
8eed1fd
Enforce unique names for `is_set` on `first_value` and `last_value` (…
marc-pydantic Oct 28, 2025
556125f
fix: support float16 for `abs()` (#18304)
Jefffrey Oct 28, 2025
a4da700
chore(deps): update testcontainers to `0.25.2` and drop ignore of `RU…
DDtKey Oct 28, 2025
1f14fa3
Using `try_append_value` from arrow-rs 57.0.0 (#18313)
samueleresca Oct 28, 2025
28fb15a
feat: Introduce `PruningMetrics` and use it in parquet file pruning m…
2010YOUY01 Oct 28, 2025
b2db7ab
minor: doc fixes for timestamp output format (#18315)
Jefffrey Oct 28, 2025
1e4d25d
minor: Add documentation to function `concat_elements_utf8view` (#18316)
2010YOUY01 Oct 28, 2025
3cdcec3
chore(deps): bump taiki-e/install-action from 2.62.38 to 2.62.40 (#18…
dependabot[bot] Oct 28, 2025
66fc1f9
Add PostgreSQL-style named arguments support for scalar functions (#1…
bubulalabu Oct 28, 2025
5dc42f4
Change default prefetch_hint to 512Kb to reduce number of object stor…
zhuqi-lucas Oct 28, 2025
32b1fe3
Fix: Add projection to generate_series (#18298)
mkleen Oct 28, 2025
3edb38a
fix: Add WITH ORDER display in information_schema.views (#18282)
gene-bordegaray Oct 28, 2025
63b4c84
Do not accept null is_set for first_value/last_value (#18301)
marc-pydantic Oct 28, 2025
cfeb8fa
Optimize merging of partial case expression results (#18152)
pepijnve Oct 28, 2025
fe68e75
chore: Format examples in doc strings - execution (#18339)
CuteChuanChuan Oct 28, 2025
469e9ec
chore: Format examples in doc strings - common (#18336)
CuteChuanChuan Oct 28, 2025
d8d8ccc
feat: Improve metrics for aggregate streams. (#18325)
EmilyMatt Oct 28, 2025
44b50c3
chore: Format examples in doc strings - crate datafusion (#18333)
CuteChuanChuan Oct 29, 2025
8b6c97f
chore: Format examples in doc strings - expr (#18340)
CuteChuanChuan Oct 29, 2025
7f6a606
chore: Format examples in doc strings - datasource crates (#18338)
CuteChuanChuan Oct 29, 2025
97523e0
Insta for enforce_distrubution (easy ones) (#18248)
blaginin Oct 29, 2025
9b33c92
feat: allow pushdown of dynamic filters having partition cols (#18172)
feniljain Oct 29, 2025
68c74d3
chore: Format examples in doc strings - macros and optmizer (#18354)
CuteChuanChuan Oct 29, 2025
bffabc7
chore: Format examples in doc strings - proto, pruning, and session (…
CuteChuanChuan Oct 29, 2025
618e496
chore: Format examples in doc strings - catalog listing (#18335)
CuteChuanChuan Oct 29, 2025
0ca4eaf
feat: support temporary views in DataFrameTableProvider (#18158)
r1b Oct 29, 2025
d21279d
feat: Better parquet row-group/page pruning metrics display (#18321)
2010YOUY01 Oct 30, 2025
a782423
ci: fix temporary file creation in tests and tighten CI check (#18374)
2010YOUY01 Oct 30, 2025
ff670d5
Run extended tests when there are changes to datafusion-testing pin (…
alamb Oct 30, 2025
6c852a4
Introduce `expr_fields` to `AccumulatorArgs` to hold input argument f…
Jefffrey Oct 30, 2025
9fe6138
Add simple unit test for `merge` in case expression (#18369)
pepijnve Oct 30, 2025
c204041
chore(deps): bump taiki-e/install-action from 2.62.40 to 2.62.41 (#18…
dependabot[bot] Oct 30, 2025
11b6b85
feat: Add Hash trait to StatsType enum (#18382)
rluvaton Oct 30, 2025
3b84777
feat: support get_field for map literal (#18371)
chenkovsky Oct 30, 2025
7a00227
fix: correct date_trunc for times before the epoch (#18356)
mhilton Oct 30, 2025
d36f8e7
fix: Preserve percent-encoding in `PartitionedFile` paths during dese…
lonless9 Oct 30, 2025
d0d8c0f
fix: SortPreservingMerge sanity check rejects valid ORDER BY with CAS…
watford-ep Oct 30, 2025
a87235f
Refactor `range`/`gen_series` signature away from user defined (#18317)
Jefffrey Oct 30, 2025
2ee13d6
fix: `DataFrame::select_columns` and `DataFrame::drop_columns` for qu…
dqkqd Oct 30, 2025
3239868
Adds Partitioned CSV test to object store access tests (#18370)
BlakeOrth Oct 30, 2025
a0f1d1d
Add reproducer for consecutive RepartitionExec (#18343)
NGA-TRAN Oct 30, 2025
868078e
feat(docs): enable navbar (#18324)
foskey51 Oct 31, 2025
5fb97ec
chore: bump substrait version to `0.60.0` to use substrait spec v0.75…
benbellick Oct 31, 2025
a380aaf
Use the upstream arrow-rs coalesce kernel (#17193)
zhuqi-lucas Oct 31, 2025
008cd3d
Extract out super slow planning benchmark to it's own benchmark (#18388)
Omega359 Oct 31, 2025
dcb858b
minor: Fix parquet pruning metrics display order (#18379)
2010YOUY01 Oct 31, 2025
2773f40
chore: use enum as `date_trunc` granularity (#18390)
comphead Oct 31, 2025
5389b95
chore(deps): bump taiki-e/install-action from 2.62.41 to 2.62.43 (#18…
dependabot[bot] Oct 31, 2025
3323552
Project record batches to avoid filtering unused columns in `CASE` ev…
pepijnve Oct 31, 2025
a045824
catch errors when simplifying cast(lit(...), ...) and bubble those up…
adriangb Oct 31, 2025
980a173
fix(docs): remove navbar padding breaking ui on mobile (#18402)
foskey51 Nov 1, 2025
3194b0a
Align `NowFunc::new()` with canonical `ConfigOptions` timezone and en…
kosiew Nov 1, 2025
eee230a
Rename `is_ordered_set_aggregate` to `supports_within_group_clause` f…
Jefffrey Nov 1, 2025
9c64893
Bump MSRV to 1.88.0 (#18403)
harshasiddartha Nov 1, 2025
15f653f
chore: Format examples in doc strings - physical expr, optimizer, and…
CuteChuanChuan Nov 1, 2025
0affa6a
feat: Add `selectivity` metrics to `FilterExec` (#18406)
2010YOUY01 Nov 2, 2025
18e5491
Fix: spark bit_count function (#18322)
kazantsev-maksim Nov 2, 2025
323cc45
chore: bump workspace rust version to 1.91.0 (#18422)
randyli Nov 2, 2025
94f5204
Minor: Remove unneccessary vec! in SortMergeJoinStream initialization…
mapleFU Nov 2, 2025
6ca652a
minor: refactor array reverse internals (#18445)
Jefffrey Nov 3, 2025
826da09
chore(deps): bump taiki-e/install-action from 2.62.43 to 2.62.45 (#18…
dependabot[bot] Nov 3, 2025
8e933c7
chore(deps): bump crate-ci/typos from 1.38.1 to 1.39.0 (#18464)
dependabot[bot] Nov 3, 2025
cf9b419
chore(deps): bump rstest from 0.25.0 to 0.26.1 (#18463)
dependabot[bot] Nov 3, 2025
88df088
chore(deps): bump wasm-bindgen-test from 0.3.54 to 0.3.55 (#18462)
dependabot[bot] Nov 3, 2025
2aea43e
chore(deps): bump postgres-types from 0.2.10 to 0.2.11 (#18461)
dependabot[bot] Nov 3, 2025
82340bb
fix: null cast not valid in substrait round trip (#18414)
gene-bordegaray Nov 3, 2025
fde51af
chore(deps): bump ctor from 0.4.3 to 0.6.1 (#18460)
dependabot[bot] Nov 3, 2025
c00f08d
chore(deps): bump libc from 0.2.176 to 0.2.177 (#18459)
dependabot[bot] Nov 3, 2025
ee1353a
chore: Format examples in doc strings - functions (#18353)
CuteChuanChuan Nov 3, 2025
0ed46e4
Change default `time_zone` to `None` (was `"+00:00"`) (#18359)
Omega359 Nov 3, 2025
b344dc2
Feat: Support array flatten() on `List(LargeList(_))` types (#18363)
sdf-jkl Nov 3, 2025
fde2e08
feat: Add `reduction_factor` metric to `AggregateExec` for EXPLAIN AN…
petern48 Nov 3, 2025
f31349c
Allow filter pushdown through AggregateExec (#18404)
LiaCastaneda Nov 3, 2025
e7ef9b8
Reproducer tests for #18380 (resorting sorted inputs) (#18352)
rgehan Nov 3, 2025
b750db8
fix: map benchmark failing (#18469)
randyli Nov 4, 2025
b5b2c3e
Update criterion to 0.7.* (#18472)
Omega359 Nov 4, 2025
91549e9
Fix instances of "the the" to be "the" in comments/docs (#18478)
corasaurus-hex Nov 4, 2025
44aa43a
fix: eliminate warning when building without sql feature (#18480)
corasaurus-hex Nov 4, 2025
8ac38e3
chore(deps): bump taiki-e/install-action from 2.62.45 to 2.62.46 (#18…
dependabot[bot] Nov 4, 2025
67cfb08
Move generate_series projection logic into LazyMemoryStream (#18373)
mkleen Nov 4, 2025
8a85d4a
Consolidate flight examples (#18142) (#18442)
cj-zhukov Nov 4, 2025
0b38255
feat: support named arguments for aggregate and window udfs (#18389)
bubulalabu Nov 5, 2025
3dce86f
Support reverse for ListView (#18424)
vegarsti Nov 5, 2025
0be8af7
fix: spark array return type mismatch when inner data type is LargeLi…
jizezhang Nov 5, 2025
70a21dc
feat: Add selectivity metric to NestedLoopJoinExec for EXPLAIN ANALYZ…
petern48 Nov 5, 2025
40e15f9
Complete migrating `enforce_distrubution` tests to insta (#18185)
blaginin Nov 5, 2025
233eee1
Add benchmark for array_reverse (#18425)
vegarsti Nov 5, 2025
740061d
chore: simplify map const (#18440)
chenkovsky Nov 5, 2025
def3939
Avoid scatter operation in `ExpressionOrExpression` case evaluation m…
pepijnve Nov 5, 2025
6553d2d
Fix an out of date comment for `snapshot_physical_expr` (#18498)
AdamGS Nov 5, 2025
6665dd0
Disable `parquet_encryption` by default in datafusion-sqllogictests (…
zhuqi-lucas Nov 6, 2025
97e52af
Make extended test to use optional parquet_encryption feature (#18507)
zhuqi-lucas Nov 6, 2025
f9cb968
Update roadmap links for DataFusion Q1 2026 (#18495)
alamb Nov 6, 2025
d5f4a0a
Consolidate udf examples (#18142) (#18493)
cj-zhukov Nov 6, 2025
e830e33
test: add prepare alias slt test (#18522)
dqkqd Nov 7, 2025
542c5ca
CI: add `clippy::needless_pass_by_value` rule (#18468)
2010YOUY01 Nov 7, 2025
67d93bd
Refactor create_hashes to accept array references (#18448)
adriangb Nov 7, 2025
8a2360a
Add a SpillingPool to manage collections of spill files (#18207)
adriangb Nov 7, 2025
c807ecf
chore: Format examples in doc strings - spark, sql, sqllogictest, sib…
CuteChuanChuan Nov 7, 2025
a9ef0c6
fix: shuffle seed (#18518)
chenkovsky Nov 8, 2025
9efb6e9
refactor: simplify `calculate_binary_math` in datafusion-functions (#…
Jefffrey Nov 8, 2025
27d3a54
ci: enforce needless_pass_by_value for datafusion-optimzer (#18533)
jizezhang Nov 8, 2025
1674780
Add comments to Cargo.toml about workspace overrides (#18526)
alamb Nov 8, 2025
d89c07e
minor: Remove inconsistent comment (#18539)
2010YOUY01 Nov 8, 2025
5e95e3e
Refactor `log()` signature to use coercion API + fixes (#18519)
Jefffrey Nov 8, 2025
2b1b58b
chore(deps): bump taiki-e/install-action from 2.62.46 to 2.62.47 (#18…
dependabot[bot] Nov 8, 2025
787f5d3
feat: Enhance `array_slice` functionality to support `ListView` and `…
Weijun-H Nov 8, 2025
8429ad2
Consolidate builtin functions examples (#18142) (#18523)
cj-zhukov Nov 8, 2025
7b4a521
refactor: update cmp and nested data in binary operator (#18256)
sunng87 Nov 8, 2025
ddd9fa1
Fix: topk_aggregate benchmark failing (#18502)
randyli Nov 8, 2025
f2d1649
refactor: Add `assert_or_internal_err!` macro for more ergonomic inte…
2010YOUY01 Nov 9, 2025
2d6d265
chore: enforce clippy lint needless_pass_by_value to datafusion-physi…
foskey51 Nov 9, 2025
cd849d7
chore: enforce clippy lint needless_pass_by_value for datafusion-sql …
foskey51 Nov 9, 2025
a854ecc
chore: enforce clippy lint needless_pass_by_value to physical-expr-co…
foskey51 Nov 9, 2025
42896c8
chore: Enforce lint rule `clippy::needless_pass_by_value` to `datafus…
corasaurus-hex Nov 9, 2025
3e883ee
Force `FileSource` to be constructed with a `Schema` (#18386)
adriangb Nov 9, 2025
b447d36
Fix out-of-bounds access in SLT runner (#18562)
theirix Nov 9, 2025
e64e8f1
feat: support complex expr for prepared statement argument (#18383)
chenkovsky Nov 9, 2025
90fd03c
feat: Implement `SessionState::create_logical_expr_from_sql_expr` (#1…
petern48 Nov 9, 2025
3fe2a5c
[main] Update version to 51.0.0, add Changelog (#18551) (#18565)
alamb Nov 9, 2025
732f4e8
Make array_reverse faster for List and FixedSizeList (#18500)
vegarsti Nov 9, 2025
299cbcd
Consolidate custom data source examples (#18142) (#18553)
cj-zhukov Nov 9, 2025
73a5c0c
Normalize partitioned and flat object listing (#18146)
BlakeOrth Nov 9, 2025
2b86ce5
refactor: include metric output_batches into BaselineMetrics (#18491)
nmbr7 Nov 9, 2025
ba5279f
feat: added clippy::needless_pass_by_value lint rule to datafusion/ex…
Gohlub Nov 10, 2025
b4dd97e
feat: support nested key for get_field (#18394)
chenkovsky Nov 10, 2025
af2554e
Support Arrow IPC Stream Files (#18457)
corasaurus-hex Nov 10, 2025
9c1782d
chore(deps): bump taiki-e/install-action from 2.62.47 to 2.62.49 (#18…
dependabot[bot] Nov 10, 2025
f2447db
chore(deps): bump maturin from 1.9.6 to 1.10.0 in /docs (#18590)
dependabot[bot] Nov 10, 2025
a6d9a04
chore: Remove unused `tokio` dependency and clippy (#18598)
comphead Nov 10, 2025
ad48005
minor: enforce `clippy::needless_pass_by_value` for crates that don't…
2010YOUY01 Nov 11, 2025
be684c4
fix: Eliminate consecutive repartitions (#18521)
gene-bordegaray Nov 11, 2025
953cbc4
refactor: merge CoalesceAsyncExecInput into CoalesceBatches (#18540)
Tim-53 Nov 11, 2025
3c7238f
cargo fmt
codetyri0n Nov 1, 2025
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
The table of contents is too big for display.
Diff view
Diff view
  •  
  •  
  •  
2 changes: 1 addition & 1 deletion .github/workflows/audit.yml
Original file line number Diff line number Diff line change
Expand Up @@ -42,7 +42,7 @@ jobs:
steps:
- uses: actions/checkout@08c6903cd8c0fde910a37f88322edcfb5dd907a8 # v5.0.0
- name: Install cargo-audit
uses: taiki-e/install-action@ebb229c6baa68383264f2822689b07b4916d9177 # v2.62.36
uses: taiki-e/install-action@44c6d64aa62cd779e873306675c7a58e86d6d532 # v2.62.49
with:
tool: cargo-audit
- name: Run audit check
Expand Down
5 changes: 3 additions & 2 deletions .github/workflows/extended.yml
Original file line number Diff line number Diff line change
Expand Up @@ -44,6 +44,7 @@ on:
- 'datafusion/physical*/**/*.rs'
- 'datafusion/expr*/**/*.rs'
- 'datafusion/optimizer/**/*.rs'
- 'datafusion-testing'
workflow_dispatch:
inputs:
pr_number:
Expand Down Expand Up @@ -123,7 +124,7 @@ jobs:
--lib \
--tests \
--bins \
--features avro,json,backtrace,extended_tests,recursive_protection
--features avro,json,backtrace,extended_tests,recursive_protection,parquet_encryption
- name: Verify Working Directory Clean
run: git diff --exit-code
- name: Cleanup
Expand Down Expand Up @@ -168,7 +169,7 @@ jobs:
rust-version: stable
- name: Run sqllogictest
run: |
cargo test --features backtrace --profile release-nonlto --test sqllogictests -- --include-sqlite
cargo test --features backtrace,parquet_encryption --profile release-nonlto --test sqllogictests -- --include-sqlite
cargo clean


Expand Down
17 changes: 13 additions & 4 deletions .github/workflows/rust.yml
Original file line number Diff line number Diff line change
Expand Up @@ -302,6 +302,15 @@ jobs:
--features serde,avro,json,backtrace,integration-tests,parquet_encryption
- name: Verify Working Directory Clean
run: git diff --exit-code
# Check no temporary directories created during test.
# `false/` folder is excuded for rust cache.
- name: Verify Working Directory Clean (No Untracked Files)
run: |
STATUS="$(git status --porcelain | sed -e '/^?? false\/$/d' -e '/^?? false$/d')"
if [ -n "$STATUS" ]; then
echo "$STATUS"
exit 1
fi

# datafusion-cli tests
linux-test-datafusion-cli:
Expand Down Expand Up @@ -425,7 +434,7 @@ jobs:
sudo apt-get update -qq
sudo apt-get install -y -qq clang
- name: Setup wasm-pack
uses: taiki-e/install-action@ebb229c6baa68383264f2822689b07b4916d9177 # v2.62.36
uses: taiki-e/install-action@44c6d64aa62cd779e873306675c7a58e86d6d532 # v2.62.49
with:
tool: wasm-pack
- name: Run tests with headless mode
Expand Down Expand Up @@ -466,7 +475,7 @@ jobs:
export RUST_MIN_STACK=20971520
export TPCH_DATA=`realpath datafusion/sqllogictest/test_files/tpch/data`
cargo test plan_q --package datafusion-benchmarks --profile ci --features=ci -- --test-threads=1
INCLUDE_TPCH=true cargo test --features backtrace --profile ci --package datafusion-sqllogictest --test sqllogictests
INCLUDE_TPCH=true cargo test --features backtrace,parquet_encryption --profile ci --package datafusion-sqllogictest --test sqllogictests
- name: Verify Working Directory Clean
run: git diff --exit-code

Expand Down Expand Up @@ -752,7 +761,7 @@ jobs:
- name: Setup Rust toolchain
uses: ./.github/actions/setup-builder
- name: Install cargo-msrv
uses: taiki-e/install-action@ebb229c6baa68383264f2822689b07b4916d9177 # v2.62.36
uses: taiki-e/install-action@44c6d64aa62cd779e873306675c7a58e86d6d532 # v2.62.49
with:
tool: cargo-msrv

Expand Down Expand Up @@ -797,4 +806,4 @@ jobs:
- uses: actions/checkout@08c6903cd8c0fde910a37f88322edcfb5dd907a8 # v5.0.0
with:
persist-credentials: false
- uses: crate-ci/typos@80c8a4945eec0f6d464eaf9e65ed98ef085283d1 # v1.38.1
- uses: crate-ci/typos@07d900b8fa1097806b8adb6391b0d3e0ac2fdea7 # v1.39.0
Loading
Loading