Skip to content

Update DataFusion and change order of optimization rules #825

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 26 commits into from
Oct 10, 2022
Merged

Update DataFusion and change order of optimization rules #825

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
26 commits
Select commit Hold shift + click to select a range
906e70d
Update DataFusion and change order of optimization rules
andygrove Sep 30, 2022
c9211aa
partial fix to Python code
andygrove Sep 30, 2022
196479d
add comment
andygrove Sep 30, 2022
44a62de
save progress
andygrove Sep 30, 2022
59d6916
revert change
andygrove Sep 30, 2022
e5d52ee
Revert chrono
andygrove Sep 30, 2022
b51c75d
add suggestion
andygrove Sep 30, 2022
7f5d7e6
python lint
andygrove Sep 30, 2022
5b21585
bump datafusion version again
andygrove Oct 5, 2022
48b962b
fmt
andygrove Oct 5, 2022
8fe5a34
upmerge
andygrove Oct 5, 2022
3d000c6
add additional test
andygrove Oct 5, 2022
aec3cb7
lint
andygrove Oct 5, 2022
4926796
fix GPU CI?
andygrove Oct 5, 2022
02ed556
bump to 13.0.0-rc1
andygrove Oct 7, 2022
1d04626
stop building aggregate schema and let DataFusion do that
andygrove Oct 7, 2022
62f20f8
Merge remote-tracking branch 'origin/main' into bump-df-0930
charlesbluca Oct 7, 2022
b2cf5ce
Add handling for string to datetime casting, switch to np.timedelta64
charlesbluca Oct 7, 2022
8639e5d
Un-xfail passing tests
charlesbluca Oct 7, 2022
7328664
Resolve style failures
charlesbluca Oct 7, 2022
9f078c7
timedelta64 doesn't accept floats
charlesbluca Oct 7, 2022
7735f67
Merge branch 'main' into bump-df-0930
charlesbluca Oct 7, 2022
ee72003
upmerge:
andygrove Oct 7, 2022
82264fa
fix regression by running SimplifyExpressions again after converting …
andygrove Oct 7, 2022
c2237ac
merge
andygrove Oct 7, 2022
48fdec8
use official DataFusion 13.0.0 release
andygrove Oct 10, 2022
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
114 changes: 69 additions & 45 deletions dask_planner/Cargo.lock
View file
Open in desktop

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

10 changes: 5 additions & 5 deletions dask_planner/Cargo.toml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -9,12 +9,12 @@ edition = "2021"
rust-version = "1.62"

[dependencies]
arrow = { version = "23.0.0", features = ["prettyprint"] }
arrow = { version = "24.0.0", features = ["prettyprint"] }
async-trait = "0.1.41"
datafusion-common = { git = "https://github.com/apache/arrow-datafusion/", rev = "1261741af2a5e142fa0c7916e759859cc18ea59a" }
datafusion-expr = { git = "https://github.com/apache/arrow-datafusion/", rev = "1261741af2a5e142fa0c7916e759859cc18ea59a" }
datafusion-optimizer = { git = "https://github.com/apache/arrow-datafusion/", rev = "1261741af2a5e142fa0c7916e759859cc18ea59a" }
datafusion-sql = { git = "https://github.com/apache/arrow-datafusion/", rev = "1261741af2a5e142fa0c7916e759859cc18ea59a" }
datafusion-common = "13.0.0"
datafusion-expr = "13.0.0"
datafusion-optimizer = "13.0.0"
datafusion-sql = "13.0.0"
env_logger = "0.9"
log = "^0.4"
mimalloc = { version = "*", default-features = false }
Expand Down
2 changes: 1 addition & 1 deletion dask_planner/src/sql.rs
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -390,7 +390,7 @@ impl DaskSQLContext {
match existing_plan.original_plan.accept(&mut visitor) {
Ok(valid) => {
if valid {
optimizer::DaskSqlOptimizer::new()
optimizer::DaskSqlOptimizer::new(true)
.run_optimizations(existing_plan.original_plan)
.map(|k| PyLogicalPlan {
original_plan: k,
Expand Down
Loading