Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[FEAT] [New Query Planner] Support for Ray runner in new query planner. #1265

Merged
merged 3 commits into from
Aug 14, 2023

Conversation

clarkzinzow
Copy link
Contributor

@clarkzinzow clarkzinzow commented Aug 12, 2023

This PR enables the Ray runner in the new query planner. This is accomplished by encapsulating the Rust-side PhysicalPlan in a Python-facing PhysicalPlanScheduler, which we then make pickleable.

TODOs

  • Add support for ResourceRequests

@github-actions github-actions bot added the enhancement New feature or request label Aug 12, 2023
@codecov
Copy link

codecov bot commented Aug 12, 2023

Codecov Report

Merging #1265 (a98ec1a) into main (6a12680) will increase coverage by 0.05%.
Report is 5 commits behind head on main.
The diff coverage is 91.11%.

Additional details and impacted files

Impacted file tree graph

@@            Coverage Diff             @@
##             main    #1265      +/-   ##
==========================================
+ Coverage   87.61%   87.66%   +0.05%     
==========================================
  Files          62       61       -1     
  Lines        6047     6009      -38     
==========================================
- Hits         5298     5268      -30     
+ Misses        749      741       -8     
Files Changed Coverage Δ
daft/__init__.py 26.31% <0.00%> (ø)
daft/execution/execution_step.py 93.67% <ø> (-0.02%) ⬇️
daft/runners/runner.py 81.48% <ø> (+1.48%) ⬆️
daft/logical/builder.py 78.57% <50.00%> (-0.89%) ⬇️
daft/logical/logical_plan.py 79.49% <75.00%> (+0.05%) ⬆️
daft/logical/rust_logical_plan.py 85.31% <83.33%> (+0.85%) ⬆️
daft/dataframe/dataframe.py 88.95% <100.00%> (-0.03%) ⬇️
daft/execution/physical_plan.py 94.56% <100.00%> (-0.02%) ⬇️
daft/execution/rust_physical_plan_shim.py 98.27% <100.00%> (+0.12%) ⬆️
daft/logical/optimizer.py 97.92% <100.00%> (-0.01%) ⬇️
... and 6 more

... and 2 files with indirect coverage changes

@clarkzinzow clarkzinzow force-pushed the clark/ray-runner-new-query-planner branch from 6ffe204 to 472810d Compare August 14, 2023 15:50
@github-actions github-actions bot added the documentation Improvements or additions to documentation label Aug 14, 2023
@clarkzinzow clarkzinzow enabled auto-merge (squash) August 14, 2023 20:33
@clarkzinzow clarkzinzow merged commit c43c76c into main Aug 14, 2023
18 checks passed
@clarkzinzow clarkzinzow deleted the clark/ray-runner-new-query-planner branch August 14, 2023 20:43
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
documentation Improvements or additions to documentation enhancement New feature or request
Projects
None yet
Development

Successfully merging this pull request may close these issues.

1 participant