Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Dynamic scheduler mega cleanup: Refactors, renames, docstrings, comments #434

Merged
merged 19 commits into from
Jan 13, 2023
Merged

Dynamic scheduler mega cleanup: Refactors, renames, docstrings, comments #434

merged 19 commits into from
Jan 13, 2023

Conversation

xcharleslin
Copy link
Contributor

@xcharleslin xcharleslin commented Jan 12, 2023 •
edited

  • Added module docstrings or class docstrings for all new modules.
  • Added function docstrings for most new functions.
  • Renamed components to more canonical names.
  • DynamicSchedule is now Physical Plan (the dynamic-ness is just an implementation detail now)
  • Construction is now ExecutionStep.
  • Refactored physical plan to be generator functions, which are noticeably cleaner.

TODO in separate PR: Clean up Runner code.

@xcharleslin
WIP. Construction -> OpenConstruction and ExecuteRequest
ef84816
@xcharleslin
Tweaks to ExecutionResult
bacc61f
@xcharleslin
ExecutionRequest and ExecutionRequestMulti
c7ca62c
@xcharleslin
-WIP dynamic_schedule generator rewrite up to join
d13a5ae
@xcharleslin
Only have IDs for ExecutionRequests
8a9d54d
@xcharleslin
More WIP generator rewrite, now with limit
cbc5b6a
@xcharleslin
Revert naming change, Execute back to Materialize
5249cdd
@xcharleslin
Almost done, but need to update names
f915dfd
@xcharleslin
Remove ray.get in sort instruction
f2a17ba
@xcharleslin
Generatorize sort and materialize. rewrite draft finished.
992d3f3
@xcharleslin
Polish naming and docstrings of Construction (now ExecutionStep).
af8377f
@xcharleslin
Rename dynamic_construction.py -> execution_step.py
0947794
@xcharleslin
DynamicSchedule docstrings and rename to physical plan.
e3a6406
@xcharleslin
mv dynamic_schedule.py -> physical_plan.py
ac3ca43
@xcharleslin
Update DynamicScheduleFactory
1abb54d
@xcharleslin
mv dynamic_schedule_factory.py -> physical_plan_factory.py
368949d
@xcharleslin
Add num_results to MaterializationRequestMulti, needed upfront by ray…
add3762 
… runner
@xcharleslin
Update runners, and linting
2c00095
@xcharleslin
All doesn't do lazy eval; fixed. DynamicRunner passes now
643d24a
@xcharleslin xcharleslin changed the title Charles/cleanupconstructions Mega cleanup for the new dynamic components. Jan 13, 2023
@xcharleslin xcharleslin changed the title Mega cleanup for the new dynamic components. Dynamic scheduler mega cleanup: Refactors, renames, docstrings, comments Jan 13, 2023
@xcharleslin xcharleslin marked this pull request as ready for review January 13, 2023 20:09
@xcharleslin
Copy link
Contributor Author

No performance regression.

2023-01-13 12:12:23.576 | INFO     | __main__:collect_metrics:88 - Running benchmarks for TPC-H q1
2023-01-13 12:12:27.095 | INFO     | __main__:collect_metrics:96 - Finished benchmarks for q1: 3.518938s
2023-01-13 12:12:27.497 | INFO     | __main__:collect_metrics:88 - Running benchmarks for TPC-H q2
2023-01-13 12:12:28.709 | INFO     | __main__:collect_metrics:96 - Finished benchmarks for q2: 1.21173s
2023-01-13 12:12:28.982 | INFO     | __main__:collect_metrics:88 - Running benchmarks for TPC-H q3
2023-01-13 12:12:31.155 | INFO     | __main__:collect_metrics:96 - Finished benchmarks for q3: 2.172531s
2023-01-13 12:12:31.362 | INFO     | __main__:collect_metrics:88 - Running benchmarks for TPC-H q4
2023-01-13 12:12:35.289 | INFO     | __main__:collect_metrics:96 - Finished benchmarks for q4: 3.926406s
2023-01-13 12:12:35.678 | INFO     | __main__:collect_metrics:88 - Running benchmarks for TPC-H q5
2023-01-13 12:12:39.613 | INFO     | __main__:collect_metrics:96 - Finished benchmarks for q5: 3.935323s
2023-01-13 12:12:39.701 | INFO     | __main__:collect_metrics:88 - Running benchmarks for TPC-H q6
2023-01-13 12:12:40.299 | INFO     | __main__:collect_metrics:96 - Finished benchmarks for q6: 0.597944s
2023-01-13 12:12:40.781 | INFO     | __main__:collect_metrics:88 - Running benchmarks for TPC-H q7
2023-01-13 12:12:43.055 | INFO     | __main__:collect_metrics:96 - Finished benchmarks for q7: 2.273594s
2023-01-13 12:12:43.617 | INFO     | __main__:collect_metrics:88 - Running benchmarks for TPC-H q8
2023-01-13 12:12:47.457 | INFO     | __main__:collect_metrics:96 - Finished benchmarks for q8: 3.839576s
2023-01-13 12:12:47.990 | INFO     | __main__:collect_metrics:88 - Running benchmarks for TPC-H q9
2023-01-13 12:12:53.392 | INFO     | __main__:collect_metrics:96 - Finished benchmarks for q9: 5.401325s
2023-01-13 12:12:53.710 | INFO     | __main__:collect_metrics:88 - Running benchmarks for TPC-H q10
2023-01-13 12:12:55.899 | INFO     | __main__:collect_metrics:96 - Finished benchmarks for q10: 2.188393s

@xcharleslin xcharleslin merged commit 461cb71 into main Jan 13, 2023
@xcharleslin xcharleslin deleted the charles/cleanupconstructions branch January 13, 2023 20:21
This was referenced Jan 13, 2023
Closed
Closed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers
No reviews
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.

None yet
1 participant
@xcharleslin