Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

PartitionCache for DynamicSchedule #404

Closed
xcharleslin opened this issue Dec 27, 2022 · 1 comment
Closed

PartitionCache for DynamicSchedule #404

xcharleslin opened this issue Dec 27, 2022 · 1 comment

Comments

@xcharleslin
Copy link
Contributor

Currently, the partition dependencies of a DynamicSchedule reside within the DynamicSchedule itself. This makes sense for ownership, but is a bit smelly of an interface as the partition constructor must reach into the DynamicSchedule to populate its dependencies.

A better approach would be to extend the existing PartitionSetCache concept to handle individual partitions and use this for DynamicSchedule. Then, partitions can be constructed agnostic of their eventual owner(s).

@xcharleslin
Copy link
Contributor Author

Resolved the ownership issue a different way in #434 (ExecutionSteps receive a result accessor; runners can implement their own result accessor) and avoided having to deal with any explicit partition management for now.

Hopefully this will suffice. I think it would be ideal to rely on reference counting and avoid explicit global partition management. In the future, individual runners might have to implement some partition management; ideally this would remain properly abstracted away from the rest of Daft.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

1 participant