You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
Currently, the partition dependencies of a DynamicSchedule reside within the DynamicSchedule itself. This makes sense for ownership, but is a bit smelly of an interface as the partition constructor must reach into the DynamicSchedule to populate its dependencies.
A better approach would be to extend the existing PartitionSetCache concept to handle individual partitions and use this for DynamicSchedule. Then, partitions can be constructed agnostic of their eventual owner(s).
The text was updated successfully, but these errors were encountered:
Resolved the ownership issue a different way in #434 (ExecutionSteps receive a result accessor; runners can implement their own result accessor) and avoided having to deal with any explicit partition management for now.
Hopefully this will suffice. I think it would be ideal to rely on reference counting and avoid explicit global partition management. In the future, individual runners might have to implement some partition management; ideally this would remain properly abstracted away from the rest of Daft.
Currently, the partition dependencies of a DynamicSchedule reside within the DynamicSchedule itself. This makes sense for ownership, but is a bit smelly of an interface as the partition constructor must reach into the DynamicSchedule to populate its dependencies.
A better approach would be to extend the existing PartitionSetCache concept to handle individual partitions and use this for DynamicSchedule. Then, partitions can be constructed agnostic of their eventual owner(s).
The text was updated successfully, but these errors were encountered: