Better control over queue movement out-of-core

[mrzv]

Issue by Dmitriy Morozov
Monday Mar 09, 2015 at 23:34 GMT

---

The current way of saying that block's queues are out of core if the block is out of core is too crude. For example, when sorting, when computing a histogram, it doesn't make sense to move the queues (that carry the histogram) out of core, just because the block is out of core. There are alternative methods one could imagine (e.g., specify maximum queue size limit below which the queue is never moved). It might be nice to think of a way to abstract it away, similarly to how we provide the skip mechanism for blocks right now.

[mrzv]

Comment by Dmitriy Morozov
Friday Mar 20, 2015 at 16:41 GMT

---

While at it, we should also batch outgoing queues together when moving them out-of-core. Incoming queues don't have a choice but to be moved out individually when they arrive (lest we end up storing all of them in memory), but the outgoing queues are finalized after a callback is executed on a block from foreach, and, therefore, they can be handled together as a group.

[mrzv]

Comment by Dmitriy Morozov
Sunday Mar 22, 2015 at 19:08 GMT

---

Changed queue movement mechanism (fixed #34):

• Outgoing queues of a block move in and out of core all together
• Added Master::QueuePolicy to control when queues should be moved out of core

→ f60f98d