Move queues out of core

[mrzv]

Issue by Dmitriy Morozov
Friday Nov 14, 2014 at 17:51 GMT

---

Add the ability to move queues out of core (when their blocks are also out of core). For example, this is essential to support IO-efficient distribution of points (through a swap-reduce). Roughly, the protocol should be:

foreach block:
  process(block)
  exchange-light   # optional

exchange-heavy

exchange-light:
  post queues
  kick outstanding messages
  if #queues too large:
    swap some queues out

exchange-heavy:
  post any unposted queues   # happens if exchange-light wasn't invoked
  kick outstanding messages
  while there are external outgoing queues:
    if #posted queues < #allowed queues:
      load and post more queues
    kick outstanding messages

[mrzv]

Comment by Dmitriy Morozov
Wednesday Nov 26, 2014 at 21:35 GMT

---

This does not address what to do with incoming queues. These need more thought. Most likely, we have to re-engineer Communicator and Master. Specifically, Master might have to own incoming and outgoing queues and transparently to Communicator decide when they need to be moved in or out of core.

[mrzv]

Comment by Dmitriy Morozov
Sunday Feb 08, 2015 at 21:30 GMT

---

Added loading/unloading of outgoing queues (fixes #1)

→ 38f2975