Wait on activity from the mirai task dispatcher?

[wlandau]

targets is still polling-based because of how crew works, but it would be more efficient to wait on a condition variable while unresolved tasks are running. @shikokuchuo, is it currently possible for me to set up such a condition variable which watches for any dispatcher activity?

[wlandau]

From #107 (comment):

it will be much easier if it works to run your functions on dispatcher itself, to be evaluated directly in that context. If you need another socket to send information back to host then you can still prototype that using the current limited interface, and we can add that as a setup step for efficiency later if needed.

Shouldn't be too hard to set up a socket and create a condition variable on the host which the targets event loop can wait on.

[wlandau]

Actually, I think shikokuchuo/nanonext#17 could get us there. targets wouldn't wait indefinitely on the CV, it would just wait every 0.25 seconds or so (controlled by seconds_interval). But this alone would get us away from the most aggressive forms of polling that burden the CPU.

[wlandau]

A first step is to implement a wait_cv() method for single controllers which just calls nanonext::until(cv = mirai::nextget("cv", .compute), msec = 250).

[wlandau]

Depends on both shikokuchuo/nanonext#20 and shikokuchuo/nanonext#21.

[wlandau]

I added a new class in R/crew_relay.R for event-driven programming on a downstream condition variable which inherits from an upstream CV. It will be straightforward to use it in the wait() method of a single controller. Controller groups are the hard part. Current plan:

1. Give each client object a list of condition variables it should forward to.
2. When a client object starts, forward mirai::nextget("cv") to each of the condition variables from (1).
3. Give each controller a relay object.
4. Give each controller group a relay object of its own.
5. When controllers are supplied to controller groups:
   A. Throw an informative error if any controller is started.
   B. Add the controller group stage object's condition variable to the forwarding list from (1) for each controller object's client.

[wlandau]

In branch 108 I got pop() and wait() working with condition variables for simple controllers. Now all that's left is wait() in controller groups.

[wlandau]

There's a lot of fiddling in #108 (comment), and I think it can be simpler. For controller groups, wait(mode = "all") is actually trivially easy: just call wait(mode = "all") on each controller in sequence. For wait(mode = "one"), things become really simple if the condition variable can be ephemeral: created inside the call to wait() and then automatically destroyed and cleaned up after the call ends.

[wlandau]

My workaround for shikokuchuo/nanonext#24 is to just loop over the controllers in the group and wait on their respective condition variables for seconds_interval / length(controllers) seconds. It could be burdensome on the CPU if there are a lot of controllers in the group, but in practice there should be at most around 2 or 3, so that seems like an okay temporary workaround. Later if it becomes possible to forward to 2 CVs and disable a forwarding connection without gc(), it will be easy for me to update the private wait_one() method of the controller group class.

[wlandau]

Tomorrow I will work on a robust test suite for the crew wait() method. I need to test all combinations of:

• Object type
  1. single controller
  2. controller group
• Mode
  1. Wait for a single task to be available
  2. Wait for all tasks to complete
• Task status
  1. no tasks
  2. just a completed task
  3. just a running task
  4. one completed task and one running task

Expectations to check:

1. Return value of wait()
2. Availability of results from pop()