Multiple client sockets

[wlandau]

I am developing a branch of crew based on https://github.com/shikokuchuo/mirai#example-1-connecting-to-remote-servers--remote-server-queues, and I am really optimistic about how it will turn out. Currently, however, building on mirai instead of rrq makes it difficult to implement heterogeneous workers to accommodate tasks with varying resource requirements (wlandau/crew#23).

Would you be willing to consider support for multiple concurrent client sockets instead of just the current one stored in environment(daemons)$sock? I would be fine with managing multiple socket objects and supplying the right ones to mirai() and daemons() as needed.

[wlandau]

The way I imagine it, I would have multiple sockets on the client, each with the same IP address but different ephemeral ports (which I would get from parallelly::freePort()). Each socket would correspond to a different worker type: e.g. one socket for GPU-capable servers and another for cheaper servers without GPUs.

[shikokuchuo]

Conceptually it would be as easy as making sock a list.

However, please tell me exactly what you envisage - just "cpu" and "gpu", or fully flexible number of sockets? As this change is now on the critical path, I would want to think about the most efficient implementation.

[wlandau]

I am thinking fully flexible sockets because there is an enormous variety of situations where different tasks require different server types: not only GPUs, but also 1-core vs 4-core machines, different amounts of memory, etc. I would like to take responsibility to handle most of this complexity with crew. If mirai could support more than one client socket open at a time, that would help tremendously.

As an attempted workaround, I tried creating and switching out multiple sockets myself, but I found that daemons(0) was not properly closing servers if I mess with environment(daemons)$sock:

library(mirai)
daemons("tcp://xx.xx.xx.138:5000") # let's say this socket connects to expensive high-memory 4-core servers
socket1 <- environment(daemons)$sock
daemons("tcp://xx.xx.xx.94:8888")  # let's say this socket connects to cheaper low-memory 1-core servers
socket2 <- environment(daemons)$sock
# ... submit some tasks ...
environment(daemons)$sock <- socket1 # switch back to socket1
daemons(0) # does not reliably close servers listening to tcp://xx.xx.xx.138:5000
environment(daemons)$sock <- socket2 # switch back to socket2
daemons(0) # does not reliably close servers listening to tcp://xx.xx.xx.94:8888

If there could be a convenient way to get the socket created by daemons() (maybe as the return value when a TCP socket is supplied as value?), and if daemons() and mirai() could accept a socket as an optional argument (both for submitting tasks and closing servers), that would be amazing. I would be happy to maintain the list of sockets on my end. Hopefully then mirai would not have to worry about managing an entire list of sockets.

[shikokuchuo]

Interesting hackish way of getting to the sockets. You have the right idea. Just when you call daemons() with the second URL it closes the first socket. Leave this with me - I will figure out the best way to implement this.

[shikokuchuo]

It's done in 0.7.2.9009 (commit 3828506) - simply specify the .compute argument for both daemons() and mirai().

Btw. thanks for the suggestions and testing out the new features - I have added a special mention in the changelog!

[wlandau]

Fantastic! I will try it out once the r-universe builds complete.

[wlandau]

Tried it out (installing from the source on my company's SGE cluster). Works beautifully!

[wlandau]

Btw. thanks for the suggestions and testing out the new features - I have added a special mention in the changelog!

Absolutely, and thanks for the mention! I hope it is okay that I ask so many questions.