A for-dummies crew for trusted local (or remote) machines

[kendonB]

SearchGPT estimates that there are about 30x more CPUs in the USA sitting inside personal machines than in cloud or HPC compute systems. At any given time, a small fraction of those personal CPUs are in use.

Could there be a crew.local package (similar to crew.cluster) which makes it easy to spin up compute on other trusted machines? I think a lot of the infrastructure is already there (see https://wlandau.github.io/crew/articles/plugins.html).

[wlandau]

Do you mean if you want workers to be on a different personal computer than the one you are running? As long as the networking allows it, a plugin could be developed. There is of course an existing crew_controller_local(), but all the workers are on the same machine as the local process.

[wlandau]

Actually SSH logs into the remote machine (normally using pre-set credentials) by default on port 22 but could be any other, and from there it spawns a mirai daemon R session, which can either dial back directly (if the local firewall is so configured), or more likely dials into a local TCP port which is tunnelled back over the SSH connection to the same port on the local machine, where dispatcher is listening.

How asynchronous is this process?

[shikokuchuo]

It'd be up to the launcher I guess? In mirai I'm just using system2(wait = FALSE) to make the ssh calls async.

[shikokuchuo]

FYI SSH tunnelling will be even easier at mirai once this is merged: r-lib/mirai#209
You can see from the example syntax, that you only need to supply the IP addresses of the remote machines to SSH into.

[wlandau]

r-lib/mirai#209 might fit well with crew. Because it's configured with daemons() (plural) I would implement this at the level of crew_client() and then integrate it with all plugins. Would be really cool if all nontrivial plugins could just use SSH!

[wlandau]

On reflection, I'm not sure r-lib/mirai#209 fits because of the modifications it would require to crew_client(). Also, it seems like mirai::daemons(remote = ssh_config()) takes responsibility for worker launches, which is not really consistent with crew's approach of establishing a listening socket with mirai::daemons() and then allowing workers to dial in at a later point as needed.

Ideally, mirai::daemons(url = "tcp://IP:PORT") could be passed and IP and PORT that would work with the tunnel, then the launcher could create the SSH tunnel if it does not already exist. If this approach is feasible, it could be its own plugin developed outside of crew.

Unfortunately, I don't have the means to develop it myself. My workplace prohibits ssh tunneling into our existing resources, and I can't seem to be able to create a Docker container locally that I could SSH into. https://wlandau.github.io/crew/articles/plugins.html has instructions for anyone with a need to create their own launcher plugin.