Distributed PyTorch on Grid (via IPFS/PubSub)

[jvmncs]

This PR establishes a general framework for distributed tensor computation on Grid with PyTorch. This is accomplished by overloading all relevant functions and methods in the torch module, so that when a user calls torch commands, they distribute the commands across the Grid on remote nodes that contain the actual tensors involved. When we send a tensor over the network, a 0-d tensor pointer remains for future computation. When we execute a command on remote tensors that return a new tensor, the worker node returns registration attributes that are used to create a new 0-d pointer for the result. This means that the original composability of the torch library is maintained, allowing for lower level operations to make their way into higher level abstractions like Modules and Containers with minimal future effort.

There are a few other major benefits to this approach -- mainly, it's the lowest level of abstraction we've been able to successfully distribute over IPFS. This has a host of benefits other than its potential to scale up to higher level abstractions, including more fine-grained control of computation, tighter security guarantees, and allowing for larger models by keeping more IPFS blocks under the 1 MiB limit.

Currently, this work is complete for all Tensor types, although it could use a good stress test or two. The implementation for Variable is incomplete -- the only remaining bits are the special methods send_, get_, and ser, and to ensure that Variables are handled properly throughout transmission and remote computation. I'll be following up with another PR in the coming days to get autograd working.

Due to the nature of Grid at the moment, it's not entirely resilient. Future work should improve error reporting on the client side (#151), induce garbage collection on the worker nodes when a client signals they're done (or after a timeout), and figure out how to handle workers that drop out mid-computation (likely by notifying the client that a worker with one of their tensors has disconnected from IPFS/stopped listening to openmined channels). There's a whole range of other things we need to do as well (e.g. #134), but let's get this merged first. 🙂

Critical files being created and modified mostly include files in grid/services, and in particular the torch subdirectory there, although changes have been made across all files in the repository relating to compute mode. A brief demo of computation on multiple nodes over IPFS can be found at notebooks/experimental/torch_integration/Grid_MultiNode_Demo.ipynb.

Happy Torching!

🎉

[iamtrask]

This marks a new chapter in the OpenMined project - very, very excited to merge this!