Some common distributed learning algorithms built in Torch with the help of the the ipc library.
Spreads the computation of gradients for mini-batch of items across N processes. Uses AllReduce to quickly sum the gradients and distribute the total back out to every process.
local allReduceSGD = require 'distlearn.AllReduceSGD'(tree) -- Make sure all the nodes start with the same parameter values allReduceSGD.synchronizeParameters(params) for _ = 1,epochs do for _ = 1,steps -- Compute your gradients as normal local grads = computeYourGrads(...) -- Sum and normalize them allReduceSGD.sumAndNormalizeGradients(grads) -- Do your SGD as normal SGD(params, grads) end -- Before validating we should make sure all nodes have -- the exact same parameter values allReduceSGD.synchronizeParameters(params) -- Validate... end
When used in combination with Dataset you can quickly parallelize the processing of large datasets without a ton of effort. See the MNIST example for a complete working setup.
We also have a AllReduce based implementation of the Elastic Averaging algorithm as described in Deep learning with Elastic Averaging SGD. Its just as easy to add this to your training script, there are only two parameters required tau and alpha. Tau is how many steps to run before averaging the nodes and alpha is the weight used during the averaging step. You can read more about our implementation of AllReduceEA.
-- Use a tau of 10 and an alpha of 0.2 local allReduceEA = require 'distlearn.AllReduceEA'(tree, 10, 0.2) -- Make sure all the nodes start with the same parameter values allReduceEA.synchronizeParameters(params) for _ = 1,epochs do for _ = 1,steps -- Compute your gradients as normal local grads = computeYourGrads(...) -- Do your SGD as normal SGD(params, grads) -- Average the params allReduceEA.averageParameters(params) end -- Make sure the center's haven't drifted too far due to -- floating point precision error build up allReduceEA.synchronizeCenter(params) -- Validate... end
See a complete working example of EA and MNIST
Licensed under the Apache License, Version 2.0. See LICENSE file.