Tensor Parameter Server
Tensor Parameter Server(TPS) is the core component of TensorOnSpark that combines the parameter values asynchronously from the model training executors. It defines diverse built-in parameter-combine policies for different training requirements, as well as allowing user-defined ones. TPS lies in the master of the SparkSession as shown in Fig. 1.
Fig. 1: Architecture of SparkSession
#Weight Combiner The weight combiner is the module to generate a new central parameter value from an old one and the new parameter value from an executor. The parameter value from the executor represents the weight of the training model after a local partition of training data are fed into the model. It needs to be pushed to TPS to generate the central parameter, which stands for the parameter of the SparkSession model and is available for the other executors.
The weight combiner needs to implement the compute method in the following format (python):
compute(self, origin_weight, new_weight, worker_id, name)
where "origin_weight" is the original central weight, "new_weight" is the weight pushed from the executor, "worker_id" is the ID of the executor, and "name" is the name of the parameter to be combined.
Here, we introduce two of the built-in combiners, Mean Weight Combiner (MWC) and Delta Weight Combiner (DWC), and show their performance in the MNIST example. Generally, they both treat the new central weight as the increment (decrement) from the original central weight. The difference is how much the central should increase/decrease, that is, the delta weight.
In MWC, the delta weight is the difference of the new executor weight (W_e) and the central weight (W_c) over the number of executors (n). That is,
W_c = W_c + (W_e - W_c) / n
In DWC, the delta weight is the difference of the new executor weight (W_e) and the last weight (W'_e) that the executor combined with the server. That is,
W_c = W_c + (W_e - W'e)