Tensor Parameter Server

Overview

Tensor Parameter Server(TPS) is the core component of TensorOnSpark that combines the parameter values asynchronously from the model training executors. It defines diverse built-in parameter-combine policies for different training requirements, as well as allowing user-defined ones. TPS lies in the master of the SparkSession as shown in Fig. 1.

Fig. 1: Architecture of SparkSession

#Weight Combiner The weight combiner is the function to generate a new central parameter value from an old one and the new parameter value from an executor. The parameter value from the executor represents the weight of the training model after a local partition of training data are fed into the model. It needs to be pushed to TPS to generate the central parameter, which stands for the parameter of the SparkSession model and is available for the other executors.

Here, we introduce two of the built-in combiners, Mean Weight Combiner and Delta Weight Combiner, and show their performance in the MNIST example.

###MeanWeightCombiner Mean Weight Combiner