-
Notifications
You must be signed in to change notification settings - Fork 1
UndocumentedDetails
Although TMVA manual is nice in general, it does not address some important details, which should therefore be deduced from the source code. In this page I summarise some findings relevant to neural networks.
This is the recommended implementation of neural networks in TMVA.
The learning rate (its initial value is set by option LearningRate, defaults to 0.02) controls the step size in the gradient descent algorithm. As the training proceeds, it is decreased using the decay rate (set by option DecayRate, defaults to 0.01). The following transformation is applied at every epoch:
learningRate *= (1. - decayRate)
except for the last 5% of epochs, when the decrease in the learning rate is accelerated:
learningRate *= (1. - sqrt(decayRate))
I am not aware of any theoretical motivation for such choice of the learning rate schedule. Sequential learning is a problem of stochastic optimisation and can be addressed by the Robbins-Monro algorithm, whose requirements impose a slower degrease in the learning rate. Instead, a typical choice of the schedule is to use at epoch t a learning rate of
learningRate0 / (1 + t / T)
where learningRate0 and T are parameters of the algorithm. Such schedule matches the requirements of the Robbins-Monro algorithm.
It is worth noting, however, that the choice of learning rate is a difficult problem, and in general case a change of weights should not depend on the value of the gradient of the loss function at all, as discussed here.
The default schedule of the learning rate in TMVA is optimised for sequential learning (assuming all inputs are roughly in the range [-1, 1]) and will lead to a poor performance in the batch mode.
The loss function is not always normalised by the number of events used for batch training. If it is not done, the learning rate used in the backpropagation algorithm should be rescaled accordingly. However, in the TMVA implementation the loss function is effectively normalised. In batch training weights are updated with the help of methods TNeuron::UpdateSynapsesBatch and TNeuron::AdjustSynapseWeights; the latter method calls TSynapse::AdjustWeight for each weight (“synapse”), which normalises the total shift of the weight by the accumulated number of one-event corrections.
Thanks to the normalisation, the optimal learning rate for batch learning is roughly of the same order as for sequential learning. However, in the latter case the learning rate should decrease more rapidly in order to stabilise the training process around the optimal point.
Training with the BFGS algorithm is always performed in the batch mode, as can be seen from this loop in a method that is called (indirectly) from BFGSMinimize. Training mode requested in parameters of the method is ignored, so there is no risk to accidentally run BFGS training in the sequential mode (which is set by default).
Except for when the importance sampling is enabled, at every epoch all events in the training set are utilised, as can be seen here. Therefore, batch and sequential training with same number of epochs require similar amounts of time.
Weight decay is enabled with the UseRegulator parameter. From the source code it seems that the weight decay method is implemented for BFGS training only.