UndocumentedDetails

Undocumented details in TMVA

Although TMVA manual is nice in general, it does not address some important details, which should therefore be deduced from the source code. In this page I summarise some findings relevant to neural networks.

MLP method

This is the recommended implementation of neural networks in TMVA.

Learning rate in the backpropagation algorithm

The learning rate (set by option LearningRate, defaults to 0.02) controls the step size in the gradient descent algorithm. As the training proceeds, it is decreased using the decay rate (set by option DecayRate, defaults to 0.01). The following transformation is applied at every epoch:

learningRate *= (1. - decayRate)

except for the last 5% of epochs, when the decrease in the learning rate is accelerated:

learningRate *= (1. - sqrt(decayRate))

I am not aware of any motivation for such choice of the learning rate schedule. Sequential learning is a problem of stochastic optimisation and can be addressed by the Robbins-Monro algorithm, whose requirements impose a slower degrease in the learning rate. Instead, a typical choice of the schedule is to use at epoch t a learning rate of

learningRate0 / (1 + t / T)

where learningRate0 and T are parameters of the algorithm. Such schedule matches the requirements of the Robbins-Monro algorithm.

The default learning and decay rates are optimised for sequential learning and will lead to a poor performance in the batch mode.

Normalisation of the loss function in batch learning

The loss function is not always normalised by the number of events used for batch training. If it is not done, the learning rate used in the backpropagation algorithm should be rescaled accordingly. However, in the TMVA implementation the loss function is effectively normalised. In batch training weights are updated with the help of methods TNeuron::UpdateSynapsesBatch and TNeuron::AdjustSynapseWeights; the latter method calls TSynapse::AdjustWeight for each weight (“synapse”), which normalises the total shift of the weight by the accumulated number of one-event corrections.

Training modes with BFGS algorithm

Training with the BFGS algorithm is always performed in the batch mode, as can be seen from this loop in a method that is called (indirectly) from BFGSMinimize. Training mode requested in parameters of the method is ignored, so there is no risk to accidentally run BFGS training in the sequential mode (which is set by default).