UndocumentedDetails

Undocumented details in TMVA

Although TMVA manual is nice in general, it does not address some important details, which should therefore be deduced from the source code. In this page I summarise some findings relevant to neural networks.

MLP method

This is the recommended implementation of neural networks in TMVA.

Learning rate in the backpropagation algorithm

The learning rate (its initial value is set by option LearningRate, defaults to 0.02) controls the step size in the gradient descent algorithm. As the training proceeds, it is decreased using the decay rate (set by option DecayRate, defaults to 0.01). The following transformation is applied at every epoch:

learningRate *= (1. - decayRate)

except for the last 5% of epochs, when the decrease in the learning rate is accelerated:

learningRate *= (1. - sqrt(decayRate))

I am not aware of any theoretical motivation for such choice of the learning rate schedule. Sequential learning is a problem of stochastic optimisation and can be addressed by the Robbins-Monro algorithm, whose requirements impose a slower degrease in the learning rate. Instead, a typical choice of the schedule is to use at epoch t a learning rate of

learningRate0 / (1 + t / T)

where learningRate0 and T are parameters of the algorithm. Such schedule matches the requirements of the Robbins-Monro algorithm.

It is worth noting, however, that the choice of learning rate is a difficult problem, and in general case a change of weights should not depend on the value of the gradient of the loss function at all, as discussed here.

The default schedule of the learning rate in TMVA is optimised for sequential learning (assuming all inputs are roughly in the range [-1, 1] and the average event weight is 1) and will lead to a poor performance in the batch mode.

Normalisation of the loss function in batch learning

The loss function is not always normalised by the number of events used for batch training. If it is not done, the learning rate used in the backpropagation algorithm should be rescaled accordingly. However, in the TMVA implementation the loss function is effectively normalised. In batch training weights are updated with the help of methods TNeuron::UpdateSynapsesBatch and TNeuron::AdjustSynapseWeights; the latter method calls TSynapse::AdjustWeight for each weight (“synapse”), which normalises the total shift of the weight by the accumulated number of one-event corrections.

Thanks to the normalisation, the optimal learning rate for batch learning is roughly of the same order as for sequential learning. However, in the latter case the learning rate should decrease more rapidly in order to stabilise the training process around the optimal point.

Training modes with BFGS algorithm

Training with the BFGS algorithm is always performed in the batch mode, as can be seen from this loop in a method that is called (indirectly) from BFGSMinimize. Training mode requested in parameters of the method is ignored, so there is no risk to accidentally run BFGS training in the sequential mode (which is set by default).

Epoch of the backpropagation algorithm

Except for when the importance sampling is enabled, at every epoch all events in the training set are utilised, as can be seen here. Therefore, batch and sequential training with same number of epochs require similar amounts of time.

Weight decay

Weight decay is enabled with the UseRegulator parameter. From the source code it seems that the weight decay method is implemented for BFGS training only.

Datasets

In TMVA several classes are responsible for handling training and testing datasets. They are DataSet, DataSetInfo, DataSetFactory, DataSetManager.

Event weights in the EqualNumEvents mode

When the normalisation mode EqualNumEvents is used, weights of signal events are rescaled such that they equal 1 on average. Weights of background events (or each class of background independently if there are more than one) are rescaled to get the same sum of weights as in signal. If the numbers of (unweighted) events in signal and background classes are not the same, the average background event weight will differ from 1 and hence also the overall average weight. It will then affect the optimal learning rate.

Note that only weights of events in the training set are rescaled, while events in the test set are not modified. As a result, the effective composition (in terms of amount of signal and background) of the two sets can differ, which would affect the value of the loss function. Depending on the balance between original weights of signal and background events, one can obtain any ratio of losses calculated on the two sets. In particular, the test loss might happen to be smaller than the training loss.