Home

Tutorial

This page contains step-by-step instructions for the tutorial on artificial neural networks in TMVA. In order to follow it, you need an access to a computer with a recent version of ROOT installed (and the TMVA package as a part of it). Start by checking out the tutorial's repository as explained in the README.

Introduction

The TMVA package includes a number of MVA methods, with neural networks among others, incorporated into a common framework. The manual can be found here; keep it opened during the session.

The tutorial addresses the task of binary classification, by an example of discrimination between tH, H->bb and semileptonic ttbar events. A number of observables have been chosen to distinguish between the two. ROOT tuples with the observables are available in the data/ directory of the repository. For each process several tuples are provided, varying in the number of events.

The tutorial is focused on the usage of TMVA rather than illustrating various aspects of neural networks as a machine learning technique.

Training and validation

In this part of the tutorial you will learn how to train neural networks, check for overfitting, and evaluate their performance. Navigate into the directory tutorial/ and open script train.C. It is mostly empty, and you will fill it step-by-step. If needed, you can consult an exemplary completed version in train_Example.C. The script can be executed with the command

root -q -b train.C

Step 1. Create a factory

A central class in TMVA is Factory, and most of interactions with the package is done via an instance of this class. It manages trees with input variables, keeps track of “booked” MVA methods, and trains them. Thus, creating a factory is the first thing you do. Copy the following code into your train.C file:

TString jobName("TutorialTrain");
TFile infoFile(TString("info") + jobName + ".root", "recreate");
TMVA::Factory factory(jobName, &infoFile, "Color=True");

The code fragment also opens a ROOT file for writing and gives it to the factory. The file will contain various control and performance histograms. The last argument is a set of parameters for the factory. The parameters control verbosity and visual appearance, a complete list can be found on p. 15 of the manual.

Step 2. Provide trees with training and testing sets

There are many ways to provide training and testing sets to a factory, ranging from a single ROOT tree that contains everything (signal and background, training and testing sets) to simple ASCII files. Details are described in section 3.1.1 of the manual. Here we stick to the more robust option, providing four independent trees:

factory.AddSignalTree(sgnTrainTree, 1., TMVA::Types::kTraining);
factory.AddBackgroundTree(bkgTrainTree, 1., TMVA::Types::kTraining);
factory.AddSignalTree(sgnTestTree, 1., TMVA::Types::kTesting);
factory.AddBackgroundTree(bkgTestTree, 1., TMVA::Types::kTesting);

The second argument is an event weight, which is applied to all events in the tree. We do not use it and specify per-event weight instead:

factory.SetWeightExpression("Weight");

“Weight” is the name of branches in the source trees. The expression provided here is parsed with TTreeFormula and thus can include mathematical operations.

Step 3. Specify input variables

Input variables are specified with the following syntax:

factory.AddVariable("abs(tHq_Eta_Recoil)");

The expression given is parsed with TTreeFormula and can include mathematical operations as in the example above. One can define short labels and human readable titles for variables to obtain a better decoration in control plots; details are in section 3.1.3.

Copy the line above to your train.C and add all the remaining observables stored in the trees. Here is the complete list: glb_Charge_Lep, glb_Sphericity, tHq_Eta_Recoil, tHq_NumBTag_Higgs, tHq_Pt_Higgs, tHq_Pt_Recoil, tt_DeltaR_Light, tt_Mass_TopHad, tt_NumPassBTag_Light. Note that the two transverse momenta and the mass have long tails, and because of this it is better to use their logarithms instead.

Step 4. Make the factory process the source trees

Copy the following line into your train.C:

factory.PrepareTrainingAndTestTree("", "NormMode=EqualNumEvents");

It makes the factory read the training and testing sets (or split the only given set in two), evaluate expressions in input variables, and store results internally. The first argument defines a preselection cut (we do not apply any). The second argument is the list of parameters. If the data were not split into training and testing sets, the splitting is performed now, and the parameters specify how it is done; details are in section 3.1.4. Since we provided separate trees for the two sets, there is no need for the splitting.

The only parameter set in the example demands that signal and background events in the training set are independently reweighted such that the total signal normalisation equals the one of the background. Such symmetrisation usually improves performance of the classification.

Step 5. Book a backpropagation neural network

Copy the following code to “book” a neural network with backpropagation:

factory.BookMethod(TMVA::Types::kMLP, jobName + "_BP",
 "VarTransform=Norm:TrainingMethod=BP:NeuronType=tanh:HiddenLayers=20:EstimatorType=CE:"
 "BPMode=sequential:NCycles=500:TestRate=5");

Input variables and training set were defined before. This method specifies architecture of the neural network and its training parameters. A complete list of supported options can be found on p. 97, 98 of the manual. Here is a description of the options used in the example:

• VarTransform=Norm All input variables are normalised, i.e. linearly transformed to occupy the range [-1, 1]. Neural networks do not like when an input spans over a large range. TMVA can also perform more advanced preprocessing of input variables, for instance, decorrelation or principal component analysis.
• HiddenLayers=20 The network contains a single hidden layer with 20 neurons in it. One can have more than one hidden layer, separating numbers of neurons in each by commas (e.g. HiddenLayers=10,5).
• NeuronType=tanh The activation function in neurons in the hidden layer is hyperbolic tangent.
• TrainingMethod=BP The training will be done with the backpropagation algorithm, using the gradient descent method.
• EstimatorType=CE Cross-entropy will be used as the loss function. For classification problems it is usually considered a better option than the default mean squared error.
• BPMode=sequential Sequential training will be used (instead of batch one).
• NCycles=500 The training will last for 500 epochs.
• TestRate=5 Every 5th epoch the loss function will be evaluated on the testing set, and the value will be stored for future examination.

As training proceeds, the learning rate is decreased by scaling it by the decay rate: learningRate *= (1 - decayRate). The two rates are crucial parameters of the algorithm and can impact quality of training drastically. The default values (0.02 and 0.01 respectively) used by TMVA are reasonable for sequential training and thus are left unchanged.

Step 6. Train the neural network and test its performance

Add the following code:

factory.TrainAllMethods();
factory.TestAllMethods();
factory.EvaluateAllMethods();

It instructs the factory to train all methods booked so far and evaluate their performance. TestAllMethods performs basic evaluation only, while EvaluateAllMethods produces a number of additional control histrograms (e.g. the ROC curve).

Run the program

At this point the program is ready to be run for the first time. Start it. The execution will take about a minute, with most of the time spent in training.

TMVA prints some useful information in the standard output. It includes comments about transformations of input variables, linear correlations between them, ranking of inputs (which is very naive and usually should not be trusted, though), linear correlations between inputs and MVA response, signal efficiencies at few working points corresponding to predefined background efficiencies, area under the ROC curve, and other details.

Check the content of the current directory. The program created a subdirectory weights/. The subdirectory contains an XML file with a complete definition of the constructed neural network, which can be loaded in TMVA and used to apply the neural network to new events (and this will be covered in the second part of the tutorial). It is also saved in a *.C file to be used without TMVA.

The program also created file infoTutorialTrain.root in the current directory. Open it in a TBrowser and inspect the content. You can find correlation matrices for inputs and 1D and 2D distributions of signal and background over inputs (in the subdirectory InputVariables_Id/). Control plots specific to the neural network are placed in the subdirectory Method_MLP/TutorialTrain_BP/. Two groups of them are especially interesting:

• estimatorHistTrain and estimatorHistTest show the evolution of the loss calculated on the training and testing sets respectively. These plots should always be checked to verify that the neural network has not overfit (in case of which the loss calculated on the testing set starts to grow at some point).
• MVA_TutorialTrain_BP_rejBvsS is the ROC curve constructed on the testing set (the one from the testing set is called like *_trainingRejBvsS, but it is of less interest). The ROC curve completely describes performance of a binary discriminator and should usually be reported.

TMVA comes with several scripts that produce (arguably) nicely decorated plots of various kinds from the ROOT file. The complete list of scripts is provided at p. 32, 34. You can try running the following:

root -q -b "$ROOTSYS/tmva/test/variables.C(\"infoTutorialTrain.root\")"
root -q -b "$ROOTSYS/tmva/test/mvas.C(\"infoTutorialTrain.root\", 3)"
root -q -b "$ROOTSYS/tmva/test/efficiencies.C(\"infoTutorialTrain.root\")"
root -q -b "$ROOTSYS/tmva/test/annconvergencetest.C(\"infoTutorialTrain.root\")"

The produced figures are saved in the plots/ subdirectory.

Step 7. Add a BFGS neural network and a projective likelihood discriminator

Same factory can include several MVA methods. Add a neural network trained with the BFGS algorithm with weight decay regularisation:

factory.BookMethod(TMVA::Types::kMLP, jobName + "_BFGS_WeightDecay",
 "VarTransform=Norm:TrainingMethod=BFGS:NeuronType=tanh:HiddenLayers=20:EstimatorType=CE:"
 "NCycles=100:TestRate=1:UseRegulator=True");

Most of parameters were already used for the backpropagation neural network. The only new one is the UseRegulator flag, which turns on the weight decay regularisation. As each iteration of the BFGS algorithm is slower than in gradient descent, the number of epochs is decreased to 100 so that training time stays within one minute approximately. However, it is visible that the training does not have enough time to reach the optimal point, which means that in a real-life application one would perform a longer training.

Usage of neural networks is only motivated when there is a significant dependence between different input variables. Otherwise the multidimensional probability density can be factorised, and the simple product of 1D likelihoods will be sufficient. Since such discriminator can be constructed easily and is very fast to train, it never hurts to add it:

factory.BookMethod(TMVA::Types::kLikelihood, jobName + "_Likelihood");

Run the final program

The program is complete now. Run it. Inspect its standard output and run the four TMVA scripts to produce some plots.

Applying existing neural network

Open file read.C. As in the previous part of the tutorial, you will be filling missing parts in the file. A completed example is in reader_Example.C.

Step 1: Create a TMVA reader

Copy the line below. The reader will accumulate information about input variables used and “booked” MVA methods.

TMVA::Reader reader("Color=True");

Step 2: Specify input variables

Provide buffers that will contain values of input variables for each event:

reader.AddVariable("glb_Charge_Lep", &glb_Charge_Lep);
reader.AddVariable("glb_Sphericity", &glb_Sphericity);
reader.AddVariable("abs(tHq_Eta_Recoil)", &abs_tHq_Eta_Recoil);
reader.AddVariable("tHq_NumBTag_Higgs", &tHq_NumBTag_Higgs);
reader.AddVariable("log(tHq_Pt_Higgs)", &log_tHq_Pt_Higgs);
reader.AddVariable("log(tHq_Pt_Recoil)", &log_tHq_Pt_Recoil);
reader.AddVariable("tt_DeltaR_Light", &tt_DeltaR_Light);
reader.AddVariable("log(tt_Mass_TopHad)", &log_tt_Mass_TopHad);
reader.AddVariable("tt_NumPassBTag_Light", &tt_NumPassBTag_Light);

The names of variables and their order must be exactly the same as in training. Buffers can only be of type Float_t or Int_t.

Step 3. Load the neural network with backpropagation

Load the neural network trained with backpropagation in the previous part of the tutorial.

reader.BookMVA("TutorialTrain_BP", "weights/TutorialTrain_TutorialTrain_BP.weights.xml");

The neural network is described in the XML file. The first argument provides a user-defined label to identify the MVA method; it is allowed from the label given to TMVA::Factory::BookMVA for training. Several MVA methods can be “booked” for same reader.

Step 4. Calculate response of the neural network

Copy this line to calculate response of the neural network loaded at the previous step:

double const response = reader.EvaluateMVA("TutorialTrain_BP");

The input variables are read from the buffers set in step 2.

Run the program

Run the program. TMVA prints a short log in the standard output. The result of the program is the plot sgnResponse.png in the current directory.