Skip to content
This repository was archived by the owner on May 30, 2019. It is now read-only.

Home

Jump to bottom
Andrey Popov edited this page Feb 12, 2015 · 11 revisions

Tutorial

This page contains step-by-step instructions for the tutorial on artificial neural networks in TMVA. In order to follow it, you need an access to a computer with a recent version of ROOT installed (and the TMVA package as a part of it). Start by checking out the tutorial's repository as explained in the README.

Introduction

The TMVA package includes a number of MVA methods, with neural networks among others, incorporated into a common framework. The manual can be found here; download it and keep open during the session.

The tutorial addresses the task of binary classification, by an example of discrimination between tH, H->bb and semileptonic ttbar events. A number of observables have been chosen to distinguish between the two. ROOT tuples with the observables are available in the data/ directory of the repository. For each process several tuples are provided, varying in the number of events.

The tutorial is focused on the usage of TMVA rather than illustrating various aspects of neural networks as a machine learning technique.

Training and validation

In this part of the tutorial you will learn how to train neural networks, check for overfitting, and evaluate their performance. Navigate into the directory tutorial/ and open script train.C in an editor. It is mostly empty, and you will fill it step-by-step. If needed, you can consult an exemplary completed version in train_Example.C. The script can be executed with the command

root -q -b train.C

You can try running it after each step you take in order to make sure that there are no errors; however, first meaningful results will be obtained after step 6 only.

Step 1. Create a factory

A central class in TMVA is Factory, and most of interactions with the package are done via an instance of this class. It manages trees with input variables, keeps track of “booked” MVA methods, and trains them. Thus, creating a factory is the first thing you do. Copy the following code into your train.C file:

TString jobName("TutorialTrain");
TFile infoFile(TString("info") + jobName + ".root", "recreate");
TMVA::Factory factory(jobName, &infoFile, "Color=True");

The code fragment also opens a ROOT file for writing and gives it to the factory. The file will contain various control and performance histograms. The last argument is a set of parameters for the factory. The parameters control verbosity and visual appearance, a complete list can be found on p. 15 of the manual.

Step 2. Provide trees with training and testing sets

There are many ways to feed training and testing sets to a factory, ranging from a single ROOT tree that contains everything (signal and background, training and testing sets) to simple ASCII files. Details are described in section 3.1.1 of the manual. Here we stick to the most robust option, providing four independent trees:

factory.AddSignalTree(sgnTrainTree, 1., TMVA::Types::kTraining);
factory.AddBackgroundTree(bkgTrainTree, 1., TMVA::Types::kTraining);
factory.AddSignalTree(sgnTestTree, 1., TMVA::Types::kTesting);
factory.AddBackgroundTree(bkgTestTree, 1., TMVA::Types::kTesting);

The second argument is a global event weight, which is applied to all events in the tree. We do not use it and specify per-event weight instead:

factory.SetWeightExpression("Weight");

“Weight” is the name of a branch in the source trees. The expression provided here is parsed with TTreeFormula and thus can include mathematical operations.

Step 3. Specify input variables

Open a ROOT session and inspect one of the input files (say, ../data/tH_10k.root) in a TBrowser. It only contains a single tree, with branches of type Float_t. Have a look at distributions of stored variables. The branch with event weights was already mentioned at the previous step. The remaining branches encode potential input variables. Note that some of them (the lepton charge and the numbers of b-tagged jets associated with a composite object) are discrete, which is not a problem for neural networks. Some other variables (e.g. mass of the hadronically decaying top quark) have very long tails.

Now return to the editor. Input variables are specified with the following syntax:

factory.AddVariable("abs(tHq_Eta_Recoil)");

The expression given is parsed with TTreeFormula and can include mathematical operations, as in the example above. One can define short labels and human readable titles for variables to obtain a better decoration in control plots; details are in section 3.1.3.

Copy the line above to your train.C and add all the remaining observables stored in the trees. Here is the complete list: glb_Charge_Lep, glb_Sphericity, tHq_Eta_Recoil, tHq_NumBTag_Higgs, tHq_Pt_Higgs, tHq_Pt_Recoil, tt_DeltaR_Light, tt_Mass_TopHad, tt_NumPassBTag_Light. Use logarithms of observables with long tails (the two transverse momenta and the mass).

Step 4. Let the factory process the source trees

Copy the following line into your train.C:

factory.PrepareTrainingAndTestTree("", "NormMode=EqualNumEvents");

It makes the factory read the training and testing sets (or split the only given set in two), evaluate expressions in input variables, and store results internally. The first argument defines a preselection cut (we do not apply any). The second argument is the list of parameters. If the data were not split into training and testing sets, the splitting is performed now, and the parameters specify how it is done; details are in section 3.1.4. Since we provided separate trees for the two sets, there is no need for the splitting.

The only parameter set in the example demands that signal and background events in the training set are independently reweighted such that the total signal normalisation equals the one of the background (more details here). Such symmetrisation usually improves performance of the classification.

Step 5. Book a backpropagation neural network

We are about to define a neural network. Note that problems typically encountered in high-energy physics data analysis are usually simple by standards of machine learning. Because of this, one should always start from a simple architecture and try a more elaborate one if only performance obtained is not satisfactory. In practice it means that usage of neural networks with more than one hidden layer is discouraged. A rule of thumb to choose the number of neurons in the hidden layer is that the optimal value is usually between the number of inputs and twice the number. However, these recommendations should not be taken too literally, and the best way is always to try several alternatives to choose the one that suits your problem most.

Technically, a neural network with backpropagation can be “booked” with the following code:

factory.BookMethod(TMVA::Types::kMLP, jobName + "_BP",
 "VarTransform=Norm:TrainingMethod=BP:NeuronType=tanh:HiddenLayers=20:"
 "EstimatorType=CE:BPMode=sequential:NCycles=500:TestRate=5");

Input variables and training set were defined before. This method specifies architecture of the neural network and its training parameters. A complete list of supported options can be found on p. 97, 98 of the manual. Here is a description of the options used in the example:

  • VarTransform=Norm All input variables are normalised, i.e. linearly transformed to occupy the range [-1, 1]. Neural networks do not like when an input spans over a large range. TMVA can also perform more advanced preprocessing of input variables, for instance, decorrelation or principal component analysis.
  • HiddenLayers=20 The network contains a single hidden layer with 20 neurons in it. One can have more than one hidden layer, separating numbers of neurons in each by commas (e.g. HiddenLayers=10,5). As mentioned above, neural networks with a single hidden layer should usually be tried first.
  • NeuronType=tanh The activation function in neurons in the hidden layer is hyperbolic tangent. The default sigmoid activation is just as good.
  • TrainingMethod=BP The training will be done with the backpropagation algorithm, using the gradient descent method.
  • EstimatorType=CE Cross-entropy will be used as the loss function. For classification problems it is usually considered a better option than the default mean squared error.
  • BPMode=sequential Sequential training will be used (instead of batch one). It should usually be preferred when working with the TMVA implementation.
  • NCycles=500 The training will last for 500 epochs. Reasonable numbers span from hundreds to tens of thousands. TMVA can also be configured to stop the training prematurely, when it detects that the loss on the training has stabilised.
  • TestRate=5 Every 5th epoch the loss function will be evaluated on the testing set, and the value will be stored for future examination.

As training proceeds, the learning rate is decreased, as controlled by the decay rate parameter (details are given here). The two rates are crucial parameters of the algorithm and can impact quality of training drastically. The default values (0.02 and 0.01 respectively) used by TMVA are reasonable for sequential training and thus are left unchanged.

Step 6. Train the neural network and test its performance

Add the following code:

factory.TrainAllMethods();
factory.TestAllMethods();
factory.EvaluateAllMethods();

It instructs the factory to train all methods booked so far and evaluate their performance. TestAllMethods performs basic evaluation only, while EvaluateAllMethods produces a number of additional control histrograms (e.g. the ROC curve).

Run the program

At this point the program is ready to be run for the first time. Start it. The execution will take about a minute, with most of the time spent in training.

TMVA prints some useful information in the standard output. Most notable are current values of the loss function calculated on the training and testing sets (note that their ratio does not mean much, as explained here). It also includes comments about transformations of input variables, linear correlations between them, ranking of inputs (which is very naive and usually should not be trusted, though), linear correlations between inputs and MVA response, signal efficiencies at few working points corresponding to predefined background efficiencies, area under the ROC curve, and other details.

Check the content of the current directory. The program created a subdirectory weights/. The subdirectory contains an XML file with a complete definition of the constructed neural network, which can be loaded in TMVA and used to apply the neural network to new events (and this will be covered in the second part of the tutorial). It is also saved in a *.C file to be used without TMVA.

The program also created file infoTutorialTrain.root in the current directory. Open it in a TBrowser and inspect the content. You can find correlation matrices for inputs and 1D and 2D distributions of signal and background over inputs (in the subdirectory InputVariables_Id/). Control plots specific to the neural network are placed in the subdirectory Method_MLP/TutorialTrain_BP/. Two groups of them are especially interesting:

  • estimatorHistTrain and estimatorHistTest show the evolution of the loss calculated on the training and testing sets respectively. These plots should always be checked to verify that the neural network has not overfit (in case of which the loss calculated on the testing set starts to grow at some point).
  • MVA_TutorialTrain_BP_rejBvsS is the ROC curve constructed on the testing set (the one from the testing set is called like *_trainingRejBvsS, but it is of less interest). The ROC curve completely describes performance of a binary discriminator and should usually be reported.

TMVA comes with several scripts that produce (arguably) nicely decorated plots of various kinds from the ROOT file. The complete list of scripts is provided at p. 32, 34. You can try running the following:

root -q -b "$ROOTSYS/tmva/test/variables.C(\"infoTutorialTrain.root\")"
root -q -b "$ROOTSYS/tmva/test/mvas.C(\"infoTutorialTrain.root\", 3)"
root -q -b "$ROOTSYS/tmva/test/efficiencies.C(\"infoTutorialTrain.root\")"
root -q -b "$ROOTSYS/tmva/test/annconvergencetest.C(\"infoTutorialTrain.root\")"

The produced figures are saved in the plots/ subdirectory.

Step 7. Add a BFGS neural network and a projective likelihood discriminator

Same factory can include several MVA methods. Add a neural network trained with the BFGS algorithm with weight decay regularisation:

factory.BookMethod(TMVA::Types::kMLP, jobName + "_BFGS_WeightDecay",
 "VarTransform=Norm:TrainingMethod=BFGS:NeuronType=tanh:HiddenLayers=20:EstimatorType=CE:"
 "NCycles=100:TestRate=1:UseRegulator=True");

Most of parameters were already used for the backpropagation neural network. The only new one is the UseRegulator flag, which turns on the weight decay regularisation. As each iteration of the BFGS algorithm is slower than in gradient descent, the number of epochs is decreased to 100 so that training time stays within one minute approximately. However, it is visible that the training does not have enough time to reach the optimal point, which means that in a real-life application one would perform a longer training.

Usage of neural networks is only motivated when there is a significant dependence between different input variables. Otherwise the multidimensional probability density can be factorised, and the simple product of 1D likelihoods will be sufficient. Since such discriminator can be constructed easily and is very fast to train, it never hurts to add it:

factory.BookMethod(TMVA::Types::kLikelihood, jobName + "_Likelihood");

Run the final program

The program is complete now. Run it. Inspect its standard output and run the four TMVA scripts to produce some plots.

Applying existing neural network

Now we will study how to apply a neural network created before. Open file read.C in an editor. As in the previous part of the tutorial, you will be filling missing parts in the file. A completed example is in reader_Example.C.

Step 1: Create a TMVA reader

Copy the line below. The reader will host information about input variables used and “booked” MVA methods.

TMVA::Reader reader("Color=True");

Step 2: Specify input variables

Provide buffers that will contain values of input variables for each event:

reader.AddVariable("glb_Charge_Lep", &glb_Charge_Lep);
reader.AddVariable("glb_Sphericity", &glb_Sphericity);
reader.AddVariable("abs(tHq_Eta_Recoil)", &abs_tHq_Eta_Recoil);
reader.AddVariable("tHq_NumBTag_Higgs", &tHq_NumBTag_Higgs);
reader.AddVariable("log(tHq_Pt_Higgs)", &log_tHq_Pt_Higgs);
reader.AddVariable("log(tHq_Pt_Recoil)", &log_tHq_Pt_Recoil);
reader.AddVariable("tt_DeltaR_Light", &tt_DeltaR_Light);
reader.AddVariable("log(tt_Mass_TopHad)", &log_tt_Mass_TopHad);
reader.AddVariable("tt_NumPassBTag_Light", &tt_NumPassBTag_Light);

The names of variables and their order must be exactly the same as was specified for training. Buffers can only be of type Float_t or Int_t.

Step 3. Load the neural network with backpropagation

Load the neural network trained with backpropagation in the previous part of the tutorial.

reader.BookMVA("TutorialTrain_BP", "weights/TutorialTrain_TutorialTrain_BP.weights.xml");

The neural network is described in the XML file. The first argument provides a user-defined label to identify the MVA method; it can differ from the label given to TMVA::Factory::BookMVA for training. Several MVA methods can be “booked” in the same reader.

Step 4. Calculate response of the neural network

Copy this line to calculate response of the neural network loaded at the previous step:

double const response = reader.EvaluateMVA("TutorialTrain_BP");

The input variables are read from the buffers set in step 2. Thus, some user code should take care of updating the buffers, like how it is done in train.C.

Run the program

Run the program. TMVA prints a short log in the standard output (which can be suppressed by giving an appropriate option to the constructor of the reader). The result of the program is the plot sgnResponse.png in the current directory.

Clone this wiki locally