TensorFlow Benchmark for the HEP Deep Learning Model used in arXiv:1708.05256 which uses 224x224x3 images and a smaller variant using 64x64x3 images.
*** Copyright Notice *** High Energy Physics Deep Learning Convolutional Neural Network Benchmark (HEPCNNB) Copyright (c) 2017, The Regents of the University of California, through Lawrence Berkeley National Laboratory (subject to receipt of any required approvals from the U.S. Dept. of Energy). All rights reserved.
If you have questions about your rights to use or distribute this software, please contact Berkeley Lab's Innovation and Partnerships Office at IPO@lbl.gov.
NOTICE. This Software was developed under funding from the U.S. Department of Energy and the U.S. Government consequently retains certain rights. As such, the U.S. Government has been granted for itself and others acting on its behalf a paid-up, nonexclusive, irrevocable, worldwide license in the Software to reproduce, distribute copies to the public, prepare derivative works, and perform publicly and display publicly, and to permit other to do so.
The main python training scripts are in scripts
. These scripts do the command
line parsing, set up distributed training, and call the training loop.
hep_classifier_tf_train.py
is for standard Tensorflow serial and distributed training.hep_classifier_tf_train_horovod.py
is for distributed Tensorflow training with Horovod.hep_classifier_tf_train_craype-ml.py
is for distributed Tensorflow training with the Cray ML plugin.
The CNN classifier model is defined in networks/binary_classifier_tf.py
along with
some data utility code and routines for model performance metrics.
The slurm_tf_helper/setup_clusters.py
module contains some helper code which
parses SLURM parameters and returns a cluster object which can be used by
TensorFlow for distributed training.
The configs
folder holds json configuration files for running distributed training on the NERSC Cori system, either on the Xeon Phi 7250 (Knight's Landing, KNL) partition or the Intel Xeon Haswell partition. These files further contain the network parameters for the training and the paths to the relevant data. Users need to specify the inputpath
and point it to their data directory. The data description is given [below](## Data description).
The folder run_scripts
contains batch scripts for running distributed training on the NERSC Cori machine. They set environment parameters, correct thread binding and encapsulate other boiler-plate settings so that the user does not need to worry about that. They are supposed to be submitted from that directory, otherwise the relative path logic will break. By default, the scripts submit the 224x224x3 image size case but they can also be used for submitting the smaller network. This can be done by changing the --config
argument to point to the corresponding json file in the previously mentioned configs
folder.
In order to submit these scripts at NERSC, do
sbatch -N <numnodes> run_scripts/run_cori_knl.sh
for running distributed training on <numnodes>
nodes on the Cori KNL partition.
If -N
is not specified, the training will be performed on a single node.
By default, the distributed training uses 1 parameters server if the number of nodes is bigger than one. However, this can easily be changed by using the --num_ps
variable in the run scripts.
The data represents simulated detector data from a (non-existant) general purpose detector which shares many features with ATLAS detector at CERN. The dataset was generated with Delphes, a fast Monte-Carlo generator for particle collisions. This simulator computes particle collisions and emulates a parton shower, i.e. particles produced in the collision propagate and eventually decay or get absorbed by the detector material. This whole process is called an event. The original data consisted of cylindrical coordinates with energy deopisits in the detectors Hadron Calorimeter as well as Electromagnetic Calorimeter as well as track information inferred from the pixel detector. This continuous data is binned to 64x64x3 (small network) or 224x224x3 (large network) size images. The x-dimension represents the azimuth angle
In our test, we generated particle events which can be fully described by standard model physics and particle events which contain new physics, in our case R-parity violating Supersymmetry (check this link for a review). However, the user is feel to generate his own data with different event types. The model can also be easily expanded to perform a multi-class classification or even include a regression on model parameters.
For NERSC users, we can give access to the our dataset on request. Please send an email to (consult@nersc.gov) and ask for acces to the HEP TensorFlow CNN benchmark data.
For users who want to generate their own data, this is the data format which is expected by the data iterator specified in networks/binary_classifier_tf.py
.
The users are free to edit this file or stick to our convention which is:
h5ls <trainfile-name>.h5
data Dataset {<nsamples>, 3, 224, 224}
label Dataset {<nsamples>}
normweight Dataset {<nsamples>}
weight Dataset {<nsamples>}
We will give a short decription of the data here:
<nsamples>
: number of samples in the file
data
: images which are fed to the neural network
label
: integer label
normweight
: normalized weight. Each sample is weighted by that weight given here. Currently overridden and set to 1, but can be changed. The normalization should be done over the training set.
weight
: unnormalized weight. That field can be used to compute more sophisticated performance metrics to account for highly skewed data sets.
The benchmark can be run with dummy data only. For that purpose, just append --dummy_data
to the command line arguments of the main python script. The loss is meaningless as random data will be generated for the training process but this mode can be used to perform IO insensitive scaling tests.