Skip to content
No description, website, or topics provided.
Python Shell Jupyter Notebook
Branch: test
Clone or download
Type Name Latest commit message Commit time
Failed to load latest commit information.

DEEP Open Catalogue: API for Tensorflow Benchmarks

Build Status

tf_cnn_benchmarks from TensorFlow team accessed via DEEPaaS API, currently V1.

From tf_cnn_benchmarks:

tf_cnn_benchmarks contains implementations of several popular convolutional models, and is designed to be as fast as possible. tf_cnn_benchmarks supports both running on a single machine or running in distributed mode across multiple hosts. See the High-Performance models guide for more information.

These models utilize many of the strategies in the TensorFlow Performance Guide. Benchmark results can be found here.

These models are designed for performance. For models that have clean and easy-to-read implementations, see the TensorFlow Official Models.

Project Organization

├──              <- The top-level README for developers using this project.
├── data
│   └── raw                <- The original, immutable data dump.
├── docs                   <- A default Sphinx project; see for details
├── models                 <- Trained and serialized models, model predictions, or model summaries
├── notebooks              <- Jupyter notebooks. Naming convention is a number (for ordering),
│                             the creator's initials (if many user development), 
│                             and a short `_` delimited description, e.g.
│                             `1.0-jqp-initial_data_exploration.ipynb`.
├── references             <- Data dictionaries, manuals, and all other explanatory materials.
├── reports                <- Generated analysis as HTML, PDF, LaTeX, etc.
│   └── figures            <- Generated graphics and figures to be used in reporting
├── requirements.txt       <- The requirements file for reproducing the analysis environment, e.g.
│                             generated with `pip freeze > requirements.txt`
├── test-requirements.txt  <- The requirements file for the test environment
├──               <- makes project pip installable (pip install -e .) so benchmarks_api can be imported
├── benchmarks_cnn_api    <- Source code for use in this project.
│   ├──        <- Makes benchmarks_api a Python module
│   │
│   ├── dataset            <- Scripts to download or generate data
│   │   └──
│   │
│   ├── features           <- Scripts to turn raw data into features for modeling
│   │   └──
│   │
│   ├── models             <- Scripts to train models and then use trained models to make
│   │   │                     predictions
│   │   └──
│   │
│   └── tests              <- Scripts to perfrom code testing + pylint script
│   │
│   └── visualization      <- Scripts to create exploratory and results oriented visualizations
│       └──
└── tox.ini                <- tox file with settings for running tox; see

Project based on the cookiecutter data science project template. #cookiecutterdatascience

Getting Started

To run ResNet50 with synthetic data without distortions with a single GPU, run

python --num_gpus=1 --batch_size=32 --model=resnet50 --variable_update=parameter_server

Note that the master branch of tf_cnn_benchmarks requires the latest nightly version of TensorFlow. You can install the nightly version by running pip install tf-nightly-gpu in a clean environment, or by installing TensorFlow from source. We sometimes will create a branch of tf_cnn_benchmarks, in the form of cnn_tf_vX.Y_compatible, that is compatible with TensorFlow version X.Y For example, branch cnn_tf_v1.9_compatible works with TensorFlow 1.9.

Some important flags are

  • model: Model to use, e.g. resnet50, inception3, vgg16, and alexnet.
  • num_gpus: Number of GPUs to use.
  • data_dir: Path to data to process. If not set, synthetic data is used.
  • batch_size: Batch size for each GPU.
  • variable_update: The method for managing variables: parameter_server ,replicated, distributed_replicated, independent
  • local_parameter_device: Device to use as parameter server: cpu or gpu.

To see the full list of flags, run python --help, or read OPTIONS.txt

To run ResNet50 with real data with 8 GPUs, run:

python --data_format=NCHW --batch_size=256 \
--model=resnet50 --optimizer=momentum --variable_update=replicated \
--nodistortions --gradient_repacking=8 --num_gpus=8 \
--num_epochs=90 --weight_decay=1e-4 --data_dir=${DATA_DIR} --use_fp16 \

This will train a ResNet-50 model on ImageNet with 2048 batch size on 8 GPUs. The model should train to around 76% accuracy.

Using Real datasets

Four real datasets can be used:

  • CIFAR10: 60000 32x32 colour images in 10 classes, with 6000 images per class. There are 50000 training images and 10000 test images [CIFAR page from Uni Toronto]
  • IMAGENET: classical dataset features a collection of 1.2 million labeled images with one thousand object categories used for training data in the image-net competition [Official webpage]
  • COCO: a large-scale object detection, segmentation, and captioning dataset [Official webpage]
  • LIBRISPEECH: Large-scale (1000 hours) corpus of read English speech [Official webpage]


Please, go to [CIFAR page from Uni Toronto] and download the dataset, de-archive it in a certain directory, provide this directory with --data_dir flag.

Optionally provide also --data_name=cifar10.


According the official Tensorflow Benchmark README for TF1.10, in order to use Imagenet data use these instructions as a starting point.


not yet tested by us


not yet tested by us

Running the tests

To run the tests, run

pip install portpicker
python && python --run_distributed_tests

Note the tests require portpicker.

The command above runs a subset of tests that is both fast and fairly comprehensive. Alternatively, all the tests can be run, but this will take a long time:

python --full_tests && python --full_tests --run_distributed_tests

We will run all tests on every PR before merging them, so it is not necessary to pass --full_tests when running tests yourself.

To run an individual test, such as method testParameterServer of test class TfCnnBenchmarksTest of module benchmark_cnn_test, run

python -m unittest -v benchmark_cnn_test.TfCnnBenchmarksTest.testParameterServer
You can’t perform that action at this time.