Skip to content
EvalDNN: A Toolbox for Evaluating Deep Neural Network Models
Branch: master
Clone or download

Latest commit

Fetching latest commit…
Cannot retrieve the latest commit at this time.


Type Name Latest commit message Commit time
Failed to load latest commit information.


EvalDNN is an open-source toolbox for model evaluation of deep learning systems, supporting multiple frameworks and metrics.

Author: Yongqiang Tian*, Zhihua Zeng*, Ming Wen, Yepang Liu, Tzu-yang Kuo, and Shing-Chi, Cheung.

*The first two author contribute equally.

This project is mainly supported by Microsoft Asia Cloud Research Software Fellow Award 2019

A video is here:

A paper to inroduce this tool is in submit and will be released soon.

Frameworks and Metrics

EvalDNN supports the model based on following frameworks:

  • TensorFlow
  • PyTorch
  • Keras
  • MXNet

EvalDNN supports the model based on following metrics:

  • Top-K accuracy
  • Neuron Coverage
  • Robustness



pip install EvalDNN

Evaluate a model

Check demo/demo.ipynb.

More examples are avaiable to the evaldnn/benchmarks/ and evaldnn/tests The examples covers all 4 frameworks and 3 metrics.


Add a new framework

Create a new .py under evaldnn.models then follow the exising implementation in evaldnn.models

Add a new metric

Create a new .py under evaldnn.metrics then follow the exising implementation in evaldnn.metrics


The full benchmark is available here:

The code to reproduce the results in benchmark is in evaldnn/benchmarks/. For example, run

python3 evaldnn/benchmarks/eval_keras

To evaluate the model using ImageNet dataset, please download it and put it into /EvalDNN-data/ILSVRC2012_img_val.

The file ILSVRC2012_validation_ground_truth.txt in github release should also be put into above folder.

You can’t perform that action at this time.