Skip to content
Branch: master
Find file History
bhavani-subramanian and s1113950 Fix for accuracy regression in RN50 (#250)
* Addressed review comments.

* Moved FP32 scripts to inference directory.

* Added support for calibration.

* Update model_init.py

* Update eval_image_classifier_inference.py

* Updated resnet50 unit tests

* fix the error with running inference without calibrate option, fix error when generate min maxranges.

* Update test file to remove --calibrate=False
Latest commit b2684be Mar 29, 2019
Permalink
Type Name Latest commit message Commit time
..
Failed to load latest commit information.
inference
README.md
__init__.py Flake8 style/formatting fixes (#31) Nov 14, 2018

README.md

ResNet50

This document has instructions for how to run ResNet50 for the following precisions:

Benchmarking instructions and scripts for ResNet50 model inference on Int8 and FP32 precisions.

Int8 Inference Instructions

  1. Download the full ImageNet dataset and convert to the TF records format.
  • Clone the tensorflow/models repository:
$ git clone https://github.com/tensorflow/models.git

The TensorFlow models repo provides scripts and instructions to download, process and convert the ImageNet dataset to the TF records format.

  • The ImageNet dataset directory location is only required to calculate the model accuracy.
  1. Download the pre-trained model.
$ wget https://storage.googleapis.com/intel-optimized-tensorflow/models/resnet50_int8_pretrained_model.pb
  1. Clone the intelai/models repository
$ git clone https://github.com/IntelAI/models.git
  1. Run the inference script launch_benchmark.py with the appropriate parameters to evaluate the model performance and/or calculate the accuracy. The optimized ResNet50 model files are attached to the intelai/models repo and located at models/models/image_recognition/tensorflow/resnet50/.

    The docker image (intelaipg/intel-optimized-tensorflow:PR25765-devel-mkl) used in the commands above were built using TensorFlow master (e889ea1) and PR #25765.

  • Calculate the model accuracy, the required parameters parameters include: the ImageNet dataset location (from step 1), the pre-trained final_int8_resnet50.pb input graph file (from step 2), and the --accuracy-only flag.
$ cd /home/<user>/models/benchmarks

$ python launch_benchmark.py \
    --data-location /home/<user>/dataset/FullImageNetData_directory
    --in-graph /home/<user>/resnet50_int8_pretrained_model.pb \
    --model-name resnet50 \
    --framework tensorflow \
    --precision int8 \
    --mode inference \
    --batch-size=100 \
    --accuracy-only \
    --docker-image intelaipg/intel-optimized-tensorflow:PR25765-devel-mkl

The log file is saved to the value of --output-dir.

The tail of the log output when the benchmarking completes should look something like this:

Processed 49600 images. (Top1 accuracy, Top5 accuracy) = (0.7361, 0.9155)
Processed 49700 images. (Top1 accuracy, Top5 accuracy) = (0.7361, 0.9155)
Processed 49800 images. (Top1 accuracy, Top5 accuracy) = (0.7360, 0.9154)
Processed 49900 images. (Top1 accuracy, Top5 accuracy) = (0.7361, 0.9155)
Processed 50000 images. (Top1 accuracy, Top5 accuracy) = (0.7360, 0.9154)
lscpu_path_cmd = command -v lscpu
lscpu located here: /usr/bin/lscpu
Ran inference with batch size 100
Log location outside container: {--output-dir value}/benchmark_resnet50_inference_int8_20190104_212224.log
  • Evaluate the model performance: If just evaluate performance for dummy data, the --data-location is not needed. Otherwise --data-location argument needs to be specified: Calculate the model throughput images/sec, the required parameters to run the inference script would include: the pre-trained resnet50_int8_pretrained_model.pb input graph file (from step 2), and the --benchmark-only flag. It is optional to specify the number of warmup_steps and steps as extra args, as shown in the command below. If these values are not specified, the script will default to use warmup_steps=10 and steps=50.
$ cd /home/<user>/models/benchmarks

$ python launch_benchmark.py \
    --in-graph /home/<user>/resnet50_int8_pretrained_model.pb \
    --model-name resnet50 \
    --framework tensorflow \
    --precision int8 \
    --mode inference \
    --batch-size=128 \
    --benchmark-only \
    --docker-image intelaipg/intel-optimized-tensorflow:PR25765-devel-mkl
    -- warmup_steps=50 steps=500

The tail of the log output when the benchmarking completes should look something like this:

...
steps = 470, 460.113806562 images/sec
steps = 480, 460.073982602 images/sec
steps = 490, 463.289831148 images/sec
steps = 500, 463.521427264 images/sec
lscpu_path_cmd = command -v lscpu
lscpu located here: /usr/bin/lscpu
Ran inference with batch size 128
Log location outside container: {--output-dir value}/benchmark_resnet50_inference_int8_20190223_180546.log

Note that the --verbose or --output-dir flag can be added to any of the above commands to get additional debug output or change the default output location..

FP32 Inference Instructions

  1. Download the pre-trained model.
$ wget https://storage.googleapis.com/intel-optimized-tensorflow/models/resnet50_fp32_pretrained_model.pb
  1. Clone the intelai/models repository
$ git clone https://github.com/IntelAI/models.git
  1. If running resnet50 for accuracy, the ImageNet dataset will be required (if running benchmarking for throughput/latency, then dummy data will be used).

The TensorFlow models repo provides scripts and instructions to download, process, and convert the ImageNet dataset to the TF records format.

  1. Run the inference script launch_benchmark.py with the appropriate parameters to evaluate the model performance. The optimized ResNet50 model files are attached to the intelai/models repo and located at models/models/image_recognition/tensorflow/resnet50/. If benchmarking uses dummy data for inference, --data-location flag is not required. Otherwise, --data-location needs to point to point to ImageNet dataset location.
  • To measure the model latency, set --batch-size=1 and run the benchmark script as shown:
$ cd /home/<user>/models/benchmarks

$ python launch_benchmark.py \
    --in-graph /home/<user>/resnet50_fp32_pretrained_model.pb \
    --model-name resnet50 \
    --framework tensorflow \
    --precision fp32 \
    --mode inference \
    --batch-size=1 \
    --socket-id 0 \
    --docker-image intelaipg/intel-optimized-tensorflow:latest-devel-mkl

The log file is saved to the value of --output-dir.

The tail of the log output when the benchmarking completes should look something like this:

Inference with dummy data.
Iteration 1: 0.956 sec
Iteration 2: 0.018 sec
...
Iteration 38: 0.011 sec
Iteration 39: 0.011 sec
Iteration 40: 0.011 sec
Average time: 0.011 sec
Batch size = 1
Latency: 10.924 ms
Throughput: 91.541 images/sec
lscpu_path_cmd = command -v lscpu
lscpu located here: /usr/bin/lscpu
Ran inference with batch size 1
Log location outside container: {--output-dir value}/benchmark_resnet50_inference_fp32_20190104_215326.log
  • To measure the model Throughput, set --batch-size=128 and run the benchmark script as shown:
$ cd /home/<user>/models/benchmarks

$ python launch_benchmark.py \
    --in-graph /home/<user>/resnet50_fp32_pretrained_model.pb \
    --model-name resnet50 \
    --framework tensorflow \
    --precision fp32 \
    --mode inference \
    --batch-size=128 \
    --socket-id 0 \
    --docker-image intelaipg/intel-optimized-tensorflow:latest-devel-mkl

The log file is saved to the value of --output-dir.

The tail of the log output when the benchmarking completes should look something like this:

Inference with dummy data.
Iteration 1: 1.777 sec
Iteration 2: 0.657 sec
Iteration 3: 0.652 sec
...
Iteration 38: 0.653 sec
Iteration 39: 0.649 sec
Iteration 40: 0.652 sec
Average time: 0.653 sec
Batch size = 128
Throughput: 196.065 images/sec
lscpu_path_cmd = command -v lscpu
lscpu located here: /usr/bin/lscpu
Ran inference with batch size 128
Log location outside container: {--output-dir value}/benchmark_resnet50_inference_fp32_20190104_215655.log
  • To measure the model accuracy, use the --accuracy-only flag and pass the ImageNet dataset directory from step 3 as the --data-location:
$ cd /home/<user>/models/benchmarks

$ python launch_benchmark.py \
    --in-graph /home/<user>/resnet50_fp32_pretrained_model.pb \
    --model-name resnet50 \
    --framework tensorflow \
    --precision fp32 \
    --mode inference \
    --accuracy-only \
    --batch-size 100 \
    --socket-id 0 \
    --data-location /home/<user>/dataset/ImageNetData_directory \
    --docker-image intelaipg/intel-optimized-tensorflow:latest-devel-mkl

The log file is saved to the value of --output-dir. The tail of the log output when the accuracy run completes should look something like this:

...
Processed 50000 images. (Top1 accuracy, Top5 accuracy) = (0.7430, 0.9188)
lscpu_path_cmd = command -v lscpu
lscpu located here: /usr/bin/lscpu
Ran inference with batch size 100
Log location outside container: {--output-dir value}/benchmark_resnet50_inference_fp32_20190104_213452.log
  • The --output-results flag can be used along with above benchmarking or accuracy test, in order to also output a file with the inference results (file name, actual label, and the predicted label). The results output can only be used with real data.

For example, the command below is the same as the accuracy test above, except with the --output-results flag added:

$ cd /home/<user>/models/benchmarks

$ python launch_benchmark.py \
    --in-graph /home/<user>/resnet50_fp32_pretrained_model/freezed_resnet50.pb \
    --model-name resnet50 \
    --framework tensorflow \
    --precision fp32 \
    --mode inference \
    --accuracy-only \
    --output-results \
    --batch-size 100 \
    --socket-id 0 \
    --data-location /home/<user>/dataset/ImageNetData_directory \
    --docker-image intelaipg/intel-optimized-tensorflow:latest-devel-mkl

The results file will be written to the models/benchmarks/common/tensorflow/logs directory, unless another output directory is specified by the --output-dir arg. Below is an example of what the inference results file will look like:

filename,actual,prediction
ILSVRC2012_val_00025616.JPEG,96,96
ILSVRC2012_val_00037570.JPEG,656,656
ILSVRC2012_val_00038006.JPEG,746,746
ILSVRC2012_val_00023384.JPEG,413,793
ILSVRC2012_val_00014392.JPEG,419,419
ILSVRC2012_val_00015258.JPEG,740,740
ILSVRC2012_val_00042399.JPEG,995,995
ILSVRC2012_val_00022226.JPEG,528,528
ILSVRC2012_val_00021512.JPEG,424,424
...

Note that the --verbose or --output-dir flag can be added to any of the above commands to get additional debug output or change the default output location.

You can’t perform that action at this time.