Grad-CAM implementation in Tensorflow
This repo is a TensorFlow implementation of Gradient class activation maps(Grad-CAM), one of visualization techniques for deep learning networks.
- Tensorflow 1.x
(which includes the pretrained(using the Imagenet dataset) VGG16 classification model file
VGG16.npy(see the README on how to download it))
python grad-cam-tf.py <path_to_image> <path_to_VGG16_npy> [top_n]
path_to_image: an image for which Grad-CAM is calculated.
path_to_VGG16_npy: path to the pretrained VGG16 model data provided in
top_n: Optional. Grad-CAM is calculated for each 'top_n' class, which is predicted by VGG16.
The following images are saved to the same directory as
- input image overlaid by Grad-CAM heatmap with the suffix
gradcamappended to the
- black-and-white Grad-CAM heatmap(suffix:
- segmented image by Grad-CAM heatmap(suffix:
- Guided Backpropagation (suffix:
- Also rank of predicted class, the class name and the probability are appended as suffix to each file names above.
- Example image from the original implementation:
the most probable class: 'boxer'(242)
The below are original image, image overlaid by Grad-CAM heatmap and segmented image by Grad-CAM heatmap.
- Image from :
the most probable class: 'desktop computer'(527)
ground truth(the second probable class): 'desk'(526)
- Image from fatal crash by Uber self-driving SUV:
the most probable class: 'traffic_light'(920)
The below are two predicted classes which are related bicycles.
the 6-th probable class: 'motor_scooter'(670)
the 19-th probable class: 'moped'(665)
Lines below are the top 20 classes predicted VGG16 model.
As you can see, VGG16 fails to detect bicycle related objects from this picture(though
motor_scooter is detected with low probability).
0: class id: 920, class name: traffic_light, probability: 0.105, synset: n06874185 traffic light, traffic signal, stoplight 1: class id: 627, class name: limousine, probability: 0.092, synset: n03670208 limousine, limo 2: class id: 475, class name: car_mirror, probability: 0.039, synset: n02965783 car mirror 3: class id: 818, class name: spotlight, probability: 0.038, synset: n04286575 spotlight, spot 4: class id: 971, class name: bubble, probability: 0.037, synset: n09229709 bubble 5: class id: 670, class name: motor_scooter, probability: 0.027, synset: n03791053 motor scooter, scooter 6: class id: 656, class name: minivan, probability: 0.024, synset: n03770679 minivan 7: class id: 845, class name: syringe, probability: 0.024, synset: n04376876 syringe 8: class id: 864, class name: tow_truck, probability: 0.022, synset: n04461696 tow truck, tow car, wrecker 9: class id: 867, class name: trailer_truck, probability: 0.018, synset: n04467665 trailer truck, tractor trailer, trucking rig, rig, articulated lorry, semi 10: class id: 555, class name: fire_engine, probability: 0.016, synset: n03345487 fire engine, fire truck 11: class id: 468, class name: cab, probability: 0.016, synset: n02930766 cab, hack, taxi, taxicab 12: class id: 407, class name: ambulance, probability: 0.015, synset: n02701002 ambulance 13: class id: 736, class name: pool_table, probability: 0.013, synset: n03982430 pool table, billiard table, snooker table 14: class id: 836, class name: sunglass, probability: 0.013, synset: n04355933 sunglass 15: class id: 772, class name: safety_pin, probability: 0.010, synset: n04127249 safety pin 16: class id: 837, class name: sunglasses, probability: 0.010, synset: n04356056 sunglasses, dark glasses, shades 17: class id: 754, class name: radio, probability: 0.010, synset: n04041544 radio, wireless 18: class id: 665, class name: moped, probability: 0.010, synset: n03785016 moped 19: class id: 444, class name: bicycle-built-for-two, probability: 0.009, synset: n02835271 bicycle-built-for-two, tandem bicycle, tandem
I think Uber has built in-house classification models for self-driving cars. But detecting persons on roads especially at night could be difficult only by visible light. As reported in Report: Uber's Self-Driving Car Sensors Ignored Cyclist In Fatal Accident it seems a woman crossing the street on a bicycle was ignored.
Detectors using an infrared camera are also needed to avoid such tragic accidents.
For your information,
Grad-CAM++, a variant of
Grad-CAM, is presented in . The authors provided their implementation.
As pointed by Questions for computing of derivatives,
their implementation is weird.
 Ramprasaath R. Selvaraju, Michael Cogswell, Abhishek Das, Ramakrishna Vedantam, Devi Parikh, Dhruv Batra. Grad-CAM: Visual Explanations from Deep Networks via Gradient-based Localization, arXiv, 2016
 J. T. Springenberg, A. Dosovitskiy, T. Brox, and M. Riedmiller. Striving for Simplicity: The All Convolutional Net, arXiv, 2014
 Aditya Chattopadhyay, Anirban Sarkar, Prantik Howlader, Vineeth N Balasubramanian. Grad-CAM++: Generalized Gradient-based Visual Explanations for Deep Convolutional Networks, arXiv, 2017