Skip to content
PyTorch code for Class Visualization Pyramid for intpreting spatio-temporal class-specific activations throughout the network
Python
Branch: master
Clone or download
Fetching latest commit…
Cannot retrieve the latest commit at this time.
Permalink
Type Name Latest commit message Commit time
Failed to load latest commit information.
images Public version v1 Sep 18, 2019
models added workflow for visualizations Jul 19, 2019
utils visualization of unrotated tubes Jul 19, 2019
weights added mfnet Jul 13, 2019
.gitignore added mfnet Jul 13, 2019
LICENSE
README.md Correct arXiv link Sep 23, 2019
flops.py added workflow for visualizations Jul 19, 2019
main.py Added support for cross-channel 3d convs Jul 16, 2019
saliency_tubes.py

README.md

Class Feature Pyramids

supported versions GitHub license supported versions Tweet


Introduction

We introduce Class Feature Pyramids, a method that traverses an entire network structure and incrementally discovers kernels at different network depths that are informative for a specific class. Our method does not depend on the network’s architecture or the type of 3D convolutions, supporting grouped and depth-wise convolutions, convolutions in fibers, and convolutions in branches.

CFVP-active

2019 ICCV Workshop on
Interpreting and Explaining Visual Artificial Intelligence Models    
[arXiv preprint]

Dependencies

Make sure that the following packages are installed in your machine:

  • OpenCV
  • Scipy
  • PyTorch

Alternatively, they cane be installed with the following command:

$ pip install opencv-python scipy torch torchvision

We offer an additional frame-reading method based on a frame SQL database, in the case that the frames are stores in such format (for smaller inode requirements and faster loading times).

Models and weights used in the paper, are based on the following repositories:

Installation

Please make sure, Git is installed in your machine:

$ sudo apt-get update
$ sudo apt-get install git
$ git clone https://github.com/alexandrosstergiou/Class_Feature_Visualization_Pyramid.git

Getting started

CFVP-static

For using Class Feature Pyramids ensure that the main file is called with the following parser arguments:

python main.py
--num_classes [number of classes]
--model_weights [The filepath of the .pth weight file]
--frame_dir [directory of the video/clip]
--frames_start [starting frame number]
--frames_end [ending frame number]
--label [class to backstep for]
--threshold [numeric threshold for the kernel-based activations]
--backprop_depth [backwards network depth to backstep to - if 1 only the predictions layer is used]
--visualisation_method [defines the kernels to be visualised]

The network to be used as well as the number of GPUs are currently manually defined in the code (lines 513 & 517). However, they will be integrated in the parser soon.

Apart from creating a folder containing layer and kernel based saliency tubes, a JSON file is also created that contains a full list of all connections across kernels and layers for the specific class and example chosen. This can be used in conjunction with a visualising tool such as D3.


Results

Example results of biking class of HMDB-51 with different networks and network depths:

CFVP-active

Performance of Class Feature Pyramids

Running times were based on a 2 x Nvidia GTX1080Ti machine with an Intel i7-8700 CPU.

Network GFLOPS Back-step time (msec) # layers theta
Multi-FiberNet 22.70 24.43 3 0.6
I3D 55.79 23.21 1 + mixed5c 0.65
ResNet50-3D 80.32 21.39 3 0.55
ResNet101-3D 110.98 39.48 3 0.6
ResNet152-3D 148.91 31.06 3 0.6
ResNeXt101-3D 76.96 70.49 3 0.6

Citing Class Feature Pyramids

If you use our code in your research or wish to refer to the baseline results, please use the following BibTeX entry:

@article{stergiou2019classfeaturepyramid,
title={Class Feature Pyramids for Video Explanation},
author={Stergiou, Alexandros and Kapidis, Georgios and Kalliatakis, Grigorios and Chrysoulas, Christos and Veltkamp, Remco and Poppe, Ronald},
journal={arXiv preprint arXiv:1909.08611},
year={2019}
}

:octocat:
We use GitHub issues to track public bugs. Report a bug by opening a new issue.

You can’t perform that action at this time.