VERA - The Shooter Localization System

This repository contains the code and models for the following paper:

Technical Report of the Video Event Reconstruction and Analysis (VERA) System - Shooter Localization, Models, Interface, and Beyond
Junwei Liang, Jay D. Aronson, Alexander Hauptmann

Our work received Best Demo Award at CBMI 2019.

You can find more information/try out the shooter localization system at our Project Page. Also, check out our 3D Reconstruction demo here.

Introduction

We introduce the Video Event Reconstruction and Analysis (VERA) system, enabled by established machine learning techniques and physics models, that can localize the shooter location only based on a couple of user-generated videos that capture the gunshot sound.

Demo Video     Explanatory Video

If you find this system/code useful in your research/report then please cite

@inproceedings{liang2019shooter,
  title={Shooter Localization Using Social Media Videos},
  author={Liang, Junwei and Aronson, Jay D and Hauptmann, Alexander},
  booktitle={Proceedings of the 27th ACM International Conference on Multimedia},
  pages={2280--2283},
  year={2019},
  organization={ACM}
}
@inproceedings{liang2017synchronization,
  title={Synchronization for multi-perspective videos in the wild},
  author={Liang, Junwei and Huang, Poyao and Chen, Jia and Hauptmann, Alexander},
  booktitle={2017 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP)},
  pages={1592--1596},
  year={2017},
  organization={IEEE}
}
@inproceedings{liang2017temporal,
  title={Temporal localization of audio events for conflict monitoring in social media},
  author={Liang, Junwei and Jiang, Lu and Hauptmann, Alexander},
  booktitle={2017 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP)},
  pages={1597--1601},
  year={2017},
  organization={IEEE}
}

Purpose of This Repository

This repository includes all the necessary code and models to host our VERA system on your local machine. Please report issues including:

• Bugs/security concerns
• Instsallation problems
• Explanation of the code
• Additional details needed to be explained in the technical report (or even typos)
• Feature requests

Code Overview

• web_interface includes all the code needed for the web interface. It is written in PHP with Yii framework (v1.3). There are some Chinese comments in the code since the code is from my earlier days as a website designer in China. File an issue if you have a question. To find the code for a URL, for example, for "..index.php/application/cGunshot?videoname=firstshots_36_mXwckuEw.mp4", the code is in web_interface/protected/controllers/ApplicationController.php and in function "actionCGunshot". The view code is in web_interface/themes/basic/views/application/cGunshot.php
• python_server is the backend server we designed for PHP to communicate with machine learning code, which is usually written in PYTHON with Tensorflow. Currently we use shell calls within the python server so that we could change the machine learning code on the fly without restarting the backend server.
• ml_code includes the inferencing code for gunshot detection, gun type classification and audio synchronization.

To install the system on your machine you need to know about basic Linux Shell command, Apache server and MySQL database. To modify the web interface code, you will need to know PHP, Javascript with JQuery, html/css, and some knowledge of the Model/View/Controller (MVC) framework (Yii v1). To modify the machine learning code, you will need to know PYTHON and read our paper on Video Synchronization and Audio Event Temporal Localization. I have tested the code on a CPU machine with Ubuntu 16.04.

Installation

Instructions for installing the system on your local machine can be found here.

Future Directions

1. Automatic visual synchronization of videos;
2. Automatic detection of muzzle blast sound and shockwave sound in videos;
3. Better gun type/bullet type classification to get estimation of the bullet speed range;
4. Automatic video localization - putting camera on the map.

Other Notes

1. Method 1 code is in python_server/run.py "gunshotLocalizationMethod1".
2. Method 2 code is in web_interface/themes/basic/views/application/cMainPage.php javascript function "declare_canvas_class_method2"