Video Anomaly Detection - with Azure ML and MLOps

The automation of detecting anomalous event sequences in videos is a challenging problem, but also has broad applications across industry verticals.

The approach followed in this repository involves self-supervised training deep neural networks to develop an in-depth understanding of the physical and causal rules in the observed scenes. The model effectively learns to predict future frames in the video in a self-supervised fashion.

The trained model can then be used to detect anomalies in videos. As the model tries to predict each next frame, one can calculate the error in the model's prediction. If the error is large, it is likely that an anomalous even occurred.

The approach can be used both in a supervised and unsupervised fashion, thus enabling the detection of pre-defined anomalies, but also of anomalous events that have never occurred in the past.

Post on LinkedIn (includes video)

Example scenarios

The following snapshots are also links to video demonstrations.

Unsupervised Approach:

Supervised Approach:

Learning Goals

You will learn:

1. How to adapt an existing neural network architecture to your use-case.
2. How to prepare video data for deep learning.
3. How to perform hyperparameter tuning with HyperDrive to improve the performance of your model.
4. How to deploy a deep neural network as a webservice for video processing.
5. How to post-process the output of a Keras model for secondary tasks (here, anomaly detection).
6. How to define a build pipeline for DevOps.

Pre-requisites

Skills

1. Some familiarity with concepts and frameworks for neural networks:
   • Framework: Keras and Tensorflow
   • Concepts: convolutional, recurrent, and pooling layers.
2. Knowledge of basic data science and machine learning concepts. Here and here you'll find short introductory material.
3. Moderate skills in coding with Python and machine learning using Python. A good place to start is here.

Software Dependencies

• Various python modules. We recommend working with a conda environement (see config/environment.yml and Documentation).
  • We recommend you begin by installing Miniconda.
  • Also consider running the following command to update you conda base image: conda update -n base -c defaults conda
  • Then, you could run: conda env create -f config/environment.yml
• If you are using a DSVM:
  • We recommend VS code https://code.visualstudio.com/ with ssh - remote extension.
  • We recommend X2Go https://wiki.x2go.org/doku.php

Hardware Dependencies

A computer with a GPU, for example a Linux Azure VM. Compare VM sizes and prices). If you don't know what to choose, we recommend the Standard NC6, the most affordable VM with a GPU.

You could create a VM in the Azure Portal.

Dataset

UCSD Anomaly Detection Dataset

Agenda

The recommended first step is to clone this repository.

Getting Started

1. Data Preparation - Download and prepare data for training/testing.
2. Azure ML Configuration - Configure your Azure ML workspace.
3. AML Pipelines - Automate data preparation, training, and re-training.
4. Deployment - How to deploy your anomaly detection as a webservice on AKS.

Deep-dive

1. Model Development - Understand model architecture and training.
2. Fine Tuning - Perform transfer learning with pretrained model onnew data.
3. Hyperparameter tuning - Tune hyperparameters with HyperDrive.
4. Anomaly Detection - Use Model errors for detecting anomalies.

Contribute

We invite contributions to this repository. The preferred method would be to fork this repository and to create a pull request.

References / Resources

• Research Article: Deep predictive coding networks for video prediction and unsupervised learning by Lotter, W., Kreiman, G. and Cox, D., 2016.

   @article{lotter2016deep,
   title={Deep predictive coding networks for video prediction and unsupervised learning},
   author={Lotter, William and Kreiman, Gabriel and Cox, David},
   journal={arXiv preprint arXiv:1605.08104},
   year={2016}
   }
  

• Original Prednet implementation is on github.com.

• Interesting blog post on Self-Supervised Video Anomaly Detection by Steve Shimozaki

Troubleshooting

Missing cuda libraries

We noticed that tensorflow might be looking for the wrong the of cudatoolkit.

You might see the following error:

Could not load dynamic library 'libcudart.so.10.0'; dlerror: libcudart.so.10.0: cannot open shared object file: No such file or directory

We found that it helps to look at which version of these libraries you have installed, and pin the version of cudatoolkit to the version that tensorflow is looking for. E.g. above, TF is looking for 10.0, so we pinned the version of the cudatoolkit to 10.0 in config/environment.yml.