Video Analytics Serving

Video Analytics Serving is

Docker container based Video AI Serving Pipeline for Object detection, Emotion detection & Audio detection powered by Microservice and Optimized python package for deploying performance optimized media analytics pipelines.
It offers pipelines with APIs to 💡 DISCOVER, START, STOP, CUSTOMIZE and MONITOR pipeline execution with support for GStreamer* or FFmpeg*
Video Analytics Serving is based on OpenVINO^™ Toolkit DL Streamer and FFmpeg Video Analytics.

Features Include:


Customizable Media Analytics Containers	Scripts and dockerfiles to build and run container images with the required dependencies for hardware optimized media analytics pipelines.
No-Code Pipeline Definitions and Templates	JSON based definition files, a flexible way for developers to define and parameterize pipelines while abstracting the low level details from their users.
Deep Learning Model Integration	A simple way to package Deep NN based Cognitive models in pipeline definitions. The precision of a model can be auto-selected at runtime based on the chosen inference device.
Video Analytics Serving Python API	A python module to discover, start, stop, customize and monitor pipelines based on their no-code definitions.
Video Analytics Serving Microservice	A RESTful microservice providing endpoints and APIs matching the functionality of the python module.

IMPORTANT: Video Analytics Serving is provided as a sample. It is not intended to be deployed into production environments without modification. Developers deploying Video Analytics Serving should review it against their production requirements.

Getting Started

The sample microservice includes three media analytics pipelines.


object_detection	Detect and label objects such as bottles and bicycles.
emotion_recognition	Detect the emotions of a person within a video stream.
audio_detection	Analyze audio streams for events such as breaking glass or barking dogs.

Prerequisites


Docker	Video Analytics Serving requires Docker for it's build, development, and runtime environments. Please install the latest for your platform. Docker.
bash	Video Analytics Serving's build and run scripts require bash and have been tested on systems using versions greater than or equal to: `GNU bash, version 4.3.48(1)-release (x86_64-pc-linux-gnu)`. Most users shouldn't need to update their version but if you run into issues please install the latest for your platform. Instructions for macOS®* users here.
curl	The samples below use the `curl` command line program to issue standard HTTP requests to the microservice. Please install the latest for your platform. Note: any other tool or utility that can issue standard HTTP requests can be used in place of `curl`.

Building the Microservice

Build the sample microservice with the following command:

./docker/build.sh

The script will automatically include the sample models, pipelines and required dependencies.

Note: When running this command for the first time, the default base image for Video Analytics Serving will take a long time to build (likely over an hour). For instructions on how to re-use pre-built base images to speed up the build time please see the following documentation.

To verify the build succeeded execute the following command:

docker images video-analytics-serving-gstreamer:latest

Expected output:

REPOSITORY                          TAG                 IMAGE ID            CREATED             SIZE
video-analytics-serving-gstreamer   latest              f51f2695639f        2 minutes ago          1.39GB

Running the Microservice

Start the sample microservice with the following command:

./docker/run.sh -v /tmp:/tmp

This script issues a standard docker run command to launch the container, run a Tornado based web service on port 8080, and mount the /tmp folder. The /tmp folder is mounted to share sample results with the host and is optional in actual deployments.

Expected output:

{"levelname": "INFO", "asctime": "2020-08-06 12:37:12,139", "message": "=================", "module": "pipeline_manager"}
{"levelname": "INFO", "asctime": "2020-08-06 12:37:12,139", "message": "Loading Pipelines", "module": "pipeline_manager"}
{"levelname": "INFO", "asctime": "2020-08-06 12:37:12,139", "message": "=================", "module": "pipeline_manager"}
(gst-plugin-scanner:14): GStreamer-WARNING **: 12:37:12.476: Failed to load plugin '/root/gst-video-analytics/build/intel64/Release/lib/libvasot.so': libopencv_video.so.4.4: cannot open shared object file: No such file or directory
{"levelname": "INFO", "asctime": "2020-08-06 12:37:13,207", "message": "FFmpeg Pipelines Not Enabled: ffmpeg not installed\n", "module": "pipeline_manager"}
{"levelname": "INFO", "asctime": "2020-08-06 12:37:13,208", "message": "Loading Pipelines from Config Path /home/video-analytics-serving/pipelines", "module": "pipeline_manager"}
{"levelname": "INFO", "asctime": "2020-08-06 12:37:13,223", "message": "Loading Pipeline: audio_detection version: 1 type: GStreamer from /home/video-analytics-serving/pipelines/audio_detection/1/pipeline.json", "module": "pipeline_manager"}
{"levelname": "INFO", "asctime": "2020-08-06 12:37:13,230", "message": "Loading Pipeline: object_detection version: 1 type: GStreamer from /home/video-analytics-serving/pipelines/object_detection/1/pipeline.json", "module": "pipeline_manager"}
{"levelname": "INFO", "asctime": "2020-08-06 12:37:13,240", "message": "Loading Pipeline: emotion_recognition version: 1 type: GStreamer from /home/video-analytics-serving/pipelines/emotion_recognition/1/pipeline.json", "module": "pipeline_manager"}
{"levelname": "INFO", "asctime": "2020-08-06 12:37:13,241", "message": "===========================", "module": "pipeline_manager"}
{"levelname": "INFO", "asctime": "2020-08-06 12:37:13,241", "message": "Completed Loading Pipelines", "module": "pipeline_manager"}
{"levelname": "INFO", "asctime": "2020-08-06 12:37:13,241", "message": "===========================", "module": "pipeline_manager"}
{"levelname": "INFO", "asctime": "2020-08-06 12:37:13,333", "message": "Starting Tornado Server on port: 8080", "module": "__main__"}

Detecting Objects in a Video

Example Request:

Endpoint	Verb	Request	Response
pipelines/object_detection/1	POST	JSON { "source": { "uri": "https://example.mp4", "type": "uri" }, "destination": { "type": "file", "path": "/tmp/results_objects.txt", "format": "json-lines" } }	200 Pipeline Instance Id

Curl Command:

Start a new shell and execute the following command to issue an HTTP POST request, start a pipeline and analyze a sample video.

curl localhost:8080/pipelines/object_detection/1 -X POST -H \
'Content-Type: application/json' -d \
'{ 
  "source": {
    "uri": "https://github.com/intel-iot-devkit/sample-videos/blob/master/bottle-detection.mp4?raw=true",
    "type": "uri"
  },
  "destination": {
    "type": "file",
    "path": "/tmp/results_objects.txt",
    "format": "json-lines"
  }
}'

Detection Results:

To view incremental results, execute the following command from the shell.

tail -f /tmp/results_objects.txt

As the video is being analyzed and as objects appear and disappear you will see detection results in the output.

Expected Output:

{"objects":[{"detection":{"bounding_box":{"x_max":0.9022353887557983,"x_min":0.7940621376037598,"y_max":0.8917602300643921,"y_min":0.30396613478660583},"confidence":0.7093080282211304,"label":"bottle","label_id":5},"h":212,"roi_type":"bottle","w":69,"x":508,"y":109}],"resolution":{"height":360,"width":640},"source":"https://github.com/intel-iot-devkit/sample-videos/blob/master/bottle-detection.mp4?raw=true","timestamp":39553072625}

After pretty-printing:

{
  "objects": [
    {
      "detection": {
        "bounding_box": {
          "x_max": 0.9022353887557983,
          "x_min": 0.7940621376037598,
          "y_max": 0.8917602300643921,
          "y_min": 0.30396613478660583
        },
        "confidence": 0.7093080282211304,
        "label": "bottle",
        "label_id": 5
      },
      "h": 212,
      "roi_type": "bottle",
      "w": 69,
      "x": 508,
      "y": 109
    }
  ],
  "resolution": {
    "height": 360,
    "width": 640
  },
  "source": "https://github.com/intel-iot-devkit/sample-videos/blob/master/bottle-detection.mp4?raw=true",
  "timestamp": 39553072625
}

Industrialization Samples & Code Sets

Emotion Recognition

Recognizing Emotions in a Video

Example Request:

Endpoint	Verb	Request	Response
/pipelines/emotion_recognition/1	POST	JSON { "source": { "uri": "https://example.mp4", "type": "uri" }, "destination": { "type": "file", "path": "/tmp/results_emotions.txt", "format": "json-lines" } }	200 Pipeline Instance Id

Curl Command:

Start a new shell and execute the following command to issue an HTTP POST request, start a pipeline and analyze a sample video.

curl localhost:8080/pipelines/object_detection/1 -X POST -H \
'Content-Type: application/json' -d \
'{ 
  "source": {
    "uri": "https://github.com/intel-iot-devkit/sample-videos/blob/master/head-pose-face-detection-male.mp4?raw=true",
    "type": "uri"
  },
  "destination": {
    "type": "file",
    "path": "/tmp/results_emotions.txt",
    "format": "json-lines"
  }
}'

Detection Results:

To view incremental results, execute the following command from the shell.

tail -f /tmp/results_emotions.txt

As the video is being analyzed and as the person's emotions appear and disappear you will see recognition results in the output.

Expected Output:

{"objects":[{"detection":{"bounding_box":{"x_max":0.567557156085968,"x_min":0.42375022172927856,"y_max":0.5346322059631348,"y_min":0.15673652291297913},"confidence":0.9999996423721313,"label":"face","label_id":1},"emotion":{"label":"neutral","model":{"name":"0003_EmoNet_ResNet10"}},"h":163,"roi_type":"face","w":111,"x":325,"y":68}],"resolution":{"height":432,"width":768},"source":"https://github.com/intel-iot-devkit/sample-videos/blob/master/head-pose-face-detection-male.mp4?raw=true","timestamp":13333333333}

After pretty-printing:

{
  "objects": [
    {
      "detection": {
        "bounding_box": {
          "x_max": 0.567557156085968,
          "x_min": 0.42375022172927856,
          "y_max": 0.5346322059631348,
          "y_min": 0.15673652291297913
        },
        "confidence": 0.9999996423721313,
        "label": "face",
        "label_id": 1
      },
      "emotion": {
        "label": "neutral",
        "model": {
          "name": "0003_EmoNet_ResNet10"
        }
      },
      "h": 163,
      "roi_type": "face",
      "w": 111,
      "x": 325,
      "y": 68
    }
  ],
  "resolution": {
    "height": 432,
    "width": 768
  },
  "source": "https://github.com/intel-iot-devkit/sample-videos/blob/master/head-pose-face-detection-male.mp4?raw=true",
  "timestamp": 13333333333
}

Audio Event Detection

Detecting Audio Events in an Audio Recording

Example Request:

Endpoint	Verb	Request	Response
/pipelines/audio_detection/1	POST	JSON { "source": { "uri": "https://example.wav", "type": "uri" }, "destination": { "type": "file", "path": "/tmp/results_audio_events.txt", "format": "json-lines" } }	200 Pipeline Instance Id

Curl Command:

Start a new shell and execute the following command to issue an HTTP POST request, start a pipeline and analyze a sample audio.

curl localhost:8080/pipelines/audio_detection/1 -X POST -H \
'Content-Type: application/json' -d \
'{ 
  "source": {
    "uri": "https://github.com/opencv/gst-video-analytics/blob/preview/audio-detect/samples/gst_launch/audio_detect/how_are_you_doing.wav?raw=true",
    "type": "uri"
  },
  "destination": {
    "type": "file",
    "path": "/tmp/results_audio_events.txt",
    "format": "json-lines"
  }
}'

Detection Results:

To view incremental results, execute the following command from the shell.

tail -f /tmp/results_audio_events.txt

As the audio is being analyzed and events are detected, you will see detection results in the output.

Expected Output:

{"channels":1,"events":[{"detection":{"confidence":1.0,"label":"Speech","label_id":53,"segment":{"end_timestamp":2200000000,"start_timestamp":1200000000}},"end_timestamp":2200000000,"event_type":"Speech","start_timestamp":1200000000}],"rate":16000}

After pretty-printing:

{
  "channels": 1,
  "events": [
    {
      "detection": {
        "confidence": 1,
        "label": "Speech",
        "label_id": 53,
        "segment": {
          "end_timestamp": 2200000000,
          "start_timestamp": 1200000000
        }
      },
      "end_timestamp": 2200000000,
      "event_type": "Speech",
      "start_timestamp": 1200000000
    }
  ],
  "rate": 16000
}

Solar Panel Fualt Detection Drone Video AI

https://github.com/DeepHiveMind/Solar_Panel_Fault_Detection_Video_AI

Additional Reference

Documentation	Reference Guides
- Defining Media Analytics Pipelines - Building Video Analytics Serving - Running Video Analytics Serving	- Video Analytics Serving Architecture Diagram - Microservice Endpoints - Build Script Reference - Run Script Reference

Known Issues

Default GStreamer Build Fails

If the build fails with the error ModuleNotFoundError: No module named 'skbuild' follow instructions in this github issue for a workaround.

Service Will Not Start Due to Missing Audio Plugin

GStreamer framework base images are expected to include the audio detection inference plugin libgstaudioanalytics.so. If this plugin is missing the audio detection pipeline will not load, see error message below, and the Video Analytics Serving service will not start.

{"levelname": "ERROR", "asctime": "2020-08-26 01:49:40,114", "message": "Failed to Load Pipeline from: /home/video-analytics-serving/pipelines/audio_detection/1/pipeline.json", "module": "pipeline_manager"}

Currently this plugin is only present in the DL Streamer audio preview so will be not be in any base images obtained from dockerhub. Thus GStreamer images based on Open Visual Cloud or OpenVINO^™ will exhibit this problem.

As a workaround you can configure the service to ignore initialization errors when you start it.

docker/run.sh -v /tmp:/tmp -e IGNORE_INIT_ERRORS=True

Name		Name	Last commit message	Last commit date
Latest commit History 4 Commits
Audio_detection_Video_AI/1		Audio_detection_Video_AI/1
Emotion_recognition_Video_AI/1		Emotion_recognition_Video_AI/1
Face_detection_adas__Video_AI/1		Face_detection_adas__Video_AI/1
Face_detection_retail_Video_AI/1		Face_detection_retail_Video_AI/1
Face_reidentification__Video_AI/1		Face_reidentification__Video_AI/1
Landmarks_regression_Video_AI/1		Landmarks_regression_Video_AI/1
Object_detection_Video_AI/1		Object_detection_Video_AI/1
Solar_Panel_fault_detection_Binary		Solar_Panel_fault_detection_Binary
Solar_panel_fault_detection_Multiclass		Solar_panel_fault_detection_Multiclass
.gitattributes		.gitattributes
LICENSE		LICENSE
README.md		README.md
README_Azure-Serverless-AI-Pipe.md		README_Azure-Serverless-AI-Pipe.md
README_Solar_Panel_Fault_Detection.md		README_Solar_Panel_Fault_Detection.md

License

DeepHiveMind/Video-AI-ServingPipeline_Object_Anomaly_Topic_Audio_Emotion_Detection

Folders and files

Latest commit

History

Repository files navigation

Video Analytics Serving

Features Include:

Getting Started

Prerequisites

Building the Microservice

Running the Microservice

Detecting Objects in a Video

Example Request:

Curl Command:

Detection Results:

Industrialization Samples & Code Sets

Recognizing Emotions in a Video

Example Request:

Curl Command:

Detection Results:

Detecting Audio Events in an Audio Recording

Example Request:

Curl Command:

Detection Results:

Additional Reference

Related Links

Known Issues

Default GStreamer Build Fails

Service Will Not Start Due to Missing Audio Plugin

About

Resources

License

Stars

Watchers

Forks

Languages