Udacity Computer Pointer Controller

Objective of project: To control the a computer mouse pointer using gaze detection model aided by other models in a pipeline.

Description of project: The Gaze Estimation model is used to estimate the gaze of the user's eyes and it is applied to pyautogui module to change the mouse pointer position accordingly. This project demonstrates the ability to run multiple models in the same machine and coordinate the flow of data between those models.

Project Set Up and Installation

Project source structure is as shown below.

pointer-controller
├── README.md
├── bin
│   └── demo.mp4
├── intel
│   ├── face-detection-adas-binary-0001
│   │   └── FP32-INT1
│   │       ├── face-detection-adas-binary-0001.bin
│   │       └── face-detection-adas-binary-0001.xml
│   ├── gaze-estimation-adas-0002
│   │   ├── FP16
│   │   │   ├── gaze-estimation-adas-0002.bin
│   │   │   └── gaze-estimation-adas-0002.xml
│   │   ├── FP16-INT8
│   │   │   ├── gaze-estimation-adas-0002.bin
│   │   │   └── gaze-estimation-adas-0002.xml
│   │   └── FP32
│   │       ├── gaze-estimation-adas-0002.bin
│   │       └── gaze-estimation-adas-0002.xml
│   ├── head-pose-estimation-adas-0001
│   │   ├── FP16
│   │   │   ├── head-pose-estimation-adas-0001.bin
│   │   │   └── head-pose-estimation-adas-0001.xml
│   │   ├── FP16-INT8
│   │   │   ├── head-pose-estimation-adas-0001.bin
│   │   │   └── head-pose-estimation-adas-0001.xml
│   │   └── FP32
│   │       ├── head-pose-estimation-adas-0001.bin
│   │       └── head-pose-estimation-adas-0001.xml
│   └── landmarks-regression-retail-0009
│       ├── FP16
│       │   ├── landmarks-regression-retail-0009.bin
│       │   └── landmarks-regression-retail-0009.xml
│       ├── FP16-INT8
│       │   ├── landmarks-regression-retail-0009.bin
│       │   └── landmarks-regression-retail-0009.xml
│       └── FP32
│           ├── landmarks-regression-retail-0009.bin
│           └── landmarks-regression-retail-0009.xml
├── requirements.txt
├── res
│   └── pipeline.png
└── src
    ├── face_detection.py
    ├── facial_landmarks_detection.py
    ├── gaze_estimation.py
    ├── head_pose_estimation.py
    ├── input_feeder.py
    ├── main.py
    ├── model.py
    └── mouse_controller.py

Python dependencies

This project depends on openvino,numpy,pyautogui and opencv openvino-contains pr-trained libraries for deep learning based out of the box computer vision models Pyautogui-module with functions to help coordinate the movement of our mouse pointer numpy-module for numerical computations

steps to run the demo

first install the above packages with command below in your cli

$ pip install -r requirements.txt

Directions:

navigate to the src directory and run the python script

$ cd src/
$ python main.py

other Cli options:

• cam: Use camera as input
• out: Path to output image or video file (defatult: 'out.mp4')
• video: Path to input image or video file (default: '../bin/demo.mp4')
• threshold: Probability threshold for detections filtering
• verbose_stage: Enable to show visualization of intermediate stages
• device: option to specify the target device - CPU/GPU/MYRIAD/FPGA (default: CPU)
• extension: Path to shared library extension for custom layers implementation
• precision: Model precision for applicable models - FP16/FP16-INT8/FP32 (default: FP32)
• gaze_model, head_pose_model, face_model, annd landmarks_model: options to specify the corresponding model path if the models are placed elsewhere than the project directory.

NB:

• face-detection-adas-binary-0001 model has only one (FP32-INT1) precision.

Project Documentation

Pipeline: This mouse pointer project makes use of four pre-trained models provided by OpenVINO toolkit. The data flow between them is shown in the figure below.

• 1-Face Detection
• 2-Head Position Estimation
• 3-Facial Landmark Detection
• 4-Gaze Estimation Model

Benchmarksof the 4 pretrained models

Model	Precision	Load Time	Inference Time	Pre-process Time
face-detection-adas-binary-0001	FP32-INT1	211.75 msecs	9.99 msecs	0.55 msecs
gaze-estimation-adas-0002	FP16	138.08 msecs	1.52 msecs	0.05 msecs
	FP16-INT8	239.18 msecs	1.41 msecs
	FP32	122.49 msecs	1.82 msecs
head-pose-estimation-adas-0001	FP16	129.37 msecs	1.24 msecs	0.06 msecs
	FP16-INT8	207.70 msecs	1.18 msecs
	FP32	96.98 msecs	1.45 msecs
landmarks-regression-retail-0009	FP16	108.77 msecs	0.51 msecs	0.05 msecs
	FP16-INT8	132.88 msecs	0.45 msecs
	FP32	105.91 msecs	0.51 msecs
##Source openvino,Intel website

Results

It can be observed from the above benchmark table that there is difference in inference time for different precisions l models.

Reason is :

a-quantization technic used for optimization

effects

reduced accuracy but faster inference. ##conclusion Therefore the FP32 takes higher inference time but more accuracy than INT8. So Accuracy & Inference Time is in the order FP32 > FP16 > INT8. But the model size is inversely proportional to the above parameters. Hence lower the precision, lighter is the model. Which might make the storing cost lesser and loading time faster during the startup.