HippoYD: Yawn Detector using mouth

Overview

This is a mouth openness detection model. Full training code, data preparation scripts, and pretrained models are in the repository.

Open image inference in Colab.

Demo

Requirements

• Refer to requirements.txt for dependent libraries that're needed to use the code.

Image Dataset

The model was trained using two sources:

1. Main source: Video dataset YawDD
   Paper: S. Abtahi, M. Omidyeganeh, S. Shirmohammadi, and B. Hariri, “YawDD: A Yawning Detection Dataset”, Proc. ACM Multimedia Systems, Singapore, March 19 -21 2014.
2. Augmentered by Kaggle Drowsiness_dataset.

How to train

1. Convert YawDD dataset to image folders, 2 classes: closed and opened

python convert_dataset_video_to_mouth_img.py

2. Split data into 3 datasets: train, validation, test

python split_data_into_datasets.py

3. Train data with:

python train_yawn.py

Model evaluation

Dataset	AC	PR	RC	F1
Kaggle	0.932	0.992	0.871	0.927
Test split (5%)	0.988	0.960	0.975	0.967

*AC - accuracy, PR - precision, RC - recall.

Available pretrained models [full, lite] and demos

Model	Example of inference	Demo
SavedModel / Keras H5	run_yawn_inference_tf_h5.py
TFLite	run_yawn_inference_tflite.py
TensorFlowJS	image_predict.js	https://igla.su/mouth-open-js/
ONNX	run_yawn_inference_onnx_cv.py run_yawn_inference_onnx_onnxruntime.py	https://igla.su/mouth-open-js/
Frozen pb	run_yawn_inference_tf_pb.py

Model Inference Results

Configuration	Config	Model	Time (avg)	TFLite ver.
Macbook Pro, CPU 2 GHz Quad-Core Intel Core i5	CPU	TFLite (Floating)	5 ms	2.3
	CPU	TFLite (Quantized)	8 ms	2.3
	CPU	Keras H5 (Floating)	30 ms	2.3
	CPU	ONNX	2 ms	2.3
	CPU	Frozen pb	4 ms	2.3
	Wasm (Safari 14.0, Firefox 84)	TensorFlowJS	30 ms	2.3
Xiaomi MI8	GPU/CPU 3 Threads	TFLite (Floating)	4 ms	2.4
	CPU 3 Threads	TFLite (Quantized)	10 ms	2.4
Xiaomi Redmi 9	GPU/CPU 3 Threads	TFLite (Floating)	11 ms	2.4
	CPU 3 Threads	TFLite (Quantized)	9 ms	2.4
Samsung Galaxy S20+	GPU/CPU 3 Threads	TFLite (Floating)	2 ms	2.4
	CPU 3 Threads	TFLite (Quantized)	2 ms	2.4

Issues

If you find any problems or would like to suggest a feature, please feel free to file an issue

License

Copyright 2021 Igor Lashkov

Licensed under the Apache License, Version 2.0 (the "License");
you may not use this file except in compliance with the License.
You may obtain a copy of the License at

   http://www.apache.org/licenses/LICENSE-2.0

Unless required by applicable law or agreed to in writing, software
distributed under the License is distributed on an "AS IS" BASIS,
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
See the License for the specific language governing permissions and
limitations under the License.