Snooze You Loose

Snooze You Loose is a web-based tool to be used on online EdTech platforms for detecting when a student is drowsy. Upon drowsiness detection, the platform raises alerts to increase student attentiveness and promote e-education.

Problem

Last year, saw the physical classrooms converting to virtual rooms, thanks to Covid. While the classrooms have altered, students are a bit behind adapting to this change. With increasing 'Zoom Fatigue', drowsiness while attending online classes is a concern.

The Snooze You Loose tool helps alleviate the issue by detecting if a student is getting sleepy and provides alerts accordingly. The tool gathers data over the EdTech platform courses and provides analytics to the content creators so that they can then improve their courses.

Methodology

Using Artificial Intelligence, the Snooze You Looze tool analyzes the student video and detects whether the student is drowsy or alert without compromising user privacy.

The tool uses Google's MediaPipe model for detecting a face in the video stream. As this model runs on the browser (on the user's side), no video frame is ever sent to the server, protecting user privacy.

Fig 1: FaceMesh MediaPipe Output

Features based on facial landmark positions are calculated and are then sent over to the server. At the server, Deep Learning or Machine Learning models are used to provide classification scores for the facial features.

Fig 2: Alert, Drowsy Prediction

At present, three different models are provided which can be changed at run-time to see the functioning.

To understand the training process, please training code readme file.

Fig 3: Inference

WebApp Quick Start

1. Initialize and activate a virtualenv:

$ virtualenv --no-site-packages env
$ source env/bin/activate

2. Install the dependencies:

$ pip install -r requirements.txt

3. Run the development server:

$ python app.py

4. Navigate to http://localhost:5000

Fig 3: Webapp Prediction

Webapp Demo:

Fig 4: App Navigation

Fig 5: Calibration

Fig 6: Model 1 (Manual Thresholding) Inference

Fig 7: Model 3 (LSTM) Inference

Future Aspects:

Detecting drowsiness and providing alerts to the user is helpful. We believe that engaging the student with small games when detected drowsy state can help better engage the student and reduce the Zoom Fatigue.

We would like to extend the tool and develop it as an API, which can be then integrated into online EdTech platforms. This can enable the analytics feature over the courses and help improve the contents, benefitting both the student and the platforms.

Additionally, the dataset collected for this task can be further expanded which can help train models achieving higher performance.

Acknowledgements:

This project is developed as a part of HackerEarth Hack 2021. We thank the sponsors for providing us with this opportunity to build an impactful application.

Contributors:

• Samradh Agarwal
• Abhishek Tandon