Skip to content

Initial Release

Latest
Latest
Compare
Choose a tag to compare
@CommanderLee CommanderLee released this 22 Oct 03:32
· 6 commits to master since this release
v0.3
8c5cafd

EyelidGesturesDetection

How to install the app?

Download and install the APK in the Assets below (EyelidInteraction.apk) to your Android phone.
Hint: You could open this URL in the Browser (e.g., Chrome) on your Android phone, scroll down to the bottom of this page to locate the EyeInteraction.apk in the Assets section below, and download and install it.

Our app currently works on Android smartphones with version 8.0 or above. We plan to make it available on iOS in the future.

We have also open-sourced our code on Github to allow developers to build on top of our work: https://github.com/mingming-fan/EyelidGesturesDetection.

Demo video

You can watch a demo video to get a sense of how well our algorithm works in real-time and its applications using the following link: https://www.youtube.com/watch?v=GgpW4tmvdM0&feature=youtu.be

How to use the app?

  1. For the first-time user, the app needs to collect some samples to train the classifier. Complete the first three steps shown in the app's landing user interface (UI): "Step 1: Eye Detection Checking", "Step 2: Eyelid State Training", "Step 3: Eyelid Gesture Training".
    *You only need to train the model once.

  2. Click "Step 1: Eye Detection Checking" to check whether the app detects your eyes or not. You would see the real-time detection result overlay on the front-facing camera stream. Try to close an eyelid and see whether it detects the number of open eyes correctly or not.

  • Troubleshoot: if you cannot see the camera feed, you need to grant the corresponding permissions. In this case, the permission to use the Camera is needed (see AndroidManifest.xml file for details).
  1. Click "Step 2: Eyelid State Training" to train the model to detect eyelid states (i.e., both eyelids open, both eyelids close, open right eyelid only, and open left eyelid only). On the UI, you can choose a participant ID (PID). You can simply proceed with the default one.
  • There will be a short 'beep' after each data-collection block, so you will know even if you have closed both eyes (i.e., the 'both eyelid close' state).
  1. Click "Step 3: Eyelid Gesture Training" to train the model to detect nine eyelid gestures. By default, the model only needs to collect '5' samples per eyelid gesture. However, more samples per gesture would likely increase the recognition accuracy. Make sure you use the same PID as the previous step.
  • If the face detection status shows "Face detection failed", or when the eyelid state detection result keeps fluctuating, you can try to adjust the camera angle or your ambient light condition.
  • For consistency, you can try to count for certain numbers consistently for short and long gestures, respectively. E.g., you can try to count number '1' while performing R/L/B gestures (short gestures), and count numbers '1,2,3' while performing R-/L-/B- gestures (long gestures with one eyelid state), and count '1,2,3; 4,5,6' while performing B-L-/B-R- gestures (long gestures with two eyelid states).

After completing the first three steps, the model will have been created and saved. You can then test the model's performance by clicking on "Eyelid Gesture Testing."

  1. Click "Eyelid Gesture Testing." Choose the same PID for which you trained your models in the previous steps and Click "Confirm." On the next screen, click on the "Start" Button. Then, you can perform any of the nine eyelid gestures and see the real-time recognition result. The UI also plots the probability of each eyelid being open in real-time.

We have also created two applications for you to interact with your phone using your eyelid gestures. One is "Within and Between Apps Navigation," and the other one is "Cross-App Text Entry App"

  1. "Within and Between Apps Navigation": navigate within and between apps using eyelid gestures. According to our gesture matching scheme, you can navigate between apps (B-R-, B-L-), tabs/screens witin an app (R-, L-), and containers within a tab/screen (R, L) only using eyelid gestures.

  2. "Cross-App Text Entry App": One simulated app shows a target phrase, and your goal is to type the phrase into another simulated app. You can use "normal" switch method by pressing the switch app button at the bottom of the screen. You can also use "eyelid_gesture" to switch apps by closing any one of your eyelids.

How to access the source code?

We have open-sourced our code on Github to allow others to build on top of our work: https://github.com/mingming-fan/EyelidGesturesDetection.

Thank you so much for your interest! Happy coding and make the world a better place.

Assets 3