MCU Natural User Interface

The MCU Natural User Interface is a Unity-based application that allows users to interact with a virtual environment using sign language gestures for vowels and numbers. The interface utilizes the Leap Motion V2 device to track hand movements, providing real-time feedback through audio and visual cues.

Features

Current Features

• Real-time hand tracking: The Leap Motion V2 device accurately tracks hand movements, enabling users to input sign language gestures for vowels (A, E, I, O, U) and numbers (1-5).
• Audio and visual feedback: The interface provides immediate feedback through audio output and on-screen images, helping users understand the recognized sign and facilitating communication.

Future Features

• Expandable character set: The interface can be extended to include additional letters and numbers, allowing for a more comprehensive sign language learning experience.
• User-controlled audio output: Users can have the option to control when the interpreter speaks, providing a more personalized and interactive experience.
• Text-to-speech integration: Text-to-speech functionality can be incorporated to enhance the user experience by providing spoken instructions and feedback, as well as the ability to communicate complete sentences.
• Tutorial mode: A more detailed tutorial can be implemented to guide users on how to use the application.
• Learning mode: A learning mode can be added to the interface to help users learn sign language by providing a list of signs and their corresponding letters and numbers.

Getting Started

Prerequisites

• Unity 2021.3.23f1 or later for Development.
• Any version of Unity when building.
• Leap Motion V2 device: Connect the Leap Motion V2 device to your computer.
• Leap Motion SDK for Development: Download the Leap Motion SDK from the Leap Motion website.

Installation

1. Clone the repository or download the project files.
2. Open Unity and navigate to the project folder.
3. Open the project in Unity by selecting "Open" and choosing the project folder.
4. Ensure that the Leap Motion V2 device is properly connected to your computer.
5. Run the project in the Unity editor.

Usage

1. Launch the MCU Natural User Interface application.
2. Position your hand(s) within the field of view of the Leap Motion V2 device.
3. Perform sign language gestures for vowels or numbers, following the on-screen instructions.
4. Receive audio and visual feedback indicating the recognized sign.

Acknowledgments

• Leap Motion SDK: The Leap Motion SDK provided the foundation for hand tracking capabilities.
• Leap Motion Unity Assets: The Leap Motion Unity Assets provided the foundation for hand tracking capabilities.