Home

Open Accessibility Unreal Plugin

Overview

This is an Unreal Engine Editor Plugin that provides Multi-Modal Interaction for Developers. With the plugin providing users interaction on common unreal aspects through:

• Voice Commands (Push-To-Talk)
• Eye Tracking

Requirements

• Python Editor Script Plugin (More information HERE)

GPU

For CUDA supported GPUs the following NVIDIA libraries are required:

• cuBLAS for CUDA 12
• cuDNN 8 for CUDA 12

Further information around the GPU requirements can be found in the Faster-Whisper Repository.

Example

With an Open Blueprint Editor, and using the Push-To-Talk Function. Command: "NODE 4 MOVE LEFT 200"

Before:

Pre-Movement Graph Editor Image

After:

Post-Movement Graph Editor Image

Plugin Runtime

The Plugin has Two Separate Runtime Components:

• C++ (UI and Audio Gathering)
• Python (Transcription Generation)

Here is a flowchart outlining each runtime component:

Next Steps

First, read the Installation page for plugin installation. Then, follow the instructions provided in the Getting Started page.