SpeechSurf is a tool that revolutionizes interaction with digital devices by enabling users to navigate interfaces through voice commands. This AI-powered program allows precise control of the mouse and keyboard by specifying sections of the screen to click on, streamlining tasks that traditionally require physical movement. Whether it is pausing a video, turning pages in an e-book, or scrolling through a webpage, these actions can now be executed effortlessly with simple voice commands. Users can also automate keyboard presses, allowing them to enter text into search bars and text editors with ease.
We built the landing page of this application in ReactJS + CSS, using Figma for prototyping. Our backend was built entirely in Python, primarily using the following libraries: Hugging Face Inference API + PyAudio (for speech recognition), Tkinter (to display grid overlay for clicking) PyAutoGUI (to control keyboard and mouse), and Pillow (to create and process screenshots for clicking). We bundled the app into an .exe file using PyInstaller.
This project is designed to cater to a diverse audience. While many of us share the privilege of being able-bodied, there is a significant group that does not enjoy that same privilege. SpeechSurf addresses this disparity by reimagining how people interact with technology, taking into account the needs of handicapped individuals who face accessibility challenges. In other words, we empower them with a new perspective in exploring life.
Speech Surf aims to bridge the gap between technology and inclusivity, ensuring that everyone can enjoy a seamless digital experience.
Download the tool from our website at www.speechsurf.co! (Note that the program only runs on Windows currently)