🎙️ Voice-Based Memory Assistance System for Visually Impaired People

An AI-powered voice assistant that helps visually impaired users remember where they placed their belongings — entirely through natural speech, with no screen interaction required.

---

📌 Problem Statement

Visually impaired individuals often face difficulties in remembering and recalling the locations of everyday objects in their surroundings. Many digital tools today rely heavily on visual interfaces such as screens, buttons, and menus, which makes them difficult or impossible to use independently for people with severe visual impairments.

A common challenge for visually impaired users is remembering where personal items are placed. Objects like keys, wallets, glasses, medicines, or other frequently used items can easily be misplaced. Without visual cues, finding these objects again can become time-consuming and frustrating, often requiring assistance from others.

Some of the key difficulties include:

• Difficulty remembering object locations – Without visual feedback, recalling where an item was placed becomes challenging.
• Dependence on others – Misplaced objects may require help from family members or caregivers to locate them.
• Inaccessible traditional applications – Most existing applications require users to read or navigate visual screens.
• Lack of specialized tools – While general voice assistants exist, they are not specifically designed to store and retrieve personal object-location memories in a simple and focused way.

This project addresses these challenges by providing a voice-based memory assistance system that allows visually impaired users to store and retrieve the locations of objects using voice commands. Users can verbally record where an item is placed and later ask the system to recall that information. By eliminating the need for screens or manual input, the system provides a more accessible and independent way for visually impaired individuals to manage and recall object locations.

---

🧠 Project Overview

The Voice-Based Memory Assistance System is an AI-powered application that allows visually impaired users to:

• Store memories by speaking naturally (e.g., "I kept my wallet in the drawer").
• Retrieve memories by asking questions (e.g., "Where are my keys?").
• Receive spoken responses — the system always replies with synthesized voice audio, keeping the experience entirely hands-free.

The system consists of:

• A Flutter Web frontend that records the user's voice and plays back the AI's spoken response.
• A Python/FastAPI backend that transcribes the audio, runs an AI agent to understand the intent, manages a persistent memory store, and generates a text-to-speech voice response.

---

Voice-Based Memory Assistance System Architecture

Backend Module Overview

File	Responsibility
main.py	FastAPI app, API routes (/voice, /memories, /)
server.py	Core logic: STT, LLM agent, memory CRUD, TTS
memory.json	Persistent flat-file memory store
requirements.txt	Python dependencies
.env	API keys (not committed to version control)

---

⚙️ Backend Workflow

The backend processes every voice request through the following steps:

1. Audio Upload — The Flutter frontend records the user's voice and sends the audio file to POST /voice.
2. Speech-to-Text (STT) — The audio is passed to Groq Whisper (whisper-large-v3-turbo), which transcribes it into text.
3. AI Agent Processing — The transcribed text is passed to the LLaMA 3.3 70B model running as a tool-calling agent. The agent decides whether to:
   • Call extract_entities → parse objects and their locations from the sentence.
   • Call save_memory → store or update an object's location in memory.json.
   • Call retrieve_memory → look up where an object is stored.
4. Memory Operation — The corresponding memory function reads from or writes to the local memory.json file, which persists across sessions.
5. SSML Response — The agent generates a natural, friendly response wrapped in SSML markup for expressive speech output.
6. Text-to-Speech (TTS) — The SSML response is sent to Navigate Labs AI (gemini-2.5-flash-tts), which converts it to a .wav audio file served as a static asset.
7. Response Return — The API returns a JSON payload to the Flutter frontend:

   {
     "transcription": "Where are my keys?",
     "response_text": "Your keys are on the kitchen counter.",
     "response_audio_url": "http://localhost:8000/audio/abc123.wav"
   }

8. Audio Playback — The Flutter app receives the URL and plays the audio response back to the user using just_audio.

---

📸 Output Screenshots

Home / Main Interface

Voice Assistance Interface

Voice Command in Action

Memory Stored Output

Memory Retrieval Output

Memory Database

---

✅ Prerequisites

Before running the project, ensure you have the following installed and available:

Backend

• Python 3.10 or higher
• pip (Python package manager)
• A working microphone (for voice input via the browser)
• Internet connection (required for Groq API and Navigate Labs TTS)
• A valid Groq API Key — Get one here
• A valid Navigate Labs API Key — Get one here

Frontend

• Flutter SDK >=3.4.0 — Install Flutter
• A modern web browser (Chrome recommended for microphone access)

---

🔧 Installation

1. Clone the Repository

git clone https://github.com/kishoreabc/Voice-Memory-Assistant.git
cd Voice_Memory_Assistant

2. Set Up the Backend

cd backend
python -m venv venv

# Windows
venv\Scripts\activate

# macOS / Linux
source venv/bin/activate

pip install -r requirements.txt

3. Configure Environment Variables

Create a .env file inside the backend/ directory:

GROQ_API_KEY=your_groq_api_key_here
NAVIGATE_LABS_API_KEY=your_navigate_labs_api_key_here
NAVIGATE_LABS_BASE_URL=your_navigate_labs_base_url_here

⚠️ Never commit your .env file to version control. It is listed in .gitignore.

4. Set Up the Frontend

cd ../frontend
flutter pub get

Create a .env file inside the frontend/ directory:

BACKEND_URL=http://localhost:8000

---

🚀 How to Run the Project

Step 1 — Start the Backend Server

cd backend
# Activate virtual environment first if not already active
uvicorn main:app --reload

The API will be live at: http://localhost:8000

You can verify it is running by visiting: http://localhost:8000
Expected response: {"status": "Voice Memory Assistant API is running"}

Step 2 — Start the Flutter Frontend

cd frontend
flutter run -d chrome

The Flutter web app will open in Chrome. Allow microphone access when prompted.

Step 3 — Interact with the System

1. Press the microphone button on the screen.
2. Speak your command naturally.
3. Release the button — the audio is sent to the backend automatically.
4. Wait a moment while the system transcribes, processes, and generates a response.
5. The voice response will play back through your speakers/headphones.
6. The transcription and response text are displayed on screen as well.

---

🗣️ Example Voice Commands

Storing a Memory

What You Say	What Happens
"I kept my wallet in the drawer."	Saves: wallet → drawer
"My glasses are on the bedside table."	Saves: glasses → bedside table
"I put my watch on the desk and my shoes near the door."	Saves both objects and locations simultaneously
"My medication is in the bathroom cabinet."	Saves: medication → bathroom cabinet

Retrieving a Memory

What You Say	What the System Responds
"Where is my wallet?"	"Your wallet is in the drawer."
"Where are my keys?"	"I don't have a record of that yet. Could you tell me where you keep them?"
"Where did I put my glasses?"	"Your glasses are on the bedside table."
"Where is my laptop?"	"I don't have a record of that yet. Could you tell me where you keep it?"

---

📂 Project Structure

Voice_Memory_Assistant/
├── backend/
│   ├── main.py              # FastAPI routes (/voice, /memories)
│   ├── server.py            # STT, AI agent, memory CRUD, TTS logic
│   ├── memory.json          # Persistent object-location memory store
│   ├── memories.db          # SQLite database (supplementary storage)
│   ├── requirements.txt     # Python dependencies
│   ├── .env                 # API keys (not committed)
│   └── audio_responses/     # Generated TTS audio files
│
├── frontend/
│   ├── lib/
│   │   ├── main.dart        # App entry point
│   │   ├── screens/         # UI screens
│   │   ├── services/        # API communication layer
│   │   ├── providers/       # State management
│   │   ├── models/          # Data models
│   │   └── widgets/         # Reusable UI widgets
│   ├── pubspec.yaml         # Flutter dependencies
│   └── .env                 # Frontend environment config
│
└── README.md

---

🔌 API Reference

Endpoint	Method	Description
/	GET	Health check — confirms server is running
/voice	POST	Upload audio file → returns transcription, response text, and TTS audio URL
/memories	GET	Returns all stored object–location memories as JSON
/audio/{filename}	GET	Serves generated TTS audio files

---

🧰 Tech Stack Summary

Component	Library / Service
Backend Framework	FastAPI + Uvicorn
Speech Recognition	Groq Whisper (whisper-large-v3-turbo)
LLM Agent	Groq LLaMA 3.3 70B (llama-3.3-70b-versatile)
Text-to-Speech	Navigate Labs AI (gpt-4o-mini-tts)
Frontend	Flutter (Dart) Web
Audio Recording	record Flutter package
Audio Playback	just_audio Flutter package
HTTP Client	http Flutter package
State Management	provider Flutter package

---

🤝 Contributing

Contributions are welcome! To contribute:

1. Fork the repository.
2. Create a new feature branch: git checkout -b feature/your-feature-name
3. Make your changes and commit: git commit -m "Add: your feature description"
4. Push to your branch: git push origin feature/your-feature-name
5. Open a Pull Request.

---

This project is developed as a Capstone Project for academic purposes.

---

Voice Model Capstone Project
Built with ❤️ for accessibility and inclusion.

---

"Technology should serve everyone — including those who cannot see the screen."