Project-Cortex v2.0

AI-Powered Assistive Wearable for the Visually Impaired

---

🎯 Project Mission

Project-Cortex is a low-cost (<$150), high-impact AI wearable designed to assist visually impaired individuals by providing real-time scene understanding, object detection, and audio navigation. Built for the Young Innovators Awards (YIA) 2026 competition.

We aim to democratize assistive technology by disrupting the $4,000+ premium device market (OrCam, eSight) using commodity hardware and a novel "Hybrid AI" architecture.

---

🏗️ Architecture Overview

Hardware Platform

• Compute: Raspberry Pi 5 (4GB/8GB RAM)
• Vision: IMX415 8MP Low-Light Camera (MIPI CSI-2)
• Power: 30,000mAh USB-C PD Power Bank
• Cooling: Official RPi 5 Active Cooler
• Audio: USB Audio Interface + Bone Conduction Headphones
• Connectivity: Mobile Hotspot (no dedicated SIM module)

The "3-Layer AI" Brain

Layer 1: The Reflex (Local Inference)

• Purpose: Instant safety-critical object detection
• Model: YOLOv8n / TensorFlow Lite
• Latency: <100ms
• Power: 8-12W during inference
• Location: src/layer1_reflex/

Layer 2: The Thinker (Cloud Intelligence)

• Purpose: Complex scene analysis, OCR, natural language descriptions
• Model: Google Gemini 1.5 Flash (via API)
• Fallback: OpenAI GPT-4 Vision
• Latency: ~1-3s (network dependent)
• Location: src/layer2_thinker/

Layer 3: The Guide (Integration & UX)

• Features: GPS navigation, 3D spatial audio, caregiver dashboard
• Tech Stack: FastAPI (backend), React (dashboard), PyOpenAL (audio)
• Location: src/layer3_guide/

---

📁 Repository Structure

ProjectCortex/
├── Version_1/                      # Archived ESP32-CAM implementation
│   ├── Docs/                      # v1.0 technical retrospective
│   └── Code/                      # v1.0 Python/Arduino code
├── models/                         # Shared AI models (YOLO variants)
├── TTS Model/                      # Piper TTS model files
├── src/                           # Version 2.0 source code
│   ├── layer1_reflex/             # Local object detection module
│   ├── layer2_thinker/            # Cloud AI integration module
│   ├── layer3_guide/              # Navigation & UI module
│   └── main.py                    # Application entry point
├── config/                         # Configuration files (.yaml, .json)
├── tests/                          # Unit and integration tests
├── docs/                           # Technical documentation
├── utils/                          # Helper scripts and tools
├── .env.example                    # Environment variables template
├── requirements.txt                # Python dependencies
└── README.md                       # This file

---

🚀 Quick Start

Prerequisites

• Raspberry Pi 5 (4GB+ RAM) with Raspberry Pi OS (64-bit)
• IMX415 Camera Module (connected via CSI port)
• Python 3.11+
• Active internet connection (for Layer 2)

Installation

1. Clone the repository:

   git clone https://github.com/IRSPlays/ProjectCortex.git
   cd ProjectCortex

2. Set up Python environment:

   python3.11 -m venv venv
   source venv/bin/activate
   pip install --upgrade pip
   pip install -r requirements.txt

3. Configure environment variables:

   cp .env.example .env
   nano .env  # Add your API keys (Gemini, Murf AI, etc.)

4. Test camera module:

   libcamera-hello --camera 0  # Should display camera preview

5. Run the application:

   python src/main.py

---

🔧 Configuration

Power Management

Add to /boot/firmware/config.txt:

usb_max_current_enable=1
dtoverlay=imx415

Camera Settings

Configure in config/camera.yaml:

resolution: [1920, 1080]
framerate: 30
format: RGB888

AI Model Selection

Edit config/models.yaml:

layer1:
  model: "models/yolo11s.pt"
  device: "cpu"  # Change to "cuda" if using Coral TPU
  confidence: 0.5

---

🔊 3D Spatial Audio Navigation System

Project-Cortex features a binaural 3D spatial audio system that helps visually impaired users navigate their environment using audio cues. This system converts YOLO object detections into positioned audio sources, creating an "audio map" of the surroundings.

Features

Feature	Description
Audio Beacons	Continuous directional sounds that guide users to targets (e.g., "lead me to the door")
Proximity Alerts	Distance-based warnings that intensify as obstacles approach
Object Tracking	Real-time 3D audio sources for each detected object
Distance Estimation	Calculate real-world distance using known object sizes
Object-Specific Sounds	Distinct audio signatures for different object classes (car vs person vs chair)
HRTF Support	Head-Related Transfer Function for realistic binaural audio on headphones

How It Works

YOLO Detection → Position Calculator → OpenAL 3D Audio → Headphones
     │                   │                    │
     ▼                   ▼                    ▼
  [bbox]    →    [x, y, z coords]    →   [Binaural audio]

Position Mapping Algorithm:

• X-axis (Left/Right): Bbox horizontal center → audio pan
• Y-axis (Up/Down): Bbox vertical center → audio elevation
• Z-axis (Distance): Bbox size → audio volume/distance

Quick Start

from src.layer3_guide.spatial_audio import SpatialAudioManager, Detection

# Initialize spatial audio
audio = SpatialAudioManager()
audio.start()

# Update with YOLO detections
detections = [
    Detection("chair_1", "chair", 0.92, (100, 200, 300, 600)),
    Detection("person_1", "person", 0.87, (1400, 100, 1800, 900)),
]
audio.update_detections(detections)

# Start navigation beacon to guide user
audio.start_beacon("chair")  # "Follow the sound to the chair"

# Stop when done
audio.stop()

Configuration

Edit config/spatial_audio.yaml to customize:

• Distance thresholds for proximity alerts
• Object-specific sound mappings
• Ping rates and volumes for beacons
• Known object sizes for distance estimation

Components

Module	File	Purpose
SpatialAudioManager	manager.py	Central orchestrator for all spatial audio
PositionCalculator	position_calculator.py	YOLO bbox → 3D coordinates
AudioBeacon	audio_beacon.py	Navigation guidance pings
ProximityAlertSystem	proximity_alert.py	Distance-based warnings
ObjectSoundMapper	object_sounds.py	Object class → sound mapping
ObjectTracker	object_tracker.py	Multi-object audio management

Requirements

pip install PyOpenAL numpy PyYAML

Linux/Raspberry Pi:

sudo apt-get install libopenal-dev libopenal1

📖 Full documentation: docs/SPATIAL_AUDIO_IMPLEMENTATION.md

---

🧪 Testing

Run unit tests:

pytest tests/ -v

Run integration tests (requires hardware):

pytest tests/integration/ --hardware

---

📊 Performance Benchmarks

Metric	Target	Current Status
Layer 1 Latency	<100ms	TBD
Layer 2 Latency	<3s	TBD
Power Consumption	<20W avg	TBD
Battery Life	6-8 hours	TBD
Object Detection Accuracy	>85% mAP	TBD

---

🛠️ Development Roadmap

Phase 1: Core Infrastructure (Current)

• Repository restructure
• Camera integration with libcamera
• Layer 1 YOLO inference pipeline
• Layer 2 Gemini API integration
• Audio subsystem (TTS + STT)

Phase 2: Feature Development

• GPS navigation module
• 3D spatial audio engine ✅ IMPLEMENTED
• Caregiver web dashboard
• Power optimization

Phase 3: YIA Preparation

• User testing & feedback
• Documentation for judges
• Prototype enclosure design
• Demonstration video

---

📚 Documentation

• Bill of Materials (BOM) - Complete parts list with costs
• Architecture Deep Dive - Technical design decisions
• API Reference - Code documentation
• v1.0 Retrospective - Lessons learned from ESP32-CAM

---

🤝 Contributing

This is a competition prototype developed by Haziq (@IRSPlays). For questions or collaboration inquiries, please open an issue.

---

📄 License

This project is licensed under the MIT License - see the LICENSE file for details.

---

🏆 Acknowledgments

• YIA 2026 Organizers - For the opportunity to innovate
• Raspberry Pi Foundation - For affordable, powerful compute
• Ultralytics - For accessible YOLO implementations
• Google Gemini Team - For multimodal AI API access

---

📞 Contact

Project Lead: Haziq
GitHub: @IRSPlays
Repository: ProjectCortex

---

Built with 💙 for accessibility. Engineered with 🔥 for excellence.