SmartGuide AI

Smartphone-based predictive AR navigation for visually impaired users.

Unity 6 + AR Foundation (ARCore) + Unity Sentis (YOLOv8n) + free OpenStreetMap routing (Nominatim / OSRM / OSM tiles) + bilingual voice control (English + Kannada).

Purpose

SmartGuide AI helps blind and low-vision users navigate indoors and outdoors by combining:

Global GPS walking routes with turn-by-turn voice guidance
Local AR obstacle perception (YOLO objects, ARCore depth, AR planes for walls/pillars/doorways)
Predictive risk (distance, time-to-collision, corridor scoring)
Hands-free voice for destination search and navigation control

No dedicated hardware beyond an ARCore-capable Android phone.

Tech Stack

Layer	Technology
Engine	Unity 6.x
AR	AR Foundation, ARCore (environment depth, plane detection)
On-device AI	Unity Sentis, YOLOv8n (COCO-80)
Maps (free)	Nominatim geocoding, OSRM foot routing, OSM raster tiles
Maps (optional)	OpenRouteService when API key is set
Speech in	Android `SpeechRecognizer` + Intent fallback
Speech out	Android TextToSpeech (TTS) with beep fallback
Platform	Android ARM64 (ARCore required)

Features

App Modes — Presentation (clean demos), Research (metrics + boxes), Developer (FPS/coords)
Voice-first navigation — say a destination; app geocodes, routes, and speaks progress
English + Kannada — auto language detection from ASR locale + Kannada script; TTS matches
Free maps — no API keys required (Nominatim + OSRM + OSM mini-map)
YOLO object detection — people, vehicles, furniture, signs (stop sign, traffic light, keyboard, etc.)
Indoor structure — walls, pillars, doorways from AR planes + depth clustering
Sequential scan — announces one obstacle at a time, then waits for user to move forward
Risk fusion — STOP / caution / steer-left / steer-right with route override on danger
Mini-map HUD — route polyline, turn arrow, deviation tint, OSM attribution
Evaluation scene — live FPS/risk/route tables + Export Evaluation Report (IEEE CSV/JSON/MD)
Accessibility — high contrast, large fonts, haptics, emergency SOS stub

Architecture (overview)

Dual-layer stack: Layer 1 global GPS + Nominatim/OSRM routing; Layer 2 local AR perception (YOLO, ARCore depth, hybrid walls/pillars) into risk → safe corridor → navigation decision → voice.

flowchart TB
    subgraph Voice
        ASR[AndroidSpeechRecognizer]
        VAM[VoiceAssistantManager]
        TTS[AudioGuidanceManager]
    end

    subgraph Global["Layer 1 — Global Navigation"]
        GPS[GpsLocationProvider]
        ORS[OpenRouteServiceClient]
        NOM[Nominatim / OSRM]
        NAV[NavigationOrchestrator]
        MAP[MapPlaceholderService]
    end

    subgraph Local["Layer 2 — Local AR Perception"]
        YOLO[YOLODetector]
        DET[DetectionManager]
        DEP[DepthManager]
        HYB[HybridObstaclePerception]
        RISK[RiskManager]
        SEQ[SequentialScanManager]
    end

    ASR --> VAM
    VAM --> NAV
    VAM --> TTS
    GPS --> NAV
    ORS --> NOM
    ORS --> NAV
    NAV --> MAP
    NAV --> TTS
    YOLO --> DET
    DET --> RISK
    DEP --> RISK
    HYB --> RISK
    RISK --> SEQ
    SEQ --> NAV
    NAV -->|priority queue| TTS

Guidance priority: Emergency STOP > steering > caution (obstacles) > route turns.

SmartGuide AI 2.0 also uses Scene Graph, Safe Corridor Planner (L/C/R), and NavigationDecisionEngine so local danger can override GPS turn hints.

Setup

Requirements

Unity 6.x with Android Build Support
ARCore-capable Android device (ARM64)
Google app installed (recommended for speech recognition)

Open in Unity

Clone this repository and open the folder in Unity 6.x
Open SplashScene (or Main Menu) and press Play in Editor (ASR/TTS are simulated)
Assign YOLOv8n Sentis ModelAsset if not auto-found (asset name yolov8n)

Android Build (debug / internal)

File → Build Settings → Android → Switch Platform
Player Settings: package com.DefaultCompany.UncertaintyAwareARNavigation (change before Play Store), ARM64 only, Min API 25+
Build APK → install with adb install -r YourApp.apk
Grant Camera, Microphone, Location on first launch

Permissions (`Assets/Plugins/Android/AndroidManifest.xml`)

Permission	Purpose
`RECORD_AUDIO`	Voice commands
`ACCESS_FINE_LOCATION`	GPS routing
`ACCESS_COARSE_LOCATION`	GPS fallback
`INTERNET`	Nominatim, OSRM, OSM tiles

Package visibility <queries> includes Google Quick Search Box and RecognitionService for Android 11+.

Model Assignment

Place Sentis ModelAsset for YOLOv8n in Assets/
Assign on DetectionManager.yoloModel in AR scene or rely on editor auto-find by name yolov8n

Home Coordinates (optional)

PlayerPrefs keys for “go home” voice intent:

ua_ar_nav.home.lat
ua_ar_nav.home.lon

No API Keys Required

Default navigation uses free services only:

Service	URL	Role
Nominatim	`nominatim.openstreetmap.org`	Destination search (GPS-biased viewbox)
OSRM	`router.project-osrm.org/route/v1/foot`	Walking route + turn steps
OSM tiles	`tile.openstreetmap.org`	Mini-map background

Fallback chain: ORS (if key in Settings) → OSRM → straight-line with spoken “approximate route” message.

Nominatim policy: max 1 request/second, identifiable User-Agent (SmartGuideAI/1.0).

Scene Flow

SplashScene → MainMenuScene → ARNavigationScene
                            → SettingsScene
                            → EvaluationScene

AppManager (DontDestroyOnLoad) wires all managers on scene load.

Voice Commands

English

Command	Action
“Take me to [place]” / “Nearest hospital”	Search + route
“Go home”	Route to saved home coords
“Stop” / “Pause”	Pause guidance
“Resume” / “Continue”	Resume
“Repeat”	Repeat last guidance
“Go left” / “Go right”	Manual steer hint

Kannada (examples)

Command	Action
“ನಾಗಸಂದ್ರಕ್ಕೆ ಹೋಗಿ”	Navigate to Nagasandra
“ಆಸ್ಪತ್ರೆ”	Nearest hospital
“ಮನೆಗೆ ಹೋಗಿ”	Go home
“ನಿಲ್ಲಿ”	Stop

Enable Voice-First Mode in Settings for automatic listening in AR.

Detection & Indoor Guidance

YOLO (COCO)

Detects 80 classes including person, car, chair, keyboard, stop sign, traffic light, bench, fire hydrant, etc.

Spoken format: "keyboard on desk, 1.2 meters ahead" via ObjectLabelHelper.

Hybrid indoor (`HybridObstaclePerception`)

Source	Labels
AR vertical planes	wall, doorway
Depth discontinuities	wall
Depth pillar clusters	pillar

Sequential scan

Observe — speak highest-risk object once
Advance — user walks ~1 m (GPS or motion)
Next — focus next unannounced object

Bounding boxes highlight the focused track; faint boxes show other confirmed objects.

Evaluation & Telemetry

Session logs: Application.persistentDataPath/UA_AR_NAV/<timestamp>/

telemetry.csv / telemetry.jsonl — FPS, inference ms, risk, TTC, nav events
telemetry.db.jsonl — SQLite-style event store (JSONL rows)
session_summary.json — session stats for research
evaluation_summary.json / .csv — live dashboard snapshot (no fabricated F1/P/R at runtime)
offline_research_report.json / .md — paper metrics when ground_truth.json present
ieee_evaluation_report.md — one-click IEEE-oriented report
track_attributes.csv — ID/class/conf/distance/velocity/TTC/risk per track
emergency_share.jsonl — SOS stub

App Modes (Settings)

Mode	Behavior
Presentation	Voice-first, clean HUD — no boxes, no debug overlays, no FPS
Research	Live metrics overlays, detection boxes, TTC/risk HUD
Developer	Research + auto FPS/GPS coords/errors debug panel

Open Evaluation from main menu for the live dashboard (System / Perception / Navigation / Voice). Use Run Offline Analysis for precision/recall/F1/mAP with ground_truth.json.

File Structure

Assets/
├── Plugins/Android/
│   ├── AndroidManifest.xml      # permissions + package queries + UnitySpeechActivity
│   ├── SpeechIntentBridge.java  # Intent speech fallback
│   └── UnitySpeechActivity.java
├── Scripts/
│   ├── AI/                      # Detection, risk, hybrid, sequential scan
│   ├── AR/                      # ARManager, guidance renderer
│   ├── Audio/                   # AudioGuidanceManager (TTS)
│   ├── Accessibility/         # Emergency, accessibility settings
│   ├── Managers/                # AppManager, settings, logging, eval
│   ├── Navigation/              # GPS, ORS/OSRM, orchestrator, map
│   ├── UI/                      # UIManager HUD
│   └── Voice/                   # Speech recognizer, intents
└── Scenes/                      # Splash, MainMenu, ARNavigation, Settings, Evaluation

Troubleshooting

Speech recognition not working (Google app installed)

Symptom	Fix
“Speech unavailable” on Android 11+	Rebuild APK — manifest includes `<queries>` for Google app + RecognitionService
`createSpeechRecognizer` returns null	App tries Google service component + Intent UI fallback (`SpeechIntentBridge`)
“Microphone permission required”	Grant mic in Android Settings → Apps → SmartGuide
“Insufficient permissions” (error 9)	Same as above; app re-requests on error
“Speech client error” (error 5)	App switches to Intent fallback; wait 2 s and tap mic again
Works in Editor but not device	Editor uses simulation only — test on physical Android

GPS / routing

Symptom	Fix
“Waiting for GPS”	Go outdoors or near window; wait up to 15 s
Wrong destination city	Nominatim uses 50 km GPS viewbox; speak more specific place name
Straight-line only	Offline or OSRM failed — check internet; message spoken explicitly
Nominatim blocked	Respect 1 req/s; User-Agent is set automatically

Detection: boxes visible but no speech

Symptom	Fix
No TTS at all	Enable Audio Guidance in Settings; check TTS status chip
Objects detected silently	Fixed: TTS dedup was 10 s minimum — now uses caller interval (~2–4 s)
Only one object spoken	Sequential scan is by design; walk forward to hear next
Keyboard not announced	Valid COCO class — lower area filter + “keyboard on desk” label added

Build errors

Issue	Fix
ARCore not supported	Use ARCore-certified device
Sentis model missing	Import/assign `yolov8n` ModelAsset
Custom activity not found	Ensure `UnitySpeechActivity.java` is in `Plugins/Android`

Development Fixes (Recent)

App Modes & evaluation (this pass)

Presentation / Research / Developer modes in Settings (ua_ar_nav.app.mode)
Evaluation dashboard wires live Detection/Risk/Nav/Voice/Perf; proxies clearly labeled with ≈
Export Evaluation Report → JSON + CSV + markdown IEEE bundle
Track attributes export (ID, class, confidence, distance, velocity, direction, stability, TTC, risk)
Voice language auto-detect uses ASR locale + Kannada Unicode; TTS replies in same language
Corridor steer speaks “Move slightly left/right”; local danger still overrides GPS route hints
Default FPS cap 30 for Android

Speech recognition

Added Android 11+ package visibility queries in manifest
Custom UnitySpeechActivity forwards onActivityResult for Intent-based recognition
SpeechIntentBridge launches Google speech UI when continuous recognizer fails
Explicit Google RecognitionService component when creating recognizer
Query installed RecognitionService providers on API 31+
Handle ERROR_CLIENT (5) and ERROR_INSUFFICIENT_PERMISSIONS (9) with retry / permission request
Cooldown before destroy/recreate recognizer (2.5 s)

Object detection → speech

Reduced TTS dedup from forced 10 s to caller minIntervalSeconds
VoiceSuppressed only during active route — local-only AR speaks directly
ObjectLabelHelper — keyboard, signs, doorway, distance phrases
Lower bbox area filter for small objects (keyboard)
Sequential scan observe delay 2.5 s → 1.5 s

Maps

Straight-line / offline route spoken clearly on route received
Nominatim GPS viewbox bias (50 km) already in OpenRouteServiceClient

Indoor

Doorway detection from vertical AR plane dimensions
Wall / pillar speech via hybrid + sequential scan pipeline

License & Attribution

YOLOv8 / COCO pretrained weights — verify license for your distribution
ARCore — Google ARCore Terms

Quick Test (Android, no API keys)

Leave ORS key blank; enable Voice-First Mode
Open AR Navigation → say “Take me to [nearby place]”
Confirm Nominatim search → OSRM route → turn TTS
Point camera at wall/desk/keyboard — hear obstacle description
Walk toward obstacle — STOP overrides route hint
Open Evaluation — verify metrics export

SmartGuide AI — assistive navigation research prototype.

Name		Name	Last commit message	Last commit date
Latest commit History 3 Commits
.utmp		.utmp
Assets		Assets
Packages		Packages
ProjectSettings		ProjectSettings
scripts		scripts
.gitignore		.gitignore
Final_Report.pdf		Final_Report.pdf
README.md		README.md

Folders and files

Latest commit

History

Repository files navigation