SoulBot is a real-time, AI-powered mental wellness platform featuring expressive 3D avatars that listen, speak, and emotionally respond — turning conversations into clinically meaningful insights.

---

📖 Table of Contents

• ✨ Overview
• 🏗️ System Architecture
• 🔄 Request Lifecycle
• 📡 Real-Time Flow Sequence
• 🤖 Dual LLM Pipeline
• 📊 Mental Health Scoring
• 🧰 Tech Stack
• 🚀 Getting Started
• 🔌 Socket Events
• 📁 Project Structure
• 🌿 Environment Variables

---

✨ Overview

SoulBot creates immersive, emotionally aware AI therapy sessions through:

• 🎙️ Real-time Speech-to-Text via Sarvam AI Streaming API
• 🧠 Dual LLM Processing — one for response generation, one for avatar emotion
• 🗣️ Text-to-Speech streaming with avatar-specific voice profiles
• 😊 Dynamic Avatar Expressions driven by a dedicated emotion LLM
• 📋 Automated Clinical Reports scoring PHQ-9 (depression) and GAD-7 (anxiety)

---

🏗️ System Architecture

---

🔄 Request Lifecycle

flowchart LR
    A([🎤 User Speaks]) --> B[Audio Chunks\nvia WebSocket]
    B --> C{STT\nSarvam AI}
    C --> D[Transcript\nText]
    D --> E{LangGraph\nOrchestrator}

    E --> F["🧠 LLM 1\nResponse Generator"]
    E --> G["😊 LLM 2\nExpression Engine"]

    F --> H[Response\nTranscript]
    G --> I[Expression\nJSON]

    H --> J{TTS\nSarvam AI}
    J --> K[Audio Stream\nby Avatar Voice]

    K --> L([🔊 Avatar Speaks])
    I --> M([😊 Avatar Reacts])

    E --> N[(SQLite\nSession Log)]
    N --> O[📋 PHQ-9 / GAD-7\nReport]
    O --> P([📊 Dashboard])

    style A fill:#e94560,color:#fff,stroke:none
    style L fill:#533483,color:#fff,stroke:none
    style M fill:#533483,color:#fff,stroke:none
    style P fill:#0f3460,color:#fff,stroke:none
    style E fill:#1a1a2e,color:#eee,stroke:#e94560

Loading

---

📡 Real-Time Flow Sequence

sequenceDiagram
    autonumber
    actor User
    participant FE as 🖥️ Frontend
    participant WS as ⚡ Socket.io
    participant BE as 🔧 NestJS Backend
    participant STT as 🎙️ Sarvam STT
    participant LG as 🧠 LangGraph
    participant L1 as 💬 LLM · Response
    participant L2 as 😊 LLM · Emotion
    participant TTS as 🗣️ Sarvam TTS
    participant DB as 🗄️ SQLite

    User->>FE: Speaks into microphone
    FE->>WS: emit(audio:chunk, audioBuffer)

    loop Streaming STT
        WS->>BE: audio chunk received
        BE->>STT: stream audio buffer
        STT-->>BE: partial transcript
    end

    STT-->>BE: final transcript ✅
    BE->>DB: save user message to session

    BE->>LG: invoke graph with transcript + history

    par Parallel LLM Inference
        LG->>L1: generate therapeutic response
        L1-->>LG: response transcript
    and
        LG->>L2: analyze emotional context
        L2-->>LG: expression { emotion, intensity }
    end

    LG-->>BE: { transcript, expression }
    BE->>DB: save assistant response

    BE->>WS: emit(avatar:expression, { emotion, intensity })
    WS->>FE: avatar plays expression animation 😊

    BE->>TTS: stream transcript with avatar voice profile

    loop Streaming TTS
        TTS-->>BE: audio chunk
        BE->>WS: emit(audio:response, audioChunk)
        WS->>FE: play audio chunk in realtime
    end

    FE->>User: Avatar speaks with matching expression 🎭

    Note over BE,DB: On session end
    BE->>LG: generate clinical report
    LG->>L1: summarize + score PHQ-9 / GAD-7
    L1-->>LG: scores + insights
    LG-->>BE: final report JSON
    BE->>DB: persist report
    BE->>WS: emit(session:report, reportData)
    WS->>FE: display dashboard 📊

Loading

📊 Mental Health Scoring

At the end of every session, SoulBot generates a comprehensive clinical report:

graph TD
    LOG[📝 Full Conversation Log] --> ANALYZER[LangGraph\nReport Analyzer]

    ANALYZER --> PHQ["📋 PHQ-9 Analysis\nPatient Health Questionnaire\n9-item Depression Scale"]
    ANALYZER --> GAD["📋 GAD-7 Analysis\nGeneralized Anxiety Disorder\n7-item Scale"]
    ANALYZER --> SUM["📝 Session Summary\nKey themes & observations"]

    PHQ --> SCORE1["Score: 0–27\n✅ Minimal · 🟡 Mild\n🟠 Moderate · 🔴 Severe"]
    GAD --> SCORE2["Score: 0–21\n✅ Minimal · 🟡 Mild\n🟠 Moderate · 🔴 Severe"]

    SCORE1 --> REPORT[(📊 Report JSON\nSQLite)]
    SCORE2 --> REPORT
    SUM --> REPORT

    REPORT --> DASH[🖥️ Frontend Dashboard\nVisual Analytics]

    style LOG fill:#1a1a2e,stroke:#e94560,color:#eee
    style ANALYZER fill:#0f3460,stroke:#533483,color:#eee
    style REPORT fill:#533483,stroke:#e94560,color:#eee
    style DASH fill:#16213e,stroke:#0f3460,color:#eee

Loading

---

🧰 Tech Stack

Layer	Technology	Purpose
Frontend	React+Vite+TailwindCSS	Component-based UI framework
3D Graphics	Three.js	Core WebGL 3D rendering engine
Framework	NestJS + TypeScript	Scalable server architecture
Real-time	Socket.io	Bidirectional audio & event streaming
Speech-to-Text	Sarvam AI Streaming API	Low-latency audio transcription
Text-to-Speech	Sarvam AI Streaming API	Avatar voice synthesis
LLM Orchestration	LangGraph	Stateful multi-agent graph execution
AI Models	OpenAI SDK	Response generation & emotion classification
Database	SQLite	Session & report persistence

---

🚀 Getting Started

Prerequisites

• Node.js >= 18.x
• npm >= 9.x
• Sarvam AI API Key
• OpenAI API Key

Installation

Step 1 — Clone the repository

git clone git clone https://github.com/luv29/innovate-you-backend.git
cd innovate-you-backend

Step 2 — Install dependencies

npm install

Step 3 — Configure environment variables

cp .env.example .env

Open .env and fill in your API keys and configuration (see Environment Variables below).

Step 4 — Start the development server

npm run start:dev

The server will be live at http://localhost:3000 🚀

Built with ❤️ for mental wellness · Powered by AI · Real-time · Empathetic