Skip to content
BranchesTags

Repository files navigation

AI Companion

An emotionally intelligent, privacy-focused AI companion platform.

License: MIT Python Node FastAPI React TypeScript

Features • Quick Start • Architecture • Project Structure • Contributing

📖 Overview

AI Companion is a full-stack web application for building and chatting with personalized AI companions. It combines real-time streaming chat, long-term vector memory, retrieval-augmented document understanding, agentic tool use, and 3D avatars into a single platform.

The project is built as a production-oriented reference implementation: multi-provider LLM support (OpenAI + Google Gemini), WebSocket-based real-time messaging, a tiered memory system on top of pgvector, Supabase-backed auth/storage, and Stripe-based subscription tiers.

✨ Features

User-facing

  • Customizable companions — personality, relationship type, communication style, voice, and ReadyPlayerMe 3D avatar
  • Real-time chat — WebSocket messaging with token streaming and typing indicators
  • Long-term memory — companion-scoped semantic memory stored as pgvector embeddings; persists across conversations
  • Document RAG — upload PDF, DOCX, XLSX, PPTX, TXT, or Markdown; chunks are embedded and retrieved contextually during chat
  • 3D avatars — Three.js viewer with progressive loading, 2D fallback, and emotion states (neutral, happy, sad, excited, thinking, speaking)
  • Voice I/O — text-to-speech via OpenAI TTS, speech-to-text via Google Cloud Speech
  • Vision — image analysis via GPT-4 Vision / Gemini Vision
  • Agentic mode — LangGraph-based tool calling with Composio for third-party integrations (Google Calendar, web search, etc.)
  • Activities & gamification — interactive exercises with XP, milestones, and reward tracking
  • Analytics dashboard — conversation sentiment, usage metrics, session trends
  • Multi-provider AI — users can choose between OpenAI and Google Gemini; the backend routes by task complexity
  • Subscriptions — Stripe-powered free / pro / premium tiers with feature gating

Platform

  • Supabase authentication (email/password, email verification, password reset, OAuth)
  • JWT-based API auth with refresh tokens
  • PostgreSQL 15+ with pgvector for embeddings
  • Redis for caching, rate limiting, and pub/sub
  • Supabase Storage for uploaded documents and avatars
  • Structured logging, tiered rate limiting, CORS, trusted-host middleware
  • OpenAPI docs auto-generated by FastAPI

🚀 Quick Start

Prerequisites

  • Python 3.11+
  • Node.js 18+ and npm 9+
  • PostgreSQL 15+ with the pgvector extension (or a Supabase project)
  • Redis 7+ (optional for local dev, required for production features)
  • Accounts / API keys you'll need:
    • OpenAI and/or Google AI Studio (Gemini)
    • Supabase project (for auth, database, storage)
    • Stripe (optional, only for subscription features)
    • Google Cloud (optional, only for TTS/STT via Vertex AI)

1. Clone and install

git clone https://github.com/NinjaCodeTurtle/AICompanion.git
cd AICompanion
npm run install:all

install:all installs both the frontend (npm install in frontend/) and the backend (pip install -r requirements.txt in backend/). For the backend you may want to create a virtualenv first:

cd backend
python3 -m venv venv
source venv/bin/activate   # Windows: venv\Scripts\activate
pip install -r requirements.txt
cd ..

2. Configure environment

cp backend/.env.example backend/.env
cp frontend/.env.example frontend/.env

Fill in the values — at minimum you need a database URL, a Supabase project, and an LLM API key (OpenAI or Google AI). See backend/.env.example for the full list with inline documentation.

3. Run database migrations

cd backend
source venv/bin/activate
alembic upgrade head
cd ..

4. Start the app

From the repository root:

npm run dev

This runs the frontend (Vite) and backend (uvicorn with reload) concurrently:

You can also start them individually with npm run dev:frontend / npm run dev:backend.

🏗️ Architecture

┌────────────────────────────────────────────────────────────────┐
│                    Frontend  (React 19 + Vite)                 │
│  Chat · Companions · Memory · Documents · Activities · Admin   │
│           Three.js + ReadyPlayerMe 3D avatars                  │
└──────────────────────────┬─────────────────────────────────────┘
                           │ REST + WebSocket (Socket.io / native)
┌──────────────────────────┴─────────────────────────────────────┐
│                   Backend  (FastAPI, async)                    │
│  API routers:  auth · chat · companion · memory · documents   │
│                activity · agent · analytics · subscription    │
│                images · voice · tts · tools                   │
│                                                                │
│  Services:     AI providers (OpenAI / Gemini factory)         │
│                Conversation + context management              │
│                Memory (pgvector, privacy, classification)     │
│                Document processing (PDF/DOCX/XLSX/MD)         │
│                LangGraph agents + Composio tools              │
│                Avatar, OAuth, Stripe, WebSocket services      │
└──────────────────────────┬─────────────────────────────────────┘
                           │
┌──────────────────────────┴─────────────────────────────────────┐
│                          Data Layer                            │
│   Supabase Postgres (pgvector)   ·   Redis   ·   Supabase CDN  │
│   OpenAI  ·  Google Gemini / Vertex AI  ·  Stripe  ·  Composio │
└────────────────────────────────────────────────────────────────┘

Tech stack

Frontend

  • React 19.1 + TypeScript 5.8
  • Vite 7 build tool
  • Tailwind CSS 3.4 + Framer Motion
  • Zustand (state) + TanStack Query (server state)
  • React Router 7
  • Three.js + @react-three/fiber + @react-three/drei + @react-three/postprocessing
  • @readyplayerme/react-avatar-creator for avatar customization
  • Socket.io-client for real-time, Axios for REST
  • Recharts for analytics, Zod for validation, Lucide icons

Backend

  • FastAPI 0.109+ on Python 3.11+
  • SQLAlchemy 2.0 async + asyncpg
  • Pydantic 2.7+, Alembic migrations
  • Structlog for structured logging

Data & infrastructure

  • PostgreSQL 15+ with pgvector
  • Redis 7+
  • Supabase (auth, Postgres, storage)

AI & ML

  • OpenAI — GPT-4 Turbo / GPT-3.5 Turbo, Ada-002 embeddings, Whisper, TTS
  • Google Gemini — Gemini 2.5 Pro / 2.5 Flash, text-embedding-004, Imagen via Vertex AI
  • Google Cloud Speech — speech-to-text and text-to-speech
  • LangChain + LangGraph — agent orchestration
  • Composio — managed third-party tool catalog

Payments & real-time

  • Stripe (subscriptions, webhooks, billing portal)
  • Native WebSockets + Socket.io for chat

📁 Project Structure

AICompanion/
├── backend/
│   ├── app/
│   │   ├── api/v1/           # FastAPI routers (auth, chat, companion,
│   │   │                     #   memory, documents, activity, agent,
│   │   │                     #   analytics, subscription, images,
│   │   │                     #   voice, tts, tools, user_preferences)
│   │   ├── core/             # Config, deps, security, logging, middleware
│   │   ├── models/           # SQLAlchemy models
│   │   ├── repositories/     # Data access layer
│   │   ├── schemas/          # Pydantic request/response models
│   │   ├── services/         # Business logic (AI providers, memory,
│   │   │                     #   documents, agents, analytics, avatar,
│   │   │                     #   OAuth, Stripe, WebSocket, etc.)
│   │   └── main.py           # FastAPI app entrypoint
│   ├── alembic/              # Database migrations
│   ├── tests/                # Backend tests
│   ├── requirements.txt
│   └── .env.example
├── frontend/
│   ├── src/
│   │   ├── pages/            # Route-level components
│   │   ├── components/
│   │   │   ├── avatar/       # 3D viewer, progressive loading, fallback
│   │   │   ├── chat/         # Chat container, input, renderer, voice
│   │   │   ├── companions/   # Builder, form, card
│   │   │   ├── memory/       # List, search, document linker
│   │   │   ├── documents/    # Uploader, list, viewer
│   │   │   ├── activities/   # Cards, execution, games
│   │   │   ├── analytics/    # Charts, metrics, sentiment
│   │   │   ├── subscription/ # Plans, payment, billing
│   │   │   ├── rewards/      # XP, milestones, celebrations
│   │   │   ├── auth/         # Protected routes, forms
│   │   │   ├── layout/       # App layout, nav, sidebar
│   │   │   └── ui/           # Primitives (button, input, card, modal)
│   │   ├── services/         # API clients
│   │   ├── stores/           # Zustand stores
│   │   ├── hooks/
│   │   ├── types/
│   │   └── App.tsx
│   ├── public/
│   ├── package.json
│   └── .env.example
├── docs/                     # Architecture, API spec, DB schema, design
├── LICENSE
├── package.json              # Root workspace scripts
└── README.md

🛠️ Development

Scripts

From the repository root (package.json):

Command What it does
npm run dev Runs frontend + backend concurrently
npm run dev:frontend Vite dev server on :5173
npm run dev:backend uvicorn with reload on :8001
npm run install:all Install both frontend and backend dependencies
npm run build:frontend Production build of the frontend
npm run test Run frontend and backend test suites
npm run test:backend pytest in backend/
npm run lint Lint frontend (eslint) and backend (black, isort, mypy)
npm run format Format both projects

Backend

cd backend
source venv/bin/activate

pytest                         # run all tests
pytest --cov=app               # with coverage
pytest tests/test_auth.py      # a single file

black app/                     # format
isort app/                     # sort imports
mypy app/                      # type check
flake8 app/                    # lint

Frontend

cd frontend

npm run dev                    # dev server
npm run build                  # production build
npm run preview                # preview production build
npm run lint                   # eslint
npm run type-check             # tsc --noEmit

📚 Documentation

Deeper documentation lives under docs/:

Document Description
system-design.md Architecture, scalability, and design decisions
api-specification.md REST endpoints and request/response schemas
database-schema.md PostgreSQL schema, pgvector usage, RLS policies
requirement-specification.md Feature requirements and success metrics
tech-stack.md Technology choices and justifications
web-design-specification.md UI components and design system
user-journey.md User flows and engagement patterns

🤝 Contributing

Contributions are welcome. To propose a change:

  1. Fork the repository
  2. Create a feature branch: git checkout -b feature/your-feature
  3. Make your changes and add tests
  4. Ensure lint and tests pass: npm run lint && npm run test
  5. Commit using Conventional Commits: feat(chat): ..., fix(auth): ..., docs: ...
  6. Push and open a Pull Request

Before opening a large PR, please open an issue to discuss direction.

🔒 Security

  • JWT-based authentication with refresh tokens
  • Supabase Row Level Security policies on user data
  • Pydantic / Zod validation at every boundary
  • Parameterized queries via SQLAlchemy ORM (no raw SQL from user input)
  • Tiered rate limiting tied to subscription plan
  • CORS and trusted-host middleware
  • Secrets loaded from environment only — never commit real keys

If you discover a security vulnerability, please open a private security advisory on GitHub rather than a public issue.

📄 License

This project is licensed under the MIT License. See LICENSE for details.

About

ai-companion-nine-mu.vercel.app

Resources

Readme
MIT license
Activity

Stars

0 stars

Watchers

0 watching

Forks

0 forks
Report repository

Releases

Packages

Contributors

Languages