AskDocs — Document Intelligence Platform

A Retrieval-Augmented Generation (RAG) system that lets administrators upload documents, and allows users to have AI-powered conversations grounded strictly in those documents. Built with Laravel 13, React 19, and OpenAI + Cohere.

---

Table of Contents

• Description
• Features
• Tech Stack
• Architecture Overview
• Folder Structure
• Installation & Setup
• Environment Variables
• Usage
• API / Route Documentation
• UI Notes
• Deployment
• Future Improvements
• Contributing
• License

---

Description

AskDocs is a full-stack SaaS-style document-chat platform. Administrators upload PDF documents which are parsed, chunked, embedded via OpenAI, and stored in a vector database. Admins can then chat with any document and manage which users have access to which documents. End users log in and chat only with documents explicitly assigned to them.

Every chat query follows a RAG pipeline:

1. The user message is converted to an embedding vector.
2. Up to 25 semantically similar chunks are retrieved from the vector store.
3. Cohere re-ranks those candidates and returns the top 10 most relevant chunks.
4. GPT-4o mini answers the question using only that context — streamed back in real time.

---

Features

Admin

• Upload PDF documents (up to 50 MB)
• Automatic async document processing: text extraction → chunking → OpenAI embedding → vector storage
• Document search and paginated listing
• Assign / revoke document access per user
• Create, update, enable/disable, and delete user accounts
• Full chat interface — chat with any uploaded document
• Rename and delete individual chats or clear all chat history
• Dashboard overview

User (Viewer)

• Log in and chat with documents assigned by the admin
• Persistent chat history per document
• Real-time streamed AI responses with Markdown rendering
• Rename and delete chats

RAG Pipeline

• PDF parsing via pdftotext (Poppler CLI) with fallback to Smalot PdfParser
• Overlapping text chunking via TextChunker
• Embeddings via OpenAI text-embedding-3-small (1536 dimensions)
• Cosine-similarity vector search with a 0.3 threshold
• Cohere rerank-v4.0-fast re-ranking for precision (cached 10 min)
• Query embedding caching (1 hour)
• AI completions via OpenAI gpt-4o-mini-2024-07-18 — streamed via SSE
• Conversation history (up to last 50 messages) passed to the agent

Platform

• Role-based access control: admin vs viewer
• Two-factor authentication (via Laravel Fortify)
• Queue-based document processing (3 retries, 5 min timeout, 60 s backoff)
• Storage-disk abstraction: local (public) in dev, S3/R2 in production
• TypeScript + ESLint + Prettier code quality tooling

---

Tech Stack

Layer	Technology
Backend Framework	Laravel 13 (PHP ^8.3)
Frontend Framework	React 19 + TypeScript 5.7
SPA Bridge	Inertia.js 3 (server-driven SPA)
Build Tool	Vite 8 + laravel-vite-plugin
Styling	Tailwind CSS 4
UI Components	Radix UI primitives, Headless UI, Lucide icons
Auth	Laravel Fortify 1.34 (email/password + 2FA)
AI	laravel/ai ^0.4.2 — OpenAI GPT-4o mini + embeddings
Reranking	Cohere API v2 (rerank-v4.0-fast)
PDF Parsing	pdftotext (Poppler) + smalot/pdfparser ^2.12
Database	SQLite (dev) / MySQL (prod)
Queue	Laravel database queue
Storage	Local (public disk, dev) / AWS S3 (prod)
Cache	Laravel database cache
Testing	Pest 4
Observability	Laravel Nightwatch 1.26
Routing Codegen	Laravel Wayfinder (type-safe routes in TS)

---

Architecture Overview

┌────────────────────────────────────────────────────────────┐
│                          Browser                           │
│              React 19 SPA  (Inertia.js)                    │
└──────────────────────────┬─────────────────────────────────┘
                           │ HTTP / SSE
┌──────────────────────────▼─────────────────────────────────┐
│                  Laravel 13 Application                     │
│  ┌──────────────────┐   ┌────────────────────────────────┐ │
│  │  Admin Routes    │   │     User (Viewer) Routes       │ │
│  │  /admin/*        │   │     /chat/*                    │ │
│  └────────┬─────────┘   └───────────────┬────────────────┘ │
│           │                             │                   │
│  ┌────────▼─────────────────────────────▼────────────────┐ │
│  │            AiChatService  (RAG pipeline)               │ │
│  │  1. Embed query  →  2. Vector search  →               │ │
│  │  3. Cohere rerank  →  4. Build prompt  →              │ │
│  │  5. Stream GPT-4o mini response                        │ │
│  └───────────────────────────────────────────────────────┘ │
│                                                             │
│  ┌─────────────────────────────────────────────────────┐   │
│  │           Queue Worker  (document ingestion)         │   │
│  │  Upload  →  Parse PDF  →  Chunk  →  Embed  →  DB    │   │
│  └─────────────────────────────────────────────────────┘   │
└────────────────────────────────────────────────────────────┘
          │                     │              │
     ┌────▼────┐         ┌──────▼──────┐  ┌───▼──────┐
     │ SQLite/ │         │  S3 / Local │  │ OpenAI + │
     │  MySQL  │         │   Storage   │  │  Cohere  │
     └─────────┘         └─────────────┘  └──────────┘

Document lifecycle:

Admin uploads PDF
      │
      ▼
DocumentController.store()
  → stores file on disk
  → creates Document (status=processing)
  → dispatches ProcessDocument job
      │
      ▼  (queue worker)
ProcessDocument.handle()
  → DocumentParser.extractText()  (pdftotext → Smalot fallback)
  → TextChunker.chunk()
  → OpenAI Embeddings (batched, 100 chunks/request)
  → bulk-insert DocumentChunks
  → Document (status=ready)

---

Folder Structure

ask-docs/
├── app/
│   ├── Ai/
│   │   └── Agents/
│   │       └── AskDoc.php          # Laravel AI agent — system instructions + conversation history
│   ├── Enums/
│   │   └── UserRole.php            # admin | viewer
│   ├── Http/
│   │   ├── Controllers/
│   │   │   ├── Admin/              # Admin-only: Dashboard, Documents, Users, Chat
│   │   │   └── User/               # Viewer: Chat
│   │   └── Middleware/             # admin / viewer guards
│   ├── Jobs/
│   │   └── ProcessDocument.php     # Async ingestion job
│   ├── Models/
│   │   ├── Document.php            # PDF doc + storage helpers
│   │   ├── DocumentChunk.php       # Chunk + embedding
│   │   ├── Chat.php
│   │   ├── Message.php
│   │   └── User.php
│   └── Services/
│       ├── AiChatService.php       # RAG orchestration (embed → search → rerank → stream)
│       ├── CohereService.php       # Cohere Rerank API v2 client
│       ├── DocumentParser.php      # PDF text extraction
│       └── TextChunker.php         # Overlapping text chunking
├── database/
│   └── migrations/                 # Full schema: users, documents, chunks, chats, messages
├── resources/
│   └── js/                         # React 19 frontend (TypeScript)
├── routes/
│   ├── web.php                     # All web routes
│   └── settings.php                # Profile / password / 2FA settings routes
├── .env.example                    # All required env variables
└── vite.config.ts

---

Installation & Setup

Prerequisites

Requirement	Version
PHP	^8.3
Composer	>= 2
Node.js	>= 20
npm	>= 10
SQLite	(dev default)
pdftotext (optional)	Poppler CLI — improves PDF extraction performance

Quick Setup (one command)

composer run setup

This runs: composer install → copy .env → generate app key → migrate → npm install → npm run build.

---

Manual Setup

# 1. Clone the repository
git clone <repo-url> ask-docs
cd ask-docs

# 2. Install PHP dependencies
composer install

# 3. Create and configure environment file
cp .env.example .env
php artisan key:generate

# 4. Configure .env (see Environment Variables section)
#    Minimum required: APP_KEY, OPENAI_API_KEY, COHERE_API_KEY

# 5. Run database migrations
php artisan migrate

# 6. Install Node dependencies and build assets
npm install
npm run build

# 7. Create the first admin user
php artisan tinker
>>> \App\Models\User::create([
...     'name'     => 'Admin',
...     'email'    => 'admin@example.com',
...     'password' => bcrypt('password'),
...     'role'     => \App\Enums\UserRole::Admin,
...     'status'   => true,
... ]);

---

Environment Variables

# Application
APP_NAME=AskDocs
APP_ENV=local
APP_KEY=                          # Generated by php artisan key:generate
APP_DEBUG=true
APP_URL=http://localhost

# Database (SQLite for dev, configure host/port for MySQL in prod)
DB_CONNECTION=sqlite
# DB_HOST=127.0.0.1
# DB_PORT=3306
# DB_DATABASE=ask_docs
# DB_USERNAME=root
# DB_PASSWORD=

# Queue & Cache (database driver — zero extra infra for dev)
QUEUE_CONNECTION=database
CACHE_STORE=database
SESSION_DRIVER=database

# File Storage
# Local dev:   DOCUMENT_STORAGE_DISK=public
# Production:  DOCUMENT_STORAGE_DISK=s3
DOCUMENT_STORAGE_DISK=public

# AWS S3 (required when DOCUMENT_STORAGE_DISK=s3)
AWS_ACCESS_KEY_ID=
AWS_SECRET_ACCESS_KEY=
AWS_DEFAULT_REGION=us-east-1
AWS_BUCKET=

# OpenAI (required) — used for embeddings and chat completions
OPENAI_API_KEY=                   # Set via laravel/ai configuration

# Cohere (required for reranking)
COHERE_API_KEY=
COHERE_RERANK_MODEL=rerank-v4.0-fast

# Mail (optional for dev — defaults to log driver)
MAIL_MAILER=log

Note: OPENAI_API_KEY is consumed by the laravel/ai package. Refer to the Laravel AI package documentation for the exact config key if not auto-detected.

---

Usage

Running in Development

composer run dev

This concurrently starts:

• php artisan serve — Laravel backend (:8000)
• php artisan queue:listen --tries=1 — Queue worker for document processing
• npm run dev — Vite HMR dev server

Key Workflows

Admin: Upload & Chat

1. Log in at /admin
2. Go to Documents → upload a PDF
3. Wait for status to change to Ready (processed asynchronously)
4. Go to Chat → select the document → start chatting

Admin: User Management

1. Go to Users → create a new viewer account (name, email, password)
2. Go to Documents → click Assign on a document → select users
3. The user can now log in at /chat and chat with that document

User: Chat

1. Log in — redirected to /chat
2. Select a document from the sidebar (only assigned documents appear)
3. Chat; responses are streamed in real time with Markdown support

---

API / Route Documentation

All routes are web routes served via Inertia.js (HTML for the first load, JSON for subsequent SPA navigation).

Auth Routes (Laravel Fortify)

Method	Path	Description
GET	/login	Login page
POST	/login	Authenticate
POST	/logout	Logout
GET	/two-factor-challenge	2FA challenge

User (Viewer) Routes — auth + verified + viewer

Method	Path	Description
GET	/chat	Chat dashboard
GET	/chat/{chat}	Load a specific chat
POST	/chat	Send a message / start a chat (SSE stream)
PUT	/chat/{chat}	Rename chat
DELETE	/chat/{chat}	Delete a chat
DELETE	/chat	Delete all chats

Admin Routes — auth + verified + admin, prefix /admin

Method	Path	Description
GET	/admin	Dashboard
GET	/admin/documents	List documents (paginated, searchable)
POST	/admin/documents	Upload PDF (dispatches ingestion job)
POST	/admin/documents/{document}/assign	Assign document to users
DELETE	/admin/documents/{document}	Delete document + file
GET	/admin/users	List users
POST	/admin/users	Create viewer user
PUT	/admin/users/{user}	Update user
DELETE	/admin/users/{user}	Delete user
POST	/admin/users/{user}/access	Toggle user access
GET	/admin/chat	Chat dashboard
GET	/admin/chat/{chat}	Load a specific chat
POST	/admin/chat	Send a message (SSE stream)
PUT	/admin/chat/{chat}	Rename chat
DELETE	/admin/chat/{chat}	Delete a chat
DELETE	/admin/chat	Delete all chats

Chat endpoint response: Content-Type: text/event-stream. The X-Chat-Id response header carries the chat ID for newly created chats.

Settings Routes

Method	Path	Description
GET/POST/PUT/DELETE	/settings/*	Profile, password, 2FA management

---

UI Notes

• Framework: React 19 + TypeScript with Inertia.js (SSR-compatible)
• Component Library: Radix UI primitives + Headless UI (fully accessible)
• Icons: Lucide React
• Styling: Tailwind CSS v4 with tw-animate-css and class-variance-authority
• Chat markdown: react-markdown + remark-gfm for GFM rendering
• Streaming: @laravel/stream-react for consuming SSE chat streams
• Code quality: ESLint 9 + Prettier 3 + TypeScript strict mode

The frontend has two separate layouts:

• /admin/* — Admin panel (documents, users, chat sidebar)
• /chat/* — Viewer chat interface (documents assigned to the logged-in user only)

---

Storage

In production set DOCUMENT_STORAGE_DISK=s3 and fill in all AWS_* variables. The application will automatically upload documents to S3 and read them back from there — no code changes required.

Future Improvements

• Multi-format support: Add Word (.docx) and plain-text (.txt) parsing to DocumentParser
• Streaming progress: Real-time document processing progress indicator in the UI
• Semantic chunking: Replace fixed-size chunks with sentence-boundary or semantic chunkers
• Bulk document upload: Multi-file upload with per-file status tracking

---

Built with Laravel · React · OpenAI · Cohere