M2: Document Indexing API on OpenAI-Compatible Server

[kovtcharov]

Priority: P0 — Required for RAG-based webapp integration

Effort: Medium (3-5 days)

Problem

The only way to index documents into GAIA's RAG system over HTTP is through the full UI server (gaia chat --ui), which brings in session management, Electron dependencies, and a heavy stack. The OpenAI-compatible API server (openai_server.py) has zero document/RAG endpoints. A webapp backend cannot programmatically index content.

Deliverable

Add document management endpoints to openai_server.py:

POST   /v1/documents          — Index a document (file path, URL, or raw text/markdown)
GET    /v1/documents          — List all indexed documents with metadata
GET    /v1/documents/:id      — Get document details (chunk count, index status)
DELETE /v1/documents/:id      — Remove a document from the index
POST   /v1/documents/query    — Query indexed documents directly (bypass agent)

Implementation approach

• Instantiate a shared RAGSDK instance in the API server (singleton, lazy-initialized)
• POST /v1/documents accepts: { "path": "/path/to/doc.md" } or { "content": "raw text...", "name": "doc-name" }
• Supported formats: .md, .txt, .pdf (via existing pypdf integration in RAGSDK)
• Index status tracking: pending, indexing, ready, error
• The ChatAgent's query_documents tool automatically searches documents indexed via this API

Files to change

• src/gaia/api/openai_server.py — add 5 new FastAPI routes
• src/gaia/api/schemas.py — add request/response models for documents
• Possibly src/gaia/rag/sdk.py — expose document listing/removal if not already available

Acceptance criteria

• A webapp can POST /v1/documents with a file path or raw text and get a document ID back
• Subsequent chat queries via /v1/chat/completions can retrieve content from indexed documents
• GET /v1/documents returns a list of all indexed documents with status
• DELETE /v1/documents/:id removes a document and its chunks from the index
• Documents persist across API server restarts (file-based index)

Context

Full milestone plan: docs/plans/webapp-integration.md