feat(rag): implement PRP-9 RAG Knowledge Base with pgvector

[w7-mgfcode]

Summary

• Implement RAG (Retrieval-Augmented Generation) knowledge base feature for semantic document indexing and retrieval
• Add pgvector-based vector storage in PostgreSQL with HNSW index for fast similarity search
• Support markdown-aware and OpenAPI-aware document chunking with tiktoken token counting
• Enable idempotent re-indexing via SHA-256 content hash comparison

Changes

New RAG Feature (app/features/rag/)

• models.py: ORM models DocumentSource and DocumentChunk with pgvector Vector column
• schemas.py: Pydantic v2 schemas with strict validation for all API contracts
• chunkers.py: MarkdownChunker (heading-aware) and OpenAPIChunker (endpoint-based)
• embeddings.py: EmbeddingService with async OpenAI API, batch processing, rate limit handling
• service.py: RAGService orchestrating indexing, retrieval, and source management
• routes.py: FastAPI endpoints for RAG operations

API Endpoints

• POST /rag/index - Index documents with automatic chunking and embedding
• POST /rag/retrieve - Semantic search with cosine similarity and relevance scoring
• GET /rag/sources - List indexed sources with chunk statistics
• DELETE /rag/sources/{source_id} - Remove source and cascade delete chunks

Configuration

• Added RAG settings to app/core/config.py (embedding, chunking, retrieval, index config)
• Updated .env.example with RAG environment variables
• Added dependencies: pgvector, openai, tiktoken, httpx

Database

• Migration b4c8d9e0f123 creates pgvector extension and RAG tables with HNSW index

Tests

• 68 unit tests covering schemas, chunkers, embeddings, and service
• 14 integration tests for API routes (requires PostgreSQL)

Test plan

• Run uv run pytest app/features/rag/tests/ -v -m "not integration" - 68 unit tests pass
• Run uv run mypy app/features/rag/ - 0 issues
• Run uv run pyright app/features/rag/ - 0 errors
• Run uv run ruff check app/features/rag/ - All checks passed
• Run integration tests with PostgreSQL + pgvector: uv run pytest app/features/rag/tests/ -v -m integration
• Apply migration and smoke test endpoints manually

🤖 Generated with Claude Code

[sourcery-ai]

Sorry @w7-mgfcode, you have reached your weekly rate limit of 500000 diff characters.

Please try again later or upgrade to continue using Sourcery

[coderabbitai]

Important

Review skipped

Auto reviews are disabled on base/target branches other than the default branch.

Please check the settings in the CodeRabbit UI or the .coderabbit.yaml file in this repository. To trigger a single review, invoke the @coderabbitai review command.

You can disable this status message by setting the reviews.review_status to false in the CodeRabbit configuration file.

• 🔍 Trigger a full review

✨ Finishing touches

🧪 Generate unit tests (beta)

• Create PR with unit tests
• Post copyable unit tests in a comment
• Commit unit tests in branch feat/prp-9-rag-knowledge-base

---

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

• X
• Mastodon
• Reddit
• LinkedIn

_{Comment @coderabbitai help to get the list of available commands and usage tips.}

[socket-security]

Review the following changes in direct dependencies. Learn more about Socket for GitHub.

Diff	Package	Supply Chain Security	Vulnerability	Quality	Maintenance	License
	openai@​2.16.0
	pgvector@​0.4.2

View full report

[w7-mgfcode]

Verification Complete ✅

Migration Applied

• b4c8d9e0f123_create_rag_tables.py successfully applied
• Created pgvector extension, document_source and document_chunk tables with HNSW index

Integration Tests

All 14 integration tests passed:

app/features/rag/tests/test_routes.py::TestIndexEndpoint::test_index_markdown_creates_chunks PASSED
app/features/rag/tests/test_routes.py::TestIndexEndpoint::test_index_same_content_returns_unchanged PASSED
app/features/rag/tests/test_routes.py::TestIndexEndpoint::test_index_updated_content_re_indexes PASSED
app/features/rag/tests/test_routes.py::TestIndexEndpoint::test_index_invalid_source_type PASSED
app/features/rag/tests/test_routes.py::TestIndexEndpoint::test_index_file_not_found PASSED
app/features/rag/tests/test_routes.py::TestRetrieveEndpoint::test_retrieve_returns_relevant_chunks PASSED
app/features/rag/tests/test_routes.py::TestRetrieveEndpoint::test_retrieve_respects_threshold PASSED
app/features/rag/tests/test_routes.py::TestRetrieveEndpoint::test_retrieve_empty_database PASSED
app/features/rag/tests/test_routes.py::TestRetrieveEndpoint::test_retrieve_validates_query PASSED
app/features/rag/tests/test_routes.py::TestSourcesEndpoint::test_list_sources_returns_all PASSED
app/features/rag/tests/test_routes.py::TestSourcesEndpoint::test_delete_source_removes_chunks PASSED
app/features/rag/tests/test_routes.py::TestSourcesEndpoint::test_delete_nonexistent_returns_404 PASSED
app/features/rag/tests/test_routes.py::TestSourcesEndpoint::test_source_not_in_list_after_delete PASSED
app/features/rag/tests/test_routes.py::TestOpenAPIIndexing::test_index_openapi_creates_endpoint_chunks PASSED

Smoke Test Results

All endpoints respond correctly at http://localhost:8123/rag/*:

• ✅ GET /rag/sources → Returns source list
• ✅ POST /rag/index → Indexes documents (requires OPENAI_API_KEY)
• ✅ POST /rag/retrieve → Retrieves relevant chunks (requires OPENAI_API_KEY)
• ✅ DELETE /rag/sources/{id} → Proper 404 handling
• ✅ Validation errors return RFC 7807 error responses

Ready for review.