Feature/rag backend

[bedanley]

Upgrading RAG to support Collections

Overview

This PR introduces a comprehensive collections management system for LISA's RAG functionality, enabling users to organize documents with different chunking strategies, access controls, and metadata without requiring infrastructure changes.

Key Features

1. Collection Management

• Create/Read/Update/Delete collections within vector stores
• Hierarchical organization with inheritance from parent repositories
• Flexible chunking strategies per collection (FIXED_SIZE, with extensibility for SEMANTIC/RECURSIVE)
• Access control with group-based permissions and private collections
• Metadata tagging for organization and filtering

2. Enhanced Document Ingestion

• Collection-aware ingestion with automatic routing
• Chunking strategy override support during ingestion
• Backward compatibility with existing model-based collections
• Improved job tracking with detailed status information

3. Access Control Framework

• Generic access control system with caching
• Permission levels: READ, WRITE, ADMIN
• Group-based access with inheritance
• Private collections for user-specific data

4. API Enhancements

• Collection CRUD endpoints with pagination and filtering
• Enhanced similarity search with collection support
• Improved document management with collection context
• Better error handling and validation

Technical Changes

Database Schema

• New DynamoDB table: LisaRagCollectionsTable with GSIs for querying
• Enhanced document table: Added CollectionIndex GSI
• Collection fields: collectionId, repositoryId, chunkingStrategy, metadata, etc.

Backend Components

• collection_repo.py: DynamoDB operations for collections
• collection_service.py: Business logic and orchestration
• collection_validation.py: Validation rules and constraints
• collection_access_control.py: Permission checking
• chunking_strategy_factory.py: Extensible chunking strategy system
• access_control.py: Generic access control framework

API Endpoints

POST /repository/{repositoryId}/collection
GET /repository/{repositoryId}/collection/{collectionId}
PUT /repository/{repositoryId}/collection/{collectionId}
DELETE /repository/{repositoryId}/collection/{collectionId}
GET /repository/{repositoryId}/collections

SDK Updates

• lisapy/collection.py: Collection management methods
• Enhanced lisapy/rag.py: Collection-aware document operations

Testing

• Unit tests: Collection validation, access control, chunking strategies
• Integration tests: End-to-end collection lifecycle with document ingestion
• Test utilities: Reusable authentication and resource management helpers

By submitting this pull request, I confirm that you can use, modify, copy, and redistribute this contribution, under the terms of your choice.