AI Compliance Testing Backend

A comprehensive backend system for testing AI models against compliance frameworks including GDPR, HIPAA, EU AI Act, and other regulatory standards. This FastAPI-based system provides unified access to multiple LLM providers and features advanced RAG (Retrieval Augmented Generation) capabilities for enhanced compliance analysis.

Key Features

• Multi-Provider LLM Support: Unified API for Anthropic, OpenAI, Google, HuggingFace, and local models
• Compliance Evaluators: Pre-built evaluators for major regulatory frameworks
• RAG System: Enhanced analysis using vector-based document retrieval
• Batch Testing: Efficient testing across multiple models and prompts
• Real-time API: RESTful endpoints with automatic documentation
• GPU Acceleration: Optimized for NVIDIA GPUs with CPU fallback

Quick Start

# Clone and setup
git clone https://github.com/SamH135/AI-Compliance-Python-Backend.git
cd AI-Compliance-Python-Backend
python -m venv venv && source venv/bin/activate
pip install -r requirements.txt

# Configure environment
cp .env.example .env
# Edit .env ONLY for Milvus connection settings and other backend configurations.
# API keys for LLM providers are NOT set in .env; they are provided in each API request payload.

# Start services
docker run -d --name milvus -p 19530:19530 milvusdb/milvus:latest  # Optional, for RAG
python api_server.py

# Test the API
curl http://localhost:8000/api

Access Documentation: http://localhost:8000/docs

Architecture Overview

Core Components

FastAPI Server (api_server.py)

• REST API endpoints for all functionality
• Automatic OpenAPI documentation at /docs
• CORS enabled for frontend integration

Unified API System (framework/api/)

• Provider-agnostic LLM interface
• Auto-discovery of providers and models
• Standardized configuration via ModelConfig
• Support for streaming and async operations

Evaluators (framework/evaluators/)

• Compliance testing engines
• Support for rule-based, LLM-enhanced, and RAG-enabled analysis
• Auto-registry system for easy extensibility

RAG System (handled by specific evaluators using RAG)

• Vector-based document retrieval using Milvus
• PDF processing and embedding generation
• Context-aware compliance analysis

Data Flow

1. Client Request: Frontend sends test request with model config and API keys
2. Model Invocation: Unified API routes to appropriate provider (Anthropic, OpenAI, etc.)
3. Evaluation: Selected evaluators analyze the response using rule-based, LLM, or RAG methods
4. Response: Compiled results returned with scores, interpretations, and recommendations

API Overview

Core Testing Endpoints

# Get available evaluators and their capabilities
GET /api/evaluators

# Get available model providers and configuration options
GET /api/models

# Run compliance tests on a single prompt
POST /api/test
{
  "model_type": "anthropic",
  "configuration": {
    "model_id": "claude-3-haiku-20240307",
    "temperature": 0.3,
    "max_tokens": 1000
  },
  "prompt": "Hello, what data do you collect about me?",
  "selected_tests": ["GenericComplianceEvaluator"],
  "api_key": "sk-your-api-key-here"
  // "rag_configuration" can also be added here if needed by an evaluator
}

# Batch testing across multiple models and prompts
POST /api/test/batch

RAG System Endpoints

# Query compliance knowledge base
POST /rag/chatbot
{
  "text": "What are GDPR requirements for data breach notification?",
  "model_type": "anthropic",
  "configuration": {"model_id": "claude-3-haiku-20240307"},
  "api_key": "sk-your-key"
}

# Upload compliance documents
POST /rag/upload-pdf

# Check processing status
GET /rag/status/{request_id}

Diagnostic Endpoints

GET /api/platform        # System information
GET /api/debug/gpu       # GPU diagnostics
GET /api/health/milvus   # Vector database status

Configuration

Environment Variables (Backend Settings Only)

API keys for LLM providers are always provided in the request payloads, NOT in environment variables. The .env file is used for backend server settings, such as:

# Required for RAG features if Milvus is used
MILVUS_HOST=localhost
MILVUS_PORT=19530

# Optional: HuggingFace token for downloading models (not for API runtime auth for HF Inference Endpoints)
HF_TOKEN=your_huggingface_download_token

# Optional: GPU selection
CUDA_VISIBLE_DEVICES=0

API Key Management

API keys are passed dynamically in each API request's JSON payload. The top-level api_key is used for the primary model being tested. If an evaluator utilizes RAG and requires its own LLM, the api_key for that RAG-specific LLM is provided within the rag_configuration block.

Example structure:

{
  "api_key": "sk-your-provider-key",           // For the primary model being tested
  "rag_configuration": {
    "model_type": "some_provider",             // RAG LLM provider
    "configuration": { /* ... */ },
    "api_key": "sk-your-rag-model-key",        // For the LLM used by the RAG-enabled evaluator
    "collection_name": "relevant_docs"         // Example: RAG collection
    // ... other RAG settings
  }
}

Supported Providers:

• Anthropic (Claude models)
• OpenAI (GPT models)
• Google (Gemini models)
• HuggingFace (Open source models)
• Local models

Evaluators

Available Evaluators

Generic Compliance Evaluator

• Multi-mode analysis (rule-based, LLM-enhanced, RAG-enabled)
• Supports GDPR, HIPAA, EU AI Act, and custom frameworks
• Configurable risk thresholds and analysis depth

Claude Compliance Evaluator

• Powered by Anthropic's Claude API
• High-accuracy compliance analysis
• Detailed explanations and remediation suggestions

Hallucination Checker

• Factual accuracy verification
• Cross-reference with authoritative sources
• Optional web search integration

Evaluator Capabilities

Each evaluator provides metadata including:

• Required vs. optional LLM support
• RAG enhancement capabilities
• Suggested providers and models
• Configuration requirements
• Risk thresholds and scoring

Installation Options

Standard Installation

# Python 3.12 required
git clone https://github.com/SamH135/AI-Compliance-Python-Backend.git
cd AI-Compliance-Python-Backend

# Virtual environment
python -m venv venv
source venv/bin/activate  # Windows: venv\Scripts\activate

# Dependencies
pip install -r requirements.txt

# Environment setup
cp .env.example .env
# IMPORTANT: Edit .env for Milvus/backend settings only. LLM API keys go in request payloads.

Docker Installation

# Full stack with Docker Compose
docker-compose up --build

# Manual Docker build
docker build -f Dockerfile.py312 -t ai-compliance .
docker run --gpus all -p 8000:8000 ai-compliance

Production Deployment

# Multiple workers for production
uvicorn api_server:app --host 0.0.0.0 --port 8000 --workers 4

# With reverse proxy (nginx recommended)
# See docs/setup.md for detailed production setup

Milvus Vector Database

Required for RAG features. Can be omitted if only using basic evaluators.

# Docker (recommended)
docker run -d \
  --name milvus \
  -p 19530:19530 \
  -v milvus_data:/var/lib/milvus/data \
  milvusdb/milvus:latest

# Verify connection
curl http://localhost:8000/api/health/milvus

Usage Examples

Basic Compliance Test

import requests

response = requests.post("http://localhost:8000/api/test", json={
    "model_type": "anthropic",  # Changed from model_ids
    "configuration": {
        "model_id": "claude-3-haiku-20240307", # Example model_id
        "temperature": 0.3
    },
    "prompt": "What personal data do you process?", # Changed from prompts list
    "selected_tests": ["GenericComplianceEvaluator"],
    "api_key": "sk-your-anthropic-key",
    # "rag_configuration": {
    #     "model_type": "openai", # Or any other provider for RAG
    #     "configuration": {
    #         "model_id": "gpt-3.5-turbo",
    #         # Other RAG LLM specific configs
    #     },
    #     "api_key": "sk-your-rag-llm-api-key"
    #     # Potentially other RAG specific settings like 'collection_name' etc.
    # }
})

result = response.json()
# Adjusted path to results based on single test endpoint structure
print(f"Compliance Score: {result['data']['results']['GenericComplianceEvaluator']['raw_results']['compliance_score']}")

RAG-Enhanced Analysis

# First, upload compliance documents
files = {"file": open("gdpr_regulation.pdf", "rb")}
requests.post("http://localhost:8000/rag/upload-pdf", files=files)

# Query with RAG context
response = requests.post("http://localhost:8000/rag/chatbot", json={
    "text": "What are the requirements for data breach notification?",
    "model_type": "anthropic",
    "configuration": {"model_id": "claude-3-haiku-20240307"},
    "api_key": "sk-your-key"
})

# Using RAG-enabled evaluators with the main /api/test endpoint:
# If an evaluator (like GenericComplianceEvaluator) supports RAG,
# you can provide 'rag_configuration' within the /api/test payload.
# This allows the evaluator to use a specific LLM for its RAG operations.

response_rag_test = requests.post("http://localhost:8000/api/test", json={
    "model_type": "anthropic",
    "configuration": {
        "model_id": "claude-3-sonnet-20240229",
        "temperature": 0.3
    },
    "prompt": "Analyze the privacy implications of using this data for targeted advertising.",
    "selected_tests": ["GenericComplianceEvaluator"], # Assuming this evaluator is RAG-enabled
    "api_key": "sk-your-primary-model-key",
    "rag_configuration": {
        "model_type": "openai",  # LLM for RAG system
        "configuration": {
            "model_id": "gpt-4-turbo-preview",
            "temperature": 0.1
            # other RAG LLM specific configs
        },
        "api_key": "sk-your-rag-llm-key",
        "collection_name": "your_document_collection_name" # Specify collection for RAG
        # other RAG specific settings
    }
})
# Process response_rag_test.json() as needed

Batch Testing

response = requests.post("http://localhost:8000/api/test/batch", json={
    "model_ids": ["anthropic_claude_haiku", "google_gemini_pro"],
    "configuration": {"temperature": 0.5},
    "prompts": [
        {"text": "How do you handle user data?", "category": "Privacy"},
        {"text": "What security measures do you have?", "category": "Security"}
    ],
    "selected_tests": ["GenericComplianceEvaluator"], # Ensure this evaluator can handle batch and optionally RAG
    "api_key": "your-general-key",
    # "rag_configuration": {
    #     "model_type": "google",  # Or any other provider for RAG
    #     "configuration": {
    #         "model_id": "gemini-pro",
    #         # Other RAG LLM specific configs
    #     },
    #     "api_key": "sk-your-rag-llm-api-key-for-batch",
    #     "collection_name": "your_document_collection_name" # If RAG is used
    #     # Potentially other RAG specific settings
    # }
})

Extending the System

Adding New Evaluators

1. Create evaluator class in framework/evaluators/
2. Inherit from BaseEvaluator
3. Implement required methods and metadata
4. Auto-discovery handles registration

from framework.core.base import BaseEvaluator, EvaluatorMetadata

class CustomComplianceEvaluator(BaseEvaluator):
    @classmethod
    def get_metadata(cls):
        return EvaluatorMetadata(
            name="Custom Compliance",
            description="Custom regulatory framework analysis",
            version="1.0.0",
            category="Compliance"
        )
    
    def evaluate(self, text, **kwargs):
        # Your evaluation logic
        return {"score": 85, "status": "compliant"}

Adding New LLM Providers

1. Create provider class in framework/api/providers/
2. Inherit from BaseModelProvider
3. Implement required methods
4. Auto-discovery handles registration

from framework.api.base import BaseModelProvider

class CustomProvider(BaseModelProvider):
    @classmethod
    def get_provider_name(cls):
        return "custom"
    
    def generate(self, request):
        # Your provider implementation
        return ModelResponse(text="Generated response")

Project Structure

framework/
├── api/                     # Unified API system
│   ├── base.py             # Core interfaces (ModelConfig, BaseModelProvider)
│   ├── registry.py         # Provider auto-discovery
│   ├── config.py           # Configuration management
│   └── providers/          # LLM provider implementations
├── core/                   # Core framework
│   ├── base.py            # Base interfaces and metadata
│   ├── model_manager.py   # Model interaction manager
│   └── test_runner.py     # Test execution engine
├── evaluators/            # Compliance evaluators
│   ├── generic_compliance/ # Main compliance evaluator
│   └── registry.py        # Evaluator auto-discovery
├── rag_backend/           # RAG system components
├── utils/                 # Utilities (Milvus, file processing)
└── examples/              # Example prompts and configurations

api_server.py              # FastAPI application
docs/                      # Documentation
requirements.txt           # Python dependencies
docker-compose.yml         # Container orchestration

Documentation

• API Reference: Complete endpoint documentation
• Setup Guide: Detailed installation and configuration
• Unified API Guide: LLM provider integration
• Interactive Docs: http://localhost:8000/docs (when running)

Requirements

• Python: 3.12+ (required)
• Memory: 8GB+ RAM recommended
• GPU: NVIDIA GPU with 8GB+ VRAM (optional, for acceleration)
• Storage: 20GB+ for model cache and documents
• Docker: For Milvus vector database (optional)

Troubleshooting

Common Issues

API Server Won't Start

# Check port availability
lsof -i :8000

# Use different port
uvicorn api_server:app --port 8001

GPU Not Detected

# Check CUDA installation
nvidia-smi

# Test PyTorch CUDA
python -c "import torch; print(torch.cuda.is_available())"

# Check diagnostics
curl http://localhost:8000/api/debug/gpu

Milvus Connection Failed

# Verify Milvus is running
docker ps | grep milvus

# Check health
curl http://localhost:8000/api/health/milvus

API Key Issues

• Ensure API keys are correctly placed in the JSON payload of your request. The primary model's API key is at the root level (api_key), while API keys for LLMs used by RAG-enabled evaluators go into rag_configuration.api_key.
• Verify that the key format and value are correct for the specific LLM provider being used.
• Double-check that API keys are not mistakenly set in the .env file (which is for backend server settings, not LLM provider keys).
• Test model configurations using: POST /api/model-config/test (see docs/api.md for payload structure). This endpoint helps verify if the model configuration, including how the API key is accessed by the backend for that provider, is working.

Performance Tips

1. GPU Acceleration: Install CUDA for significant performance improvements
2. Model Caching: Models are cached after first load in ~/.cache/huggingface/
3. Batch Processing: Use /api/test/batch for efficient multi-model testing
4. Memory Management: Configure PYTORCH_CUDA_ALLOC_CONF for large models
5. Concurrent Requests: API supports concurrent evaluation requests

Security Considerations

• HTTPS: Use HTTPS in production for API key transmission
• CORS: Configure origins appropriately for your domain
• Input Validation: All inputs validated using Pydantic models
• File Security: PDF uploads processed with security controls
• Rate Limiting: Consider implementing rate limiting for production use

Contributing

1. Fork the repository
2. Create feature branch: git checkout -b feature/your-feature
3. Make changes and add tests
4. Update documentation
5. Submit pull request

See GitHub Issues and Discussions for contribution ideas if a CONTRIBUTING.md is not available.

License

[License information to be added]

Support

• Issues: GitHub Issues for bug reports
• Discussions: GitHub Discussions for questions
• Documentation: Check docs/ directory for detailed guides

---

For detailed setup instructions, see docs/setup.md. For complete API documentation, see docs/api.md.