VisionAgent - Professional Multi-Modal AI Agent Framework

A cutting-edge, production-ready AI agent platform for image, video, and face analytics built with modern Python and state-of-the-art AI models.

🚀 Features

Core Capabilities

Face Detection & Recognition - Advanced face detection, encoding, and recognition with facial landmarks
Object Detection - YOLOv8-powered object detection with real-time inference
Video Analysis - Frame-by-frame video processing with object/face tracking
Image Classification - HuggingFace Transformers integration for image classification
Real-time Processing - WebSocket streaming for live video analytics

Technical Excellence

Modular Architecture - Easily extendable agent framework
GPU Acceleration - Automatic CUDA detection with CPU fallback
Async Processing - FastAPI with async endpoints for high performance
Production Ready - Docker support, logging, metrics, and error handling
Type Safety - Full type hints and Pydantic models
Scalable - Batch processing and concurrent request handling

🏗️ Architecture

vision-sphere/
├── agents/                 # AI Agent implementations
│   ├── base_agent.py      # Abstract base class
│   ├── face_agent.py      # Face detection & recognition
│   ├── object_agent.py    # Object detection (YOLOv8)
│   ├── video_agent.py     # Video analysis & tracking
│   └── classification_agent.py  # Image classification
├── models/                # Downloaded/trained models
├── utils/                 # Common utilities
│   └── helpers.py         # Helper functions
├── server.py              # FastAPI application
├── config.py              # Configuration management
├── cli.py                 # Command-line interface
├── requirements.txt       # Python dependencies
└── Dockerfile            # Container deployment

🛠️ Installation

Prerequisites

Python 3.11+
CUDA 11.8+ (optional, for GPU acceleration)
8GB+ RAM (16GB+ recommended for video processing)

Quick Start

Clone and Setup

git clone <repository-url>
cd vision-sphere
python -m venv venv
source venv/bin/activate  # On Windows: venv\Scripts\activate
pip install -r requirements.txt

Run API Server
```
python server.py
```
Access API Documentation
- Open http://localhost:8000/docs for interactive API docs
- Or http://localhost:8000/redoc for alternative documentation

Docker Deployment

# Build image
docker build -t visionagent .

# Run with GPU support
docker run --gpus all -p 8000:8000 visionagent

# Run CPU-only
docker run -p 8000:8000 visionagent

📖 Usage

Command Line Interface

# Face detection
python cli.py face image.jpg --output results.json

# Object detection  
python cli.py object image.jpg --confidence 0.7 --verbose

# Video analysis
python cli.py video video.mp4 --max-frames 500 --format detailed

# Image classification
python cli.py classify image.jpg --top-k 10 --confidence 0.1

# System information
python cli.py info

# Start server
python cli.py server --host 0.0.0.0 --port 8000

API Endpoints

Face Detection

# Upload file
curl -X POST "http://localhost:8000/face" \
     -F "file=@image.jpg"

# Or use image URL
curl -X POST "http://localhost:8000/face" \
     -H "Content-Type: application/json" \
     -d '{"image_url": "https://example.com/image.jpg"}'

Object Detection

curl -X POST "http://localhost:8000/object" \
     -F "file=@image.jpg"

Video Analysis

curl -X POST "http://localhost:8000/video" \
     -F "file=@video.mp4"

Image Classification

curl -X POST "http://localhost:8000/classify" \
     -F "file=@image.jpg"

Batch Processing

curl -X POST "http://localhost:8000/batch/classify" \
     -F "files=@image1.jpg" \
     -F "files=@image2.jpg" \
     -F "files=@image3.jpg"

WebSocket Streaming

// Real-time video processing
const ws = new WebSocket('ws://localhost:8000/ws/video');

ws.onopen = function() {
    // Send video frames as binary data
    ws.send(frameData);
};

ws.onmessage = function(event) {
    const result = JSON.parse(event.data);
    console.log('Analysis result:', result);
};

⚙️ Configuration

Create a config.yaml file to customize the framework:

# Global settings
default_device: "auto"  # auto, cpu, cuda
model_cache_dir: "./models"
temp_dir: "./temp"

# Face Agent
face_agent:
  enabled: true
  model:
    name: "face_recognition"
    confidence_threshold: 0.6
    custom_params:
      face_detection_model: "hog"  # hog, cnn
      num_jitters: 1
      tolerance: 0.6

# Object Agent
object_agent:
  enabled: true
  model:
    name: "yolov8s.pt"
    confidence_threshold: 0.5
    custom_params:
      iou_threshold: 0.45
      max_detections: 100

# Video Agent
video_agent:
  enabled: true
  processing_params:
    frame_skip: 1
    max_frames: 1000
    track_objects: true
    track_faces: true

# Classification Agent
classification_agent:
  enabled: true
  model:
    name: "microsoft/resnet-50"
    custom_params:
      top_k: 5
      threshold: 0.1
      return_features: false

# Server Configuration
server:
  host: "0.0.0.0"
  port: 8000
  workers: 1
  max_file_size_mb: 100
  enable_websocket: true
  rate_limit_per_minute: 60

# Logging
logging:
  level: "INFO"
  file_path: "./logs/visionagent.log"
  max_file_size_mb: 10
  backup_count: 5

Environment Variables

# Override configuration with environment variables
export VISIONAGENT_CONFIG=/path/to/config.yaml
export VISIONAGENT_DEVICE=cuda
export VISIONAGENT_HOST=0.0.0.0
export VISIONAGENT_PORT=8000
export VISIONAGENT_LOG_LEVEL=DEBUG
export VISIONAGENT_MODEL_CACHE_DIR=/app/models

🧩 Extending the Framework

Creating Custom Agents

from agents.base_agent import BaseAgent, ProcessingResult

class CustomAgent(BaseAgent):
    def initialize(self) -> bool:
        # Initialize your model here
        self._is_initialized = True
        return True
    
    def process(self, input_data: Any) -> ProcessingResult:
        # Implement your processing logic
        try:
            # Your processing code here
            result_data = {"custom_analysis": "results"}
            
            return ProcessingResult(
                success=True,
                data=result_data,
                confidence=0.95,
                inference_time=50.0
            )
        except Exception as e:
            return ProcessingResult(
                success=False,
                data={},
                error=str(e)
            )

📊 API Response Format

All endpoints return standardized responses:

{
  "success": true,
  "data": {
    "detections": [...],
    "detection_count": 5,
    "class_summary": {...}
  },
  "inference_time_ms": 45.2,
  "agent_info": {
    "agent_type": "ObjectAgent",
    "device": "cuda",
    "initialized": true
  },
  "timestamp": "2025-08-31T12:00:00.000Z",
  "request_id": "uuid-string"
}

🔧 Development

Setup Development Environment

# Install development dependencies
pip install -r requirements.txt
pip install pytest pytest-asyncio black flake8 mypy

# Run tests
pytest

# Format code
black .

# Lint code
flake8 .
mypy .

Project Structure Guidelines

agents/ - All AI agent implementations inherit from BaseAgent
models/ - Downloaded model files and weights
utils/ - Shared utilities and helper functions
server.py - FastAPI application with all endpoints
config.py - Centralized configuration management
cli.py - Command-line interface for all agents

🚀 Production Deployment

sDocker Deployment

# Build production image
docker build -t visionagent:latest .

# Run with GPU support
docker run --gpus all \
  -p 8000:8000 \
  -v $(pwd)/models:/app/models \
  -v $(pwd)/logs:/app/logs \
  -e VISIONAGENT_LOG_LEVEL=INFO \
  visionagent:latest

# Docker Compose (recommended)
docker-compose up -d

Kubernetes Deployment

apiVersion: apps/v1
kind: Deployment
metadata:
  name: visionagent
spec:
  replicas: 3
  selector:
    matchLabels:
      app: visionagent
  template:
    metadata:
      labels:
        app: visionagent
    spec:
      containers:
      - name: visionagent
        image: visionagent:latest
        ports:
        - containerPort: 8000
        env:
        - name: VISIONAGENT_DEVICE
          value: "cuda"
        resources:
          requests:
            nvidia.com/gpu: 1
          limits:
            nvidia.com/gpu: 1

📈 Performance Optimization

GPU Acceleration

Automatic CUDA detection and device selection
Batch processing for multiple images
Memory-efficient model loading

Scalability Features

Async FastAPI endpoints
WebSocket streaming for real-time processing
Configurable worker processes
Model caching and lazy loading

🔒 Security Considerations

File size limits for uploads
Input validation and sanitization
Non-root container execution
Rate limiting support
CORS configuration

🧪 Testing

# Run all tests
pytest

# Run specific test categories
pytest tests/test_agents.py
pytest tests/test_api.py
pytest tests/test_utils.py

# Run with coverage
pytest --cov=agents --cov=utils --cov-report=html

📝 License

This project is licensed under the MIT License - see the LICENSE file for details.

🤝 Contributing

Fork the repository
Create a feature branch (git checkout -b feature/amazing-feature)
Commit your changes (git commit -m 'Add amazing feature')
Push to the branch (git push origin feature/amazing-feature)
Open a Pull Request

🆘 Support

For issues and questions:

Check the API documentation
Review the [configuration guide]([def]: #configuration)
Check system requirements and GPU setup
Enable debug logging for detailed error information

🎯 Roadmap

ONNX model support for cross-platform deployment
Advanced video tracking algorithms
Real-time face recognition optimization
Model quantization for edge deployment
Multi-camera support
Advanced analytics and reporting
Model fine-tuning utilities
REST API rate limiting and authentication

Name		Name	Last commit message	Last commit date
Latest commit History 2 Commits
.github		.github
cache/semantic		cache/semantic
dist		dist
models		models
vision_agent		vision_agent
vision_agent_framework.egg-info		vision_agent_framework.egg-info
Dockerfile		Dockerfile
README.md		README.md
cli.py		cli.py
config.py		config.py
config.yaml		config.yaml
enterprise_demo.py		enterprise_demo.py
focused_demo.py		focused_demo.py
launch.py		launch.py
requirements.txt		requirements.txt
setup.py		setup.py
yolov8s.pt		yolov8s.pt

Folders and files

Latest commit

History

Repository files navigation