MAGI Agent System

Multi-Agent General Intelligence system built with microservices architecture for workflow orchestration, RAG (Retrieval-Augmented Generation), and GitOps automation.

Architecture

Services

Service	Port	Technology	Description
API Gateway	8000	Node.js/Express	Authentication, rate limiting, reverse proxy
Auth Service	8001	Node.js/Express	JWT authentication with JWKS
RAG Ingestion	8002	Python/FastAPI	Source registration, GitHub/Confluence connectors
RAG Indexer	8003	Python/FastAPI	Text chunking, embeddings, Qdrant indexing
RAG Query	8004	Python/FastAPI	Vector search, reranking, LLM synthesis
Workflow Orchestrator	8005	Python/FastAPI	LangGraph workflows, SSE streaming
MAGI Agents	8006	Python/FastAPI	Multi-agent consensus (AutoGen)
GitOps PR Automation	8007	Python/FastAPI	GitHub App PR creation and merge automation
Workflow UI	3001	React/Vite	ReactFlow workflow editor

Infrastructure

Component	Port(s)	Description
PostgreSQL	5432	Primary database for RAG metadata
Redis	6379	Celery broker, caching
Qdrant	6333, 6334	Vector database for embeddings
MinIO	9000, 9001	S3-compatible object storage
Jaeger	16686, 14268, 4317	Distributed tracing
OpenTelemetry Collector	4319, 8888, 8889	Trace/metrics collection
Prometheus	9090	Metrics storage and querying
Grafana	3000	Observability dashboards

Quick Start

Prerequisites

• Docker & Docker Compose
• OpenAI API key (required)
• GitHub App credentials (for GitOps automation)
• Optional: Anthropic, Cohere API keys for additional LLM providers

Setup

1. Clone the repository

   git clone <repository-url>
   cd MAGI-Agent

2. Configure environment variables

   cp .env.example .env
   # Edit .env with your API keys and configuration

   Required variables:

   • OPENAI_API_KEY: OpenAI API key for LLM operations
   • POSTGRES_PASSWORD: Database password
   • JWT_SECRET: Secure random string for JWT signing
   • GITHUB_APP_ID, GITHUB_INSTALLATION_ID, GITHUB_PRIVATE_KEY_PATH: For GitOps automation
   • GIT_ORG: GitHub organization name

3. Start all services

   docker-compose up -d

4. Verify services are running

   docker-compose ps

5. Access the applications

   • Workflow UI: http://localhost:3001
   • API Gateway: http://localhost:8000
   • Grafana: http://localhost:3000 (admin/admin)
   • Prometheus: http://localhost:9090
   • Jaeger UI: http://localhost:16686
   • MinIO Console: http://localhost:9001

Service Details

RAG Pipeline

Ingestion → Indexing → Query

1. RAG Ingestion (POST /api/sources): Register GitHub repos or Confluence spaces
2. RAG Indexer: Automatic chunking and embedding generation
3. RAG Query (POST /api/query): Semantic search with LLM synthesis

MAGI Multi-Agent Consensus

Three specialized agents (Balthazar, Melchior, Casper) debate and reach consensus on complex decisions using AutoGen framework.

Endpoint: POST /api/consensus

{
  "question": "Should we implement feature X?",
  "context": "Additional context...",
  "max_rounds": 5
}

Workflow Orchestration

LangGraph-based workflow execution with SSE streaming for real-time updates.

Endpoints:

• POST /api/workflows/execute: Start workflow
• GET /api/workflows/stream/{workflow_id}: SSE stream

GitOps PR Automation

Automated pull request creation, status tracking, and merge operations.

Endpoints:

• POST /api/gitops/pr/create: Create PR
• GET /api/gitops/pr/status/{repo_name}/{pr_number}: Check PR status
• POST /api/gitops/pr/merge: Auto-merge PR after checks pass

Development

Service-Specific Development

Each service can be developed independently:

cd services/<service-name>
cp .env.example .env
# Edit .env with required configuration

# For Python services:
pip install -r requirements.txt
python src/main.py

# For Node.js services:
npm install
npm start

OpenTelemetry Instrumentation

All services are instrumented with OpenTelemetry for distributed tracing and metrics:

• Traces exported to Jaeger via OTLP Collector
• Metrics scraped by Prometheus
• Dashboards available in Grafana

Database Migrations

For services using PostgreSQL (RAG Ingestion, RAG Indexer):

# Migrations are handled via SQLAlchemy models
# Tables are created automatically on first run

Monitoring & Observability

Grafana Dashboards

Access Grafana at http://localhost:3000 (admin/admin)

Pre-configured data sources:

• Prometheus (metrics)
• Jaeger (traces)

Prometheus Metrics

All services expose /metrics endpoints:

• API Gateway: Request rates, latencies, rate limit hits
• RAG Services: Query performance, indexing throughput
• MAGI Agents: Consensus rounds, agent response times

Jaeger Traces

Distributed traces show request flow across services:

1. API Gateway → Auth validation
2. API Gateway → Downstream service
3. Service → Database/Vector DB
4. Service → External APIs (OpenAI, GitHub)

Security

• Secrets: All secrets configured via environment variables only
• JWT Authentication: Auth service issues tokens validated by API Gateway
• Rate Limiting: Configured on API Gateway (100 req/min default)
• GitHub App: Uses private key authentication for PR automation
• CORS: Configured for frontend origin

API Documentation

Each service exposes OpenAPI (Swagger) docs at /docs:

• API Gateway: http://localhost:8000/docs
• RAG Ingestion: http://localhost:8002/docs
• RAG Indexer: http://localhost:8003/docs
• RAG Query: http://localhost:8004/docs
• Workflow Orchestrator: http://localhost:8005/docs
• MAGI Agents: http://localhost:8006/docs
• GitOps PR Automation: http://localhost:8007/docs

Troubleshooting

Services Not Starting

# Check service logs
docker-compose logs <service-name>

# Restart specific service
docker-compose restart <service-name>

Database Connection Errors

Ensure PostgreSQL is healthy:

docker-compose ps postgres
docker-compose logs postgres

Vector Search Not Working

Check Qdrant health:

curl http://localhost:6333/healthz
docker-compose logs qdrant

GitHub App Authentication

Verify GitHub App credentials in .env:

• GITHUB_APP_ID: Numeric app ID
• GITHUB_INSTALLATION_ID: Installation ID for your org
• GITHUB_PRIVATE_KEY_PATH: Path to .pem file (mount as volume)

Contributing

See individual service PRDs in prompts/workflows/prd/ for detailed specifications.

License

[Your License Here]