Multi-Agent Research Orchestrator

Production-grade AI research orchestration system that coordinates specialized LangGraph agents to produce verified research reports with citations, source tracking, confidence scoring, and structured output.

Overview

Complex research questions often require dozens of source checks, cross-source verification, and careful synthesis. A single LLM pass is brittle for that workflow because it compresses search, extraction, verification, and writing into one opaque step.

Multi-Agent Research Orchestrator decomposes research into a LangGraph workflow with specialized agents:

Agent	Responsibility
Search Agent	Builds search strategies and retrieves sources via Tavily API
Extraction Agent	Extracts atomic evidence claims from retrieved sources
Verification Agent	Validates claims, rejects weak evidence, calibrates confidence
Synthesis Agent	Produces final report with citations and structured output
Critique Agent	Stress-tests report, routes through another evidence loop if confidence is low

Target Users: Founders, consultants, analysts, researchers, and journalists who need research that can survive scrutiny.

Features

Multi-Agent Pipeline — 5 specialized LangGraph agents working in concert
Source Verification — Automatic credibility scoring and evidence validation
Confidence Scoring — Quantified confidence based on source diversity and evidence strength
Citation Tracking — Full source attribution with URLs and credibility metrics
Structured Output — Both Markdown reports and JSON structured data
Async Execution — Non-blocking research runs with status polling
REST API — FastAPI with OpenAPI documentation
PostgreSQL Persistence — Durable storage for runs, sources, evidence, and reports
Prometheus Metrics — Built-in observability for production monitoring
Deterministic Mode — Test-friendly mode that doesn't call external APIs
Docker Compose — One-command local development stack

Screenshots

Search Agent — builds search strategies and retrieves sources

Extraction Agent — extracts atomic evidence claims from sources

Verification Agent — validates claims and calibrates confidence

Synthesis Agent — produces final report with citations

Architecture

flowchart LR
    Client[Client] --> API[FastAPI API]
    API --> Service[Research Service]
    Service --> Graph[LangGraph Workflow]
    
    Graph --> Search[Search Agent]
    Search --> Tavily[Tavily API]
    
    Graph --> Extract[Extraction Agent]
    Graph --> Verify[Verification Agent]
    Graph --> Synthesize[Synthesis Agent]
    Graph --> Critique[Critique Agent]
    
    Extract --> LLMs[OpenAI or Anthropic]
    Verify --> LLMs
    Synthesize --> LLMs
    
    Service --> Postgres[(PostgreSQL)]
    API --> Metrics[Prometheus Metrics]

Data Flow

Client sends research query via REST API
Research Service creates a run and initiates the LangGraph workflow
Search Agent retrieves relevant sources from Tavily
Extraction Agent extracts factual claims from each source
Verification Agent validates evidence and scores credibility
Synthesis Agent generates a citation-backed report
Critique Agent evaluates quality; loops back if confidence is below threshold
Report is persisted to PostgreSQL and returned to client

Quick Start

Prerequisites

Python 3.11+
Docker & Docker Compose
API keys for OpenAI, Anthropic, and Tavily (optional — deterministic mode works without them)

1. Clone and Configure

git clone https://github.com/Sneh30/multi-agent-research-orchestrator.git
cd multi-agent-research-orchestrator

cp .env.example .env
# Optional: edit .env with real API keys for live search/LLM
# Works out of the box with LLM_PROVIDER=deterministic (no keys needed)

2. Start the Stack

docker compose up --build

3. Access the Application

Frontend UI: http://localhost:8000/app
Swagger UI: http://localhost:8000/docs
Health Check: http://localhost:8000/health

4. Create a Research Run

curl -X POST http://localhost:8000/v1/research-runs \
  -H "Content-Type: application/json" \
  -H "X-API-Key: local-dev-key" \
  -d '{
    "query": "What evidence supports enterprise adoption of AI agents in regulated industries?",
    "objective": "Produce a board-ready diligence memo.",
    "constraints": {"audience": "founders", "prefer_primary_sources": true},
    "depth": "advanced",
    "max_sources": 12,
    "min_confidence": 0.72,
    "execute_async": true
  }'

5. Get the Report

curl http://localhost:8000/v1/research-runs/{run_id}/report \
  -H "X-API-Key: local-dev-key"

API Reference

Endpoints

Method	Endpoint	Description
`GET`	`/app`	Frontend UI (no auth required)
`GET`	`/health`	Health check
`POST`	`/v1/research-runs`	Create a new research run
`GET`	`/v1/research-runs`	List all research runs
`GET`	`/v1/research-runs/{id}`	Get a specific research run
`POST`	`/v1/research-runs/{id}/execute`	Execute a research run synchronously
`GET`	`/v1/research-runs/{id}/report`	Get the research report
`GET`	`/v1/research-runs/{id}/sources`	List sources for a run
`POST`	`/v1/evaluations/report`	Evaluate a report payload
`POST`	`/v1/evaluations/runs/{id}`	Evaluate a completed run

Authentication

All API endpoints require an API key via the X-API-Key header:

-H "X-API-Key: your-api-key"

Public endpoints (no auth required): /app, /docs, /metrics, /openapi.json

Request Example

{
  "query": "What are the risks of using vector databases in production?",
  "objective": "Identify technical and business risks for due diligence.",
  "constraints": {
    "audience": "technical founders",
    "prefer_primary_sources": true
  },
  "depth": "advanced",
  "max_sources": 15,
  "min_confidence": 0.8,
  "execute_async": true
}

Response Example

{
  "id": "550e8400-e29b-41d4-a716-446655440000",
  "status": "completed",
  "query": "What are the risks of using vector databases in production?",
  "confidence_score": 0.85,
  "created_at": "2024-01-15T10:30:00Z",
  "completed_at": "2024-01-15T10:31:25Z"
}

Project Structure

multi-agent-research-orchestrator/
├── backend/
│   └── research_orchestrator/
│       ├── agents/          # LangGraph agents, prompts, tools, scoring
│       ├── api/             # FastAPI routes, schemas, dependencies
│       ├── core/            # Config, logging, security, exceptions
│       ├── database/        # SQLAlchemy models and repositories
│       ├── evaluation/      # Evaluation metrics and benchmarks
│       ├── services/        # Business logic and orchestration
│       └── main.py          # FastAPI application factory
├── frontend/
│   └── index.html           # Single-file dark-themed research UI
├── database/
│   └── migrations/          # PostgreSQL migration scripts
├── docs/                    # Comprehensive documentation
├── infrastructure/          # Monitoring and proxy config
├── tests/
│   ├── unit/                # Unit tests
│   ├── integration/         # Integration tests
│   └── e2e/                 # End-to-end contract tests
├── .env.example             # Environment template
├── docker-compose.yml       # Local development stack
├── Dockerfile               # Container build
├── pyproject.toml           # Python project config
├── CONTRIBUTING.md          # Contribution guidelines
├── LICENSE                  # MIT License
├── README.md                # This file
└── SECURITY.md              # Security policy

Development

Local Development (without Docker)

# Create virtual environment
python -m venv .venv
source .venv/bin/activate  # On Windows: .venv\Scripts\activate

# Install dependencies
pip install -e ".[dev]"

# Start PostgreSQL (or use Docker for just the database)
docker compose up postgres -d

# Set deterministic mode for testing
export LLM_PROVIDER=deterministic

# Run the API
uvicorn research_orchestrator.main:app --reload --app-dir backend

Code Quality

# Linting
ruff check backend tests

# Type checking
mypy backend

# Formatting
ruff format backend tests

# Run all checks
ruff check backend tests && mypy backend && pytest

Environment Variables

Variable	Description	Default
`APP_ENV`	Environment (local/test/staging/production)	`local`
`LOG_LEVEL`	Logging level	`INFO`
`API_KEY`	API authentication key	`local-dev-key`
`DATABASE_URL`	PostgreSQL connection string	`postgresql+asyncpg://...`
`OPENAI_API_KEY`	OpenAI API key (optional in deterministic mode)	—
`ANTHROPIC_API_KEY`	Anthropic API key (optional in deterministic mode)	—
`TAVILY_API_KEY`	Tavily search API key (optional in deterministic mode)	—
`LLM_PROVIDER`	LLM provider (openai/anthropic/deterministic)	`openai`
`OPENAI_MODEL`	OpenAI model name	`gpt-4.1-mini`
`ANTHROPIC_MODEL`	Anthropic model name	`claude-3-5-sonnet-latest`
`MAX_GRAPH_ITERATIONS`	Max agent loop iterations	`3`
`DEFAULT_MAX_SOURCES`	Default max sources to retrieve	`12`
`DEFAULT_MIN_CONFIDENCE`	Default minimum confidence threshold	`0.72`

Testing

Run All Tests

export LLM_PROVIDER=deterministic
pytest

Run with Coverage

pytest --cov=research_orchestrator --cov-report=term-missing

Test Categories

Category	Location	Description
Unit	`tests/unit/`	Agent logic, scoring, evaluation metrics
Integration	`tests/integration/`	API endpoints with FastAPI TestClient
E2E	`tests/e2e/`	Research workflow contract tests

Test Architecture

The test suite uses a deterministic provider pattern:

DeterministicLLMProvider — Echoes prompts instead of calling real APIs
FakeSearchTool — Returns hardcoded source data
LLM_PROVIDER=deterministic — Disables all external API calls

This allows the entire pipeline to run in CI without API keys.

Deployment

Docker Production Build

# Build production image
docker build -t research-orchestrator:latest .

# Run with environment variables
docker run -p 8000:8000 \
  -e DATABASE_URL=postgresql+asyncpg://... \
  -e OPENAI_API_KEY=... \
  -e API_KEY=your-secure-api-key \
  research-orchestrator:latest

Environment-Specific Configuration

Environment	`APP_ENV`	`LLM_PROVIDER`	Notes
Local Dev	`local`	`openai` or `deterministic`	Docker Compose
Testing	`test`	`deterministic`	CI pipeline
Staging	`staging`	`openai` or `anthropic`	Cloud deployment
Production	`production`	`openai` or `anthropic`	Hardened config

Production Checklist

Documentation

Document	Description
Product Foundation	Product requirements and goals
Architecture	System design and data flow
API Design	API specification and examples
Database Design	Schema and data models
LangGraph Design	Agent workflow design
AI Layer	LLM integration details
Testing	Testing strategy and guidelines
Infrastructure	Deployment and monitoring
Installation Guide	Detailed setup instructions
Developer Guide	Development workflow
User Guide	End-user documentation
API Documentation	Complete API reference

Contributing

We welcome contributions! Please see our Contributing Guidelines for details.

Quick Start for Contributors

Fork the repository
Create a feature branch: git checkout -b feature/amazing-feature
Make your changes
Run tests: pytest
Run linting: ruff check backend tests
Commit: git commit -m "feat: add amazing feature"
Push: git push origin feature/amazing-feature
Open a Pull Request

License

This project is licensed under the MIT License — see the LICENSE file for details.

Acknowledgments

LangChain — LLM application framework
LangGraph — Agent orchestration
FastAPI — Modern Python web framework
Tavily — Search API for AI agents
PostgreSQL — Reliable open-source database
Prometheus — Monitoring and alerting

Built with care for production-grade AI systems

Name		Name	Last commit message	Last commit date
Latest commit History 10 Commits
.github/workflows		.github/workflows
Orchestra		Orchestra
.env.example		.env.example
.gitignore		.gitignore
README.md		README.md

Folders and files

Latest commit

History

Repository files navigation