Skip to content

Folders and files

NameName
Last commit message
Last commit date

Latest commit

 

History

3 Commits
 
 
 
 
 
 
 
 
 
 
 
 
 
 

Repository files navigation

CriticLoopAI

A multi-agent AI improvement system where three specialized agents Generator, Critic, and Improver work in an iterative loop to progressively enhance any text-based task. Features convergence detection, pattern tracking, and support for 20+ LLM providers including Ollama, Groq, DeepSeek, OpenAI, and Anthropic.


Table of Contents


What This System Does

This system solves a fundamental problem: a single AI response is rarely the best possible response.

When you ask an AI to write something, generate code, or solve a problem, it gives you one answer. That answer might be good, but it's not optimal. It might have gaps, weaknesses, or missed opportunities.

This system takes that first answer and makes it better — automatically, repeatedly, and intelligently.

Here's what happens when you run it:

  1. You give it a task (e.g., "Write a guide about Python decorators")
  2. The Generator Agent creates an initial response
  3. The Critic Agent reviews it like an expert editor — finding weaknesses, scoring it on 6 dimensions, and listing specific improvements
  4. The Improver Agent analyzes all the feedback, identifies patterns, and creates a strategic improvement plan
  5. The Generator creates an improved version based on all that feedback
  6. The Critic reviews the new version
  7. This cycle repeats until the output reaches the quality target you set

The result: an output that has been reviewed, critiqued, and improved by multiple AI experts — producing something significantly better than any single AI pass could achieve.


Why It Exists — The Problem It Solves

When you use a single AI model, you face these limitations:

Problem Without CriticLoopAI With CriticLoopAI
Single perspective One AI's view, with its biases and blind spots Multiple AI perspectives — Generator, Critic, Improver each bring different strengths
No quality gate You get what you get, no verification Every output is scored, critiqued, and verified before delivery
No iteration One-shot generation, manual re-prompting Automatic iterative improvement with smart stopping
No pattern tracking You forget what worked and what didn't Full history of every iteration, score, critique, and improvement tracked
Inconsistent quality Output varies wildly between runs Quality converges to a consistent high standard through repeated refinement
Manual feedback loop You have to manually review and re-prompt The AI reviews and improves itself — you just watch the scores climb

In short: This system turns AI from a "one-shot tool" into a self-improving quality machine.


Who Benefits From This System

For End Users

If you use AI for writing, research, content creation, or any text-based task, this system gives you:

1. Better Results, Automatically

  • Instead of getting one AI response and hoping it's good, you get a response that has been iteratively improved 5+ times
  • Each iteration catches issues the previous one missed
  • The final output is scored and verified — you know exactly how good it is

2. No Manual Re-Prompting

  • Instead of copying the AI's response, finding issues, and re-prompting ("make it more detailed", "add examples", "fix the structure"), the system does this automatically
  • The Critic identifies exactly what needs to change
  • The Generator implements those changes precisely

3. Transparency

  • You can see every iteration — what was generated, what the Critic found, what improved, and what didn't
  • The scoring system (0.0 to 1.0) gives you a concrete quality metric
  • You know exactly when to stop — when the score hits your target or when improvements plateau

4. Works With Any AI Provider

  • Use Ollama (local, free), Groq (free tier), DeepSeek (cheap), OpenAI, Anthropic, or any provider
  • No vendor lock-in — switch providers with one flag
  • Local providers mean your data never leaves your machine

5. Saves Time on Complex Tasks

  • For tasks that need high quality (blog posts, documentation, code reviews, analysis), the system produces polished output without manual refinement
  • Set it to run and come back to a finished, scored result

Example scenarios where end users benefit:

Task What the system does
Writing a blog post Generator writes draft -> Critic finds weak sections -> Improver plans fixes -> Generator rewrites -> Final: polished post
Code review Generator reviews code -> Critic finds security issues -> Improver prioritizes fixes -> Final: comprehensive review
Technical documentation Generator drafts docs -> Critic finds gaps -> Improver plans additions -> Final: complete documentation
Marketing copy Generator writes ad copy -> Critic evaluates persuasion -> Improver refines messaging -> Final: high-converting copy
Research summary Generator summarizes paper -> Critic checks accuracy -> Improver fills gaps -> Final: accurate summary

For Developers

If you're building AI-powered applications, this system provides:

1. A Production-Ready Multi-Agent Framework

  • Clean, modular Python architecture
  • BaseAgent class handles all LLM communication, retries, and error handling
  • Add new agent types by extending BaseAgent — implement get_system_prompt() and you're done
  • Async throughout — built on httpx for non-blocking API calls

2. Provider-Agnostic Design

  • Any OpenAI-compatible API works out of the box — Ollama, LM Studio, vLLM, Groq, DeepSeek, Fireworks, Together, OpenRouter, and 15+ more
  • Built-in Anthropic support (separate API format handled internally)
  • Model discovery via /v1/models endpoint — know exactly what's available before you run
  • 20+ provider presets with verified URLs and signup links

3. Intelligent Loop Management

  • Convergence detection — stops when quality plateaus
  • Early stopping — no wasted iterations when improvements stall
  • Diminishing returns detection — the Improver predicts when further iterations won't help
  • Configurable patience, thresholds, and targets

4. Pattern Tracking and Analysis

  • KnowledgeStore tracks every iteration's output, critique, improvements, and score
  • Extracts recurring issues from critique text automatically
  • Calculates improvement trends, consistency, and effectiveness
  • Persists full session history to JSON for analysis

5. Easy Integration

  • Use as a CLI tool: python main.py "your task"
  • Import as a library: from core.orchestrator import LoopOrchestrator
  • Configuration via dataclasses, environment variables, or CLI flags
  • All results saved as JSON — easy to parse, analyze, or display

6. Extensible Architecture

  • Add new agents: extend BaseAgent, implement system prompt
  • Add new providers: add API format handler in base_agent.py
  • Add new evaluation criteria: modify CriticAgent prompts
  • Add new stopping conditions: modify LoopConfig and orchestrator.py

Developer-friendly patterns used:

  • Dataclass-based configuration (typed, validated, composable)
  • Abstract base class for agents (clean inheritance)
  • Async/await throughout (non-blocking, scalable)
  • JSON-structured LLM outputs (parseable, fallback-handled)
  • Session-based storage (isolated, timestamped, auditable)

How It Works — The Core Loop

    You give a task
          |
          v
    +------------------+
    |    GENERATOR     |  Creates initial output
    | (Creativity: 0.8)|
    +------------------+
          |
          v
    +------------------+
    |     CRITIC       |  Evaluates on 6 criteria
    | (Precision: 0.3) |  Returns: critique + improvements + score
    +------------------+
          |
          v
    +------------------+
    |    IMPROVER      |  Analyzes patterns across all iterations
    |  (Balance: 0.5)  |  Creates strategic improvement plan
    +------------------+
          |
          v
    +------------------+
    |    GENERATOR     |  Creates improved version
    +------------------+
          |
          v
    Score >= target?  ----YES----> DONE: Final polished output
          |
         NO
          |
          v
    No improvement for 2 iterations?  ----YES----> DONE: Best output so far
          |
         NO
          |
          v
    Back to CRITIC (next iteration)

The Critic evaluates on 6 dimensions:

Dimension What it measures
Accuracy Is the information correct and factual?
Completeness Does it fully address the task?
Clarity Is it well-organized and easy to understand?
Depth Does it provide sufficient detail?
Creativity Does it offer unique insights or approaches?
Usability Is it practical and actionable?

Each dimension is scored 0.0-1.0. The overall score is a weighted combination. This means you get a detailed quality breakdown, not just a single number.


Real-World Use Cases

For Content Creators

# Write a professional blog post
python main.py "Write a 1500-word blog post about the future of AI in healthcare" \
    --provider groq --api-key gsk_xxx \
    --model llama-4-maverick \
    --max-iterations 6 --target-score 0.93

What happens: Generator writes draft -> Critic finds weak arguments -> Improver plans evidence additions -> Generator rewrites with evidence -> Critic confirms improvement -> Final: publish-ready post

For Developers

# Generate production-quality code
python main.py "Write a Python function that implements rate limiting with sliding window algorithm" \
    --requirements "Include type hints" \
    --requirements "Add docstring" \
    --requirements "Handle edge cases" \
    --provider deepseek --api-key sk-xxx --model deepseek-chat

What happens: Generator writes code -> Critic finds missing edge cases -> Improver prioritizes fixes -> Generator adds handling -> Final: production-ready code

For Researchers

# Create a comprehensive literature review section
python main.py "Write a literature review section on transformer architectures for NLP" \
    --requirements "Cite key papers" \
    --requirements "Compare approaches" \
    --requirements "Identify research gaps" \
    --provider openai --api-key sk-xxx --model gpt-5.6

What happens: Generator drafts review -> Critic finds missing citations -> Improver plans additions -> Generator adds citations and comparisons -> Final: thorough review

For Business

# Create marketing materials
python main.py "Write a product launch email for a new SaaS tool that helps teams collaborate remotely" \
    --requirements "Professional tone" \
    --requirements "Include social proof" \
    --requirements "Strong CTA" \
    --max-iterations 4

What happens: Generator writes email -> Critic evaluates persuasion elements -> Improver refines messaging -> Final: high-converting email


Installation

Requirements: Python 3.9+

git clone https://github.com/v2Talal/CriticLoopAI.git
cd CriticLoopAI
pip install -r requirements.txt

Dependencies:

  • httpx (>=0.27.0) — async HTTP client for API calls
  • pytest (>=7.0.0) — testing framework
  • pytest-asyncio (>=0.21.0) — async test support

No API keys needed for local providers (Ollama, LM Studio, vLLM).


Quick Start

Option 1: Local, Free (Ollama)

# Install Ollama: https://ollama.com
ollama pull llama-4-maverick
python main.py "Write a guide about Python decorators"

The system auto-detects Ollama, lists your models, and lets you pick one.

Option 2: Free Cloud (Groq)

# Get free API key: https://console.groq.com
python main.py "Write a guide about Docker" \
    --provider groq --api-key gsk_your_key_here \
    --model llama-4-maverick

Option 3: Any Provider

python main.py "Your task" \
    --base-url https://your-provider.com/v1 \
    --api-key your_key --model model-name

CLI Reference

python main.py [task] [options]

Options

Flag Short Default Description
--requirements -r Add requirements. Repeatable: -r "use examples" -r "be concise"
--initial-output -i Start from your own output (skip initial generation)
--target-score -t 0.95 Quality target (0.0-1.0). Loop stops when reached.
--max-iterations -m 5 Maximum improvement iterations
--provider -p custom Provider: ollama, groq, openai, deepseek, anthropic, etc.
--model Model name. Auto-detected for local providers.
--api-key -k not-needed API key (not needed for local)
--base-url -b Custom API endpoint URL
--output-dir -o storage Where to save results
--quiet -q off Minimal output
--discover -d off Interactively select a model before running
--list-models -M off List available models (no loop)
--list-presets -l off List all provider presets

Provider Setup

Local Providers (Free)

Run on your machine. No API key. No internet needed.

Provider Preset URL Install
Ollama ollama localhost:11434/v1 ollama.com
LM Studio lmstudio localhost:1234/v1 lmstudio.ai
vLLM vllm localhost:8000/v1 github.com/vllm-project/vllm
Text Gen WebUI textgen localhost:5000/v1 oobabooga/text-generation-webui
KoboldCpp kobold localhost:5001/v1 LostRuins/koboldcpp

Cloud - Free Tier

Provider Preset URL Details
Groq groq api.groq.com/openai/v1 Free tier, rate-limited. Very fast.
Cerebras cerebras api.cerebras.ai/v1 $5 free credits on signup
SambaNova sambanova api.sambanova.ai/v1 Free tier available
OpenRouter openrouter openrouter.ai/api/v1 25+ free models. 5.5% fee on paid.

Cloud - Paid

Provider Preset URL
DeepSeek deepseek api.deepseek.com/v1
OpenAI openai api.openai.com/v1
Fireworks AI fireworks api.fireworks.ai/inference/v1
Together AI together api.together.ai/v1
OpenCode opencode api.opencode.ai/v1
Anthropic anthropic api.anthropic.com/v1
Mistral AI mistral api.mistral.ai/v1
Cohere cohere api.cohere.ai/compatibility/v1
Google AI google generativelanguage.googleapis.com/v1beta/openai/

Any OpenAI-Compatible Provider

# Using --api-key directly
python main.py "Your task" --base-url https://your-api.com/v1 --api-key key --model name

# Using environment variable (avoids key in command history)
$env:API_KEY="your-key-here"   # PowerShell
python main.py "Your task" --base-url https://your-api.com/v1 --model name

# Linux/macOS
export API_KEY="your-key-here"
python main.py "Your task" --base-url https://your-api.com/v1 --model name

Model Discovery

Before running, discover what models are available:

# List models from any provider
python main.py --list-models --provider ollama
python main.py --list-models --provider groq --api-key gsk_xxx

# Interactive selection
python main.py "Your task" --discover --provider ollama

# Auto-discovery (no --model needed with local providers)
python main.py "Your task" --provider ollama

# List all provider presets
python main.py --list-presets

Architecture — For Developers

Project Structure

CriticLoopAI/
|-- main.py                 # CLI entry point
|-- config.py               # Configuration dataclasses
|-- requirements.txt        # Dependencies
|-- test_system.py          # Test suite
|
|-- agents/                 # AI Agent implementations
|   |-- base_agent.py       # Base class: LLM communication, retry, history
|   |-- generator.py        # Generator: creates and improves outputs
|   |-- critic.py           # Critic: evaluates, scores, provides feedback
|   |-- improver.py         # Improver: synthesizes feedback, plans strategy
|
|-- core/                   # Core system logic
|   |-- orchestrator.py     # Loop controller: manages iterations
|   |-- knowledge.py        # Pattern tracking and analysis
|
|-- utils/                  # Utilities
|   |-- logger.py           # Colored console output, file saving
|   |-- model_discovery.py  # Fetches models from any provider API
|
|-- storage/                # Auto-created at runtime (session data)

Agent System

All agents inherit from BaseAgent:

BaseAgent (abstract)
    |-- GeneratorAgent  (temperature: 0.8, creative)
    |-- CriticAgent     (temperature: 0.3, precise)
    |-- ImproverAgent   (temperature: 0.5, balanced)

BaseAgent provides:

  • LLM communication (OpenAI-compatible, Anthropic, Groq formats)
  • Retry logic with exponential backoff (1s, 2s, 4s)
  • Conversation history management (last 5 exchanges)
  • Configurable timeout (default: 60s)

To add a new agent:

  1. Create a new file in agents/
  2. Extend BaseAgent
  3. Implement get_system_prompt() returning your agent's role
  4. Add methods for your agent's specific tasks
  5. Wire it into orchestrator.py

Each agent's system prompt is designed for a specific role:

Agent Prompt Focus Temperature
Generator "Create comprehensive, well-structured outputs. Incorporate feedback precisely. Build upon previous iterations." 0.8
Critic "Be thorough but constructive. Score fairly. Identify strengths AND weaknesses. Suggest concrete improvements." 0.3
Improver "Prioritize by impact. Identify root causes, not symptoms. Create actionable plans. Track patterns across iterations." 0.5

Orchestrator

LoopOrchestrator in core/orchestrator.py manages the entire cycle:

from core.orchestrator import LoopOrchestrator
from config import SystemConfig

config = SystemConfig()
config.llm.base_url = "http://localhost:11434/v1"
config.llm.model = "llama-4-maverick"

orchestrator = LoopOrchestrator(config)
result = await orchestrator.run(
    task="Write a guide about Python",
    requirements=["include examples", "be concise"]
)

# result contains: task, final_output, best_score, iteration_history, etc.

Available methods:

  • run(task, requirements, initial_output) — Full improvement loop
  • run_with_specific_goal(task, target_score, max_iterations) — Temporary override of loop config

Knowledge Store

KnowledgeStore in core/knowledge.py tracks patterns across iterations:

from core.knowledge import KnowledgeStore

store = KnowledgeStore("storage")
store.add_iteration({"iteration": 1, "output": "...", "score": 0.7})

summary = store.get_summary()
# Returns: total_iterations, current_score, best_score, average_score,
#          improvement_trend, recurring_issues, successful_strategies

trend = store.get_improvement_trend()
# Returns: trend ("improving"/"stable"/"declining"), average_improvement, etc.

effectiveness = store.analyze_effectiveness()
# Returns: best_improvement_iteration, total_improvement, consistency

Configuration

Five dataclasses in config.py:

from config import SystemConfig

# From environment variables
config = SystemConfig.from_env()

# Or build manually
config = SystemConfig()
config.llm.provider = "openai_compatible"
config.llm.model = "llama-4-maverick"
config.llm.base_url = "http://localhost:11434/v1"
config.llm.api_key = "not-needed"
config.loop.max_iterations = 5
config.loop.convergence_score = 0.95
config.loop.early_stop_patience = 2
Config Class Controls
LLMConfig Provider, model, API key, base URL, temperature, max tokens
AgentConfig Agent name, temperature, retry attempts, timeout
LoopConfig Max iterations, convergence score, early stop patience, thresholds
StorageConfig Storage directory, save settings, export format
SystemConfig Wraps all configs + verbose flag

Logging and Storage

Console output:

  • Color-coded: Cyan (info), Green (success), Yellow (warning), Red (error), Magenta (header)
  • Icons: (success), (warning), (error), (header)

File storage per session:

storage/YYYYMMDD_HHMMSS/
    iteration_1.json      # Output + critique + improvements + score
    iteration_2.json      # Same for iteration 2
    ...
    final_result.json     # Complete result with all history
    summary.json          # Session summary
    knowledge_store.json  # Pattern analysis

How Each Iteration Works

Iteration 1:

  1. Generator creates initial output for the task
  2. Critic evaluates -> returns critique + improvements + score
  3. Output and score saved

Iteration 2+:

  1. Improver analyzes all previous data:
    • Recurring issues from critique text
    • Score trend (improving/stable/declining)
    • Convergence proximity
  2. Generator creates improved output:
    • Keeps what works
    • Fixes all issues from critique
    • Implements all improvements
    • Must not introduce new issues
  3. Critic evaluates new output:
    • Receives previous critiques for progress tracking
    • Returns new critique, improvements, score
  4. System updates best score/output if improved
  5. Checks stopping conditions

Convergence and Early Stopping

Condition Default What happens
Score >= target 0.95 Loop stops. Final output generated.
No improvement 2 iterations Loop stops. Best output used.
Diminishing returns improvement < 0.05 Warning printed. Loop continues.
Max iterations 5 Loop stops. Safety limit.

The Improver also predicts diminishing returns after each iteration — warning you before the loop would naturally stop.


Output Files

Each session produces:

File Purpose Open with
final_output.md Your final result in readable Markdown Any text editor, browser
session_summary.md Score progression and improvements overview Any text editor, browser
final_result.json Raw data for developers Code editor, API integrations
iteration_N.json Detailed data per iteration Code editor
knowledge_store.json Pattern tracking across sessions Code editor
summary.json Session metadata Code editor

Example session output:

============================================================
SESSION COMPLETE
============================================================

  Task: Write a Python sorting algorithm...
  Final Score: 0.82/1.00
  Iterations: 2
  Improvement: +0.07 (0.75 → 0.82)

  Files saved:
    final_output.md    ← Open this to read your result
    session_summary.md ← Overview of the improvement process
    final_result.json  ← Raw data for developers
============================================================

session_summary.md example:

# Session Summary

| Field | Value |
|-------|-------|
| Task | Write a Python sorting algorithm |
| Final Score | 0.82/1.00 |
| Total Iterations | 2 |
| Convergence | No |

## Score Progression

| Iteration | Score | Change |
|-----------|-------|--------|
| 1 | 0.75 | --- |
| 2 | 0.82 | +0.07 |

## Key Improvements Applied

### Iteration 2 (Score: 0.82)

- Focus on one primary sorting algorithm
- Add a brief summary section
- Simplify explanations for complex parts

Extending the System — For Developers

Add a New Agent

# agents/summarizer.py
from agents.base_agent import BaseAgent
from config import AgentConfig, LLMConfig

class SummarizerAgent(BaseAgent):
    def __init__(self, llm_config: LLMConfig):
        config = AgentConfig(name="Summarizer", temperature=0.4)
        super().__init__(config, llm_config)

    def get_system_prompt(self) -> str:
        return "You are an expert summarizer. Be concise and accurate."

    async def summarize(self, text: str) -> str:
        return await self.generate(f"Summarize this:\n{text}")

Then add it to orchestrator.py.

Add a New Provider

Add the API format handler in base_agent.py:

async def _call_my_provider(self, messages, temperature):
    async with httpx.AsyncClient() as client:
        response = await client.post(
            f"{self.llm_config.base_url}/chat/completions",
            headers={"Authorization": f"Bearer {self.llm_config.api_key}"},
            json={"model": self.llm_config.model, "messages": messages, "temperature": temperature},
            timeout=self.config.timeout_seconds
        )
        return response.json()["choices"][0]["message"]["content"]

Then add the routing in call_llm().

Add Custom Evaluation Criteria

Modify the Critic's system prompt in critic.py to include your domain-specific criteria (e.g., SEO optimization, accessibility, performance).

Use as a Library

import asyncio
from config import SystemConfig
from core.orchestrator import LoopOrchestrator

async def improve_my_output():
    config = SystemConfig.from_env()
    orchestrator = LoopOrchestrator(config)
    result = await orchestrator.run("Your task here")
    print(f"Best score: {result['best_score']}")
    print(f"Output: {result['final_output']}")

asyncio.run(improve_my_output())

Environment Variables

Variable Default Description
AI_PROVIDER openai_compatible Provider type
AI_MODEL llama-4-maverick Model name
API_KEY not-needed API key
AI_BASE_URL http://localhost:11434/v1 API endpoint
AI_TEMPERATURE 0.7 Generation temperature
MAX_ITERATIONS 5 Max iterations
VERBOSE true Verbose output
OPENAI_API_KEY OpenAI key (auto-detected)
ANTHROPIC_API_KEY Anthropic key (auto-detected)
GROQ_API_KEY Groq key (auto-detected)

Running Tests

python test_system.py

Verifies: all imports, default configuration, knowledge store operations.


Examples

Basic

python main.py "Write a guide about Python decorators" --provider ollama

With Requirements

python main.py "Create a REST API" \
    -r "Use RESTful conventions" -r "Include auth" -r "Support pagination" \
    --provider deepseek --api-key sk-xxx

From Existing Output

python main.py "Improve this code" \
    -i "def add(a, b): return a + b" \
    --provider groq --api-key gsk_xxx --model llama-4-maverick

High Quality Target

python main.py "Write marketing copy" \
    --provider openai --api-key sk-xxx --model gpt-5.6 \
    --max-iterations 8 --target-score 0.98

Quiet Mode

python main.py "Your task" --provider ollama -q

FAQ

Do I need an API key? No, not for local providers (Ollama, LM Studio, vLLM). For cloud providers, yes — some have free tiers (Groq, OpenRouter).

Which provider should I use? Depends on your needs. Local providers (Ollama, LM Studio) are free and private. Cloud providers vary in speed, cost, and model quality. Use --list-presets to see available options with their details.

What happens if an API call fails? Retries 3 times with exponential backoff. If all fail, the error is raised.

Can I use multiple providers? Not in one run, but switch between runs to compare.

How do I stop early? Press Ctrl+C. The system saves progress up to that point.

When does the loop stop? Score reaches target (0.95), no improvement for 2 iterations, or max iterations (5).

What tasks can I use this for? Any text-based task: writing, code, analysis, translation, documentation, marketing, research.

Can I use this as a Python library? Yes. Import LoopOrchestrator and call run() programmatically.

How do I extend the system? Add new agents by extending BaseAgent. Add providers by adding API handlers. Modify prompts for domain-specific evaluation.

About

CriticLoopAI - A multi-agent AI improvement system where three specialized agents Generator, Critic, and Improver work in an iterative loop to progressively enhance any text-based task. Features convergence detection, pattern tracking, and support for 20+ LLM providers including Ollama, Groq, DeepSeek, OpenAI, and Anthropic.

Resources

Stars

1 star

Watchers

0 watching

Forks

Releases

Packages

Contributors

Languages