refactor: replace config-based agents with lightweight registry

[enitrat]

PR Summary: refactor: replace config-based agents with lightweight enum registry and optimize performance

📜 High-Level Summary

This PR replaces the configuration-based agent system with a lightweight enum-based registry, significantly simplifying agent management. The refactoring removes TOML-based agent configurations in favor of an in-memory registry with two built-in agents: Cairo Coder (general purpose) and Scarb Assistant (Scarb-specific). Additionally, the PR optimizes configuration loading by caching parsed TOML files and removes unnecessary empty __init__.py files throughout the codebase to improve import performance.

📊 Architectural Impact & Visualizations

This diagram illustrates the simplified agent creation flow after removing the configuration-based system:

flowchart LR
    subgraph "Before"
        A1[config.toml] --> B1[ConfigManager]
        B1 --> C1[AgentConfiguration]
        C1 --> D1[AgentFactory]
        D1 --> E1[RagPipeline]
    end
    
    subgraph "After"
        A2[AgentId Enum] --> B2[registry.py]
        B2 --> C2[AgentSpec]
        C2 --> D2[AgentFactory]
        D2 --> E2[RagPipeline]
    end
    
    style A1 fill:#ffcccc
    style C1 fill:#ffcccc
    style A2 fill:#ccffcc
    style B2 fill:#ccffcc
    style C2 fill:#ccffcc

Loading

⚙️ Detailed Changeset Breakdown

---

Changeset 1: Introduce Lightweight Agent Registry

Files Affected:

• python/src/cairo_coder/agents/registry.py

Summary of Changes:

• Created new enum-based agent registry with AgentId enum and AgentSpec dataclass
• Defined two built-in agents: CAIRO_CODER and SCARB with their specifications
• Added build() method to create RagPipeline instances from specifications
• Implemented get_agent_by_string_id() helper for string-based agent lookup

[TRIAGE]: NEEDS_REVIEW

---

Changeset 2: Refactor Agent Factory to Use Registry

Files Affected:

• python/src/cairo_coder/core/agent_factory.py

Summary of Changes:

• Removed all configuration-based agent creation methods
• Simplified constructor to only take vector_db and vector_store_config
• Modified get_or_create_agent() to use the new registry
• Removed static methods create_agent(), create_agent_by_id(), and helper methods
• Updated list_available_agents() and get_agent_info() to use registry
• Removed DefaultAgentConfigurations class entirely
• Changed create_agent_factory() signature to remove config_manager and custom_agents

[TRIAGE]: NEEDS_REVIEW

---

Changeset 3: Remove Agent Configuration from Core Config

Files Affected:

• python/src/cairo_coder/core/config.py

Summary of Changes:

• Removed AgentConfiguration dataclass
• Removed RagSearchConfig dataclass
• Removed agents field from Config dataclass
• Removed default_agent_id field from Config dataclass
• Kept only VectorStoreConfig which is still needed

[TRIAGE]: NEEDS_REVIEW

---

Changeset 4: Simplify Configuration Manager

Files Affected:

• python/src/cairo_coder/config/manager.py

Summary of Changes:

• Removed get_agent_config() static method
• Removed agent-related validation from validate_config()
• Simplified load_config() to only handle vector store configuration
• Added caching mechanism for parsed TOML files to reduce file I/O

[TRIAGE]: NEEDS_REVIEW

---

Changeset 5: Update Server to Use New Agent Factory

Files Affected:

• python/src/cairo_coder/api/server.py

Summary of Changes:

• Modified agent factory initialization to use new constructor signature
• Updated chat completion endpoint to handle agent selection
• Fixed default agent handling to use "cairo-coder" as fallback
• Removed config_manager from agent factory creation

[TRIAGE]: NEEDS_REVIEW

---

Changeset 6: Update RAG Pipeline Factory

Files Affected:

• python/src/cairo_coder/core/rag_pipeline.py

Summary of Changes:

• Removed create_scarb_pipeline() method (now handled by registry)
• Simplified pipeline creation to use generation program types directly
• Removed contract_template and test_template parameters throughout

[TRIAGE]: NEEDS_REVIEW

---

Changeset 7: Update Tests for New Architecture

Files Affected:

• python/tests/conftest.py
• python/tests/test_agent_factory.py
• python/tests/test_config_manager.py
• python/tests/integration/test_server.py

Summary of Changes:

• Removed sample_agent_configs fixture
• Updated tests to use the new registry-based approach
• Simplified test setup without agent configurations
• Fixed integration tests to work with new agent factory

[TRIAGE]: NEEDS_REVIEW

---

Changeset 8: Update Documentation

Files Affected:

• python/CLAUDE.md
• .claude/commands/commit.md

Summary of Changes:

• Updated CLAUDE.md to document the new registry-based agent system
• Updated instructions for adding new agents (now requires modifying registry.py)
• Added new commit.md command documentation for smart git commits
• Removed references to TOML-based agent configuration

[TRIAGE]: APPROVED

---

Changeset 9: Remove Empty Init Files

Files Affected:

• python/src/cairo_coder/__init__.py
• python/src/cairo_coder/agents/__init__.py
• python/src/cairo_coder/api/__init__.py
• python/src/cairo_coder/config/__init__.py
• python/src/cairo_coder/core/__init__.py
• python/src/cairo_coder/dspy/__init__.py
• python/src/cairo_coder/utils/__init__.py

Summary of Changes:

• Removed empty __init__.py files throughout the codebase
• These files were not providing any functionality and removing them improves import performance

[TRIAGE]: APPROVED

---

Changeset 10: Update Optimizer Notebooks

Files Affected:

• python/src/cairo_coder/optimizers/QueryAndRetrieval.py

Summary of Changes:

• Fixed handling of missing config files in optimizer notebooks
• Added graceful fallback when config.toml is not found
• Updated agent initialization to work with new factory pattern

[TRIAGE]: APPROVED

---

Changeset 11: Update Project Configuration

Files Affected:

• python/pyproject.toml

Summary of Changes:

• Added optimizer notebooks to pytest testpaths to ensure API compatibility
• Ensures that the API used in notebooks continues to work properly

[TRIAGE]: APPROVED

[ijusttookadnatest]

Great PR