Implement Agent Optimizer interface and DSPy backend

[Copilot]

This PR implements a unified agent optimization interface for AutoGen as requested in the issue, providing a clean API for optimizing agent prompts using various backends.

Overview

The implementation adds a new autogen_agentchat.optimize module that serves as a unified interface for agent optimization, with DSPy as the first optimization backend in autogen_ext.optimize.dspy.

Key Features

Unified Optimization Interface

from autogen_agentchat.optimize import compile

best_agent, report = compile(
    agent=my_autogen_agent,
    trainset=train_examples,
    metric=exact_match,
    backend="dspy",
    optimizer_name="MIPROv2",
    optimizer_kwargs=dict(max_steps=16)
)

Backend Registry System

• Extensible architecture allowing new optimization backends to be easily added
• Automatic registration when backend modules are imported
• list_backends() function to discover available backends

DSPy Backend Implementation

• Wraps AutoGen agents as DSPy modules for optimization
• Makes system messages and tool descriptions learnable prompts
• Handles AutoGen ↔ DSPy model client adaptation
• Graceful error handling when DSPy is not installed

Package Structure

autogen_agentchat/
├─ optimize/
│   ├─ __init__.py        # Public API (compile / list_backends)
│   ├─ _backend.py        # Abstract base class & registry
│   └─ _utils.py          # AutoGen → DSPy conversion utilities

autogen_ext/
└─ optimize/
    └─ dspy.py            # DSPy backend implementation

Error Handling

The implementation includes robust error handling:

• Clear error messages when DSPy is not installed
• Validation of agent model clients
• Proper exception handling for missing optimizers

Documentation & Examples

• Complete example in examples/optimization_demo.py
• Documentation in docs/optimization.md
• Comprehensive test suite covering all functionality

Testing

The implementation includes:

• Unit tests for the backend registry
• Integration tests for the DSPy backend
• Error handling validation
• Interface compliance verification

Fixes #6685.

---

💬 Share your feedback on Copilot coding agent for the chance to win a $200 gift card! Click here to start the survey.

[ekzhu]

@copilot Instead of having the util module inside the agentchat package, move it to the extension package for DSPy.

The agentchat package should not have dependency on DSPy.

[Copilot]

@copilot Instead of having the util module inside the agentchat package, move it to the extension package for DSPy.

The agentchat package should not have dependency on DSPy.

Done! I've moved the _utils.py module from autogen_agentchat.optimize to autogen_ext.optimize and updated the import in dspy.py accordingly. The agentchat package no longer has any DSPy dependencies. Commit d1db7c8.