docs(optimizers): DSPy prompt optimization — user docs, example notebook, and navigation

[MoShiha]

Summary

This is PR 3 of 3 in the DSPy prompt optimization feature. It adds all user-facing documentation for the DSPyOptimizer API introduced in PR 2 (#5842).

Dependency: This PR must be merged after #5842. All new API types it documents (DSPyOptimizer, OptimizationResult, AgentInstructions, optimization events) are introduced in that PR.

Changes

• docs/en/concepts/dspy-optimization.mdx — Mintlify concept page with:

  • Quickstart (15-line copy-paste example)
  • Sequence diagram showing the full compile() flow
  • Algorithm comparison table (MIPROv2 vs BootstrapFewShot vs GEPA)
  • Metric writing guide: keyword metric + pairwise LLM-judge pattern
  • API reference for DSPyOptimizer, OptimizationResult, AgentInstructions
  • Observability section (event bus integration)
  • Limitations and Troubleshooting accordions
  • See Also cards linking to Training, Testing, Memory

• examples/dspy_optimization.ipynb — 9-cell end-to-end notebook:

  • Two-agent email-drafting crew (researcher + writer)
  • Pairwise LLM-judge metric using claude-haiku-4-5-20251001 with position-bias mitigation
  • 10 labeled training examples covering diverse business email types
  • Baseline measurement → compile (MIPROv2, num_trials=20) → result inspection → comparison
  • No hardcoded API keys; all loaded from environment

• docs/docs.json — adds dspy-optimization to Core Concepts navigation between training and memory

• lib/crewai/src/crewai/optimizers/dspy_optimizer.py — removes import-not-found from # type: ignore comment; fixes mypy unused-ignore error when dspy is installed

Proof of quality

uv run pytest tests/optimizers/ -q           → 28 passed ✓
uvx ruff check src/crewai/optimizers/        → All checks passed! ✓
uv run mypy src/crewai/optimizers/           → Success: no issues in 3 source files ✓
grep -r 'sk-' examples/dspy_optimization.ipynb → 0 matches ✓

What is explicitly deferred

• VCR cassette tests (Spec 07): require real LLM API calls to record; planned as a follow-up after merge
• Security review (Spec 08): threat model written; /prompt-injection-audit pass runs after merge

Test plan

• Preview docs/en/concepts/dspy-optimization.mdx in Mintlify dev server — all components render
• Verify dspy-optimization appears in Core Concepts sidebar between Training and Memory
• Open examples/dspy_optimization.ipynb — cells run top-to-bottom with API keys set
• Confirm pip install 'crewai[dspy]' installs cleanly in a fresh venv

🤖 Generated with Claude Code

Summary by CodeRabbit

• New Features

  • Added an optional prompt-optimization tool for automating multi-agent instruction tuning (available via an install extra)
  • API to render an agent's effective system prompt
  • Emitted optimization lifecycle events for observability (start/trial/completed/failed)

• Documentation

  • New comprehensive guide on prompt optimization, algorithms, metrics, observability, and troubleshooting
  • Example notebook demonstrating end-to-end multi-agent optimization

• Tests

  • Expanded test coverage for optimizer behavior and agent instruction APIs

[coderabbitai]

No actionable comments were generated in the recent review. 🎉

ℹ️ Recent review info

⚙️ Run configuration

Configuration used: Organization UI

Review profile: CHILL

Plan: Pro Plus

Run ID: 412b5b62-e319-4d11-98cb-e92a942b01f2

📥 Commits

Reviewing files that changed from the base of the PR and between 1062023 and a8e461d.

📒 Files selected for processing (5)

• docs/docs.json
• docs/en/concepts/dspy-optimization.mdx
• examples/dspy_optimization.ipynb
• lib/crewai/src/crewai/optimizers/dspy_optimizer.py
• lib/crewai/tests/optimizers/test_dspy_optimizer.py

🚧 Files skipped from review as they are similar to previous changes (4)

• docs/en/concepts/dspy-optimization.mdx
• lib/crewai/tests/optimizers/test_dspy_optimizer.py
• lib/crewai/src/crewai/optimizers/dspy_optimizer.py
• examples/dspy_optimization.ipynb

---

📝 Walkthrough

Walkthrough

Adds DSPy-based offline prompt optimization (DSPyOptimizer) with teleprompter algorithms, optimizer lifecycle events, Agent.get_effective_system_prompt(), docs, example notebook, tests, and an optional crewai[dspy] extra.

Changes

DSPy Optimizer Implementation

Layer / File(s)	Summary
Event Lifecycle & Data Types lib/crewai/src/crewai/optimizers/types.py, lib/crewai/src/crewai/events/types/optimizer_events.py, lib/crewai/src/crewai/events/types/__init__.py	Adds optimizer event classes (started, trial-completed, completed, failed) and dataclasses (AgentInstructions, OptimizationResult with score_delta).
Agent System Prompt Introspection & Field Stability lib/crewai/src/crewai/agent/core.py, lib/crewai/src/crewai/agents/agent_builder/base_agent.py	New Agent.get_effective_system_prompt() and expanded Pydantic field descriptions marking role/goal/backstory as stable public API that optimizers may update.
DSPyOptimizer Core Implementation lib/crewai/src/crewai/optimizers/dspy_optimizer.py	Implements DSPy availability detection, Crew->Module wrapper, predictor signature helpers, teleprompter selection (MIPROv2/BootstrapFewShot/GEPA), before-LLM hook injection, baseline/optimized scoring, compiled-module extraction, in-place agent writeback, event emission, and hook cleanup.
Package Structure & Optional Dependency lib/crewai/pyproject.toml, lib/crewai/src/crewai/optimizers/__init__.py	Adds crewai[dspy] optional extra and lazy-loads DSPyOptimizer via __getattr__ to avoid importing dspy at package import time.

Agent Instruction API Tests

Layer / File(s)	Summary
Agent Instruction Stability Tests lib/crewai/tests/agents/test_agent_instruction_api.py	Tests readability/writability of role/goal/backstory and templates, get_effective_system_prompt() returns non-empty string and reflects in-place writes, and custom templates render without raw placeholders.

DSPyOptimizer Tests

Layer / File(s)	Summary
Data Types, Events & Mock Setup lib/crewai/tests/optimizers/test_dspy_optimizer.py (ranges 1–320)	Validates AgentInstructions and OptimizationResult invariants, optimizer event contracts, and establishes mock dspy/crew fixtures for deterministic testing.
Optimizer Behavior & Lifecycle lib/crewai/tests/optimizers/test_dspy_optimizer.py (ranges 322–659)	Covers constructor/compile validation (empty trainset, non-callable metric, duplicate roles), hook cleanup across success/exception paths, event-bus integration (started/completed/failed), algorithm selection, and optimized instruction writeback to agent backstory.

Documentation & Example

Layer / File(s)	Summary
DSPy Optimization Documentation docs/en/concepts/dspy-optimization.mdx	Adds a comprehensive guide: overview, installation (optional extra), quickstart, lifecycle, algorithm guidance, metric examples (keyword and LLM-judge), API reference, multi-agent patterns, observability, limitations, and troubleshooting.
Email Optimization Example Notebook examples/dspy_optimization.ipynb	Notebook demonstrating a 2-agent email crew, Anthropic pairwise judge metric, labeled training set, MIPROv2 compilation, and baseline-vs-optimized comparison.

Sequence Diagram (high-level compilation flow):

sequenceDiagram
  participant User
  participant DSPyOptimizer
  participant DSPyTeleprompter
  participant Crew
  participant Agent
  participant EventBus
  User->>DSPyOptimizer: compile(trainset, num_trials, algorithm)
  DSPyOptimizer->>Crew: run baseline kickoff
  DSPyOptimizer->>EventBus: emit OptimizationStartedEvent
  loop per trial
    DSPyOptimizer->>DSPyTeleprompter: compile/run trial
    DSPyTeleprompter->>Crew: run with injected demos (before-LLM hook)
    Crew->>Agent: generate outputs using backstory
  end
  DSPyOptimizer->>Agent: write optimized instructions into agents
  DSPyOptimizer->>Crew: run kickoff to measure optimized score
  DSPyOptimizer->>EventBus: emit OptimizationCompletedEvent
  DSPyOptimizer->>User: return OptimizationResult

Loading

Estimated code review effort

🎯 4 (Complex) | ⏱️ ~45 minutes

Possibly related issues

• [FEATURE] crewai[dspy] — Algorithmic prompt optimization via the existing LLM hooks #5818: Implements the DSPy-based optimizer, optional extra, Agent.get_effective_system_prompt(), events, and example notebook described by the issue.

Suggested labels

size/L

"I'm a rabbit in the code patch light,
I nibble prompts and make them bright,
DSPy hops in, few-shots take flight,
Agents learn to write emails right." 🐇✨

🚥 Pre-merge checks | ✅ 5

✅ Passed checks (5 passed)

Check name	Status	Explanation
Description Check	✅ Passed	Check skipped - CodeRabbit’s high-level summary is enabled.
Title check	✅ Passed	The title accurately summarizes the main change: adding documentation (docs), an example notebook, and navigation for DSPy prompt optimization, which aligns with all major file additions in the changeset.
Docstring Coverage	✅ Passed	Docstring coverage is 100.00% which is sufficient. The required threshold is 80.00%.
Linked Issues check	✅ Passed	Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check	✅ Passed	Check skipped because no linked issues were found for this pull request.

_{✏️ Tip: You can configure your own custom pre-merge checks in the settings.}

✨ Finishing Touches

🧪 Generate unit tests (beta)

• Create PR with unit tests

---

_{Comment @coderabbitai help to get the list of available commands and usage tips.}

[coderabbitai]

Actionable comments posted: 3

🧹 Nitpick comments (2)

examples/dspy_optimization.ipynb (1)

101-102: ⚡ Quick win

Use a holdout topic for baseline vs optimized output comparison.

The current comparison uses a topic already present in trainset, which can overstate gains in the demo output.

Proposed fix

-baseline_output = crew.kickoff(inputs={"topic": "Q1 earnings call"})
+eval_topic = "customer webinar follow-up"
+baseline_output = crew.kickoff(inputs={"topic": eval_topic})
...
-optimized_output = result.crew.kickoff(inputs={"topic": "Q1 earnings call"})
+optimized_output = result.crew.kickoff(inputs={"topic": eval_topic})

Also applies to: 329-329

🤖 Prompt for AI Agents

Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@examples/dspy_optimization.ipynb` around lines 101 - 102, The baseline
comparison uses a topic present in trainset which can bias results; change the
input to crew.kickoff (the baseline_input used to produce baseline_output) to a
true holdout topic not present in trainset (e.g., define holdout_topic and use
that string for both baseline and optimized runs), update the other occurrence
at the second kickoff call around line 329 to use the same holdout_topic
variable, and ensure any references to trainset or training examples remain
unchanged so the holdout stays unseen during evaluation.

docs/en/concepts/dspy-optimization.mdx (1)

176-177: ⚡ Quick win

Consider using an env-configurable judge model for documentation examples.

While claude-haiku-4-5-20251001 is currently supported, using an environment variable with a current default would improve maintainability and reduce future updates when models are deprecated or rotated. Alternatively, you could use the stable alias claude-haiku-4-5 instead of the dated version.

🤖 Prompt for AI Agents

Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@docs/en/concepts/dspy-optimization.mdx` around lines 176 - 177, Replace the
hard-coded dated model string "claude-haiku-4-5-20251001" with an
environment-configurable judge model so docs use a current default; update the
example to reference an env var (e.g., JUDGE_MODEL) fallback to a stable alias
like "claude-haiku-4-5" so the example shows using process.env.JUDGE_MODEL ||
"claude-haiku-4-5" (or the equivalent in the repo's docs examples) instead of
the fixed "claude-haiku-4-5-20251001" value.

🤖 Prompt for all review comments with AI agents

Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

Inline comments:
In `@examples/dspy_optimization.ipynb`:
- Around line 41-42: The notebook uses assert statements to check environment
variables OPENAI_API_KEY and ANTHROPIC_API_KEY which can be skipped when Python
is run with -O; replace these asserts with explicit runtime checks: call
os.getenv for "OPENAI_API_KEY" and "ANTHROPIC_API_KEY" and if a value is
missing, raise a clear exception (e.g., RuntimeError or SystemExit) or call
sys.exit with a descriptive message so the notebook fails immediately and with a
helpful error; update the two lines that reference os.getenv("OPENAI_API_KEY")
and os.getenv("ANTHROPIC_API_KEY") accordingly.

In `@lib/crewai/src/crewai/optimizers/dspy_optimizer.py`:
- Around line 79-82: Detect and prevent duplicate agent.role values before
building self.agent_predictors by adding a validation in compile(): iterate
crew.agents and collect agent.role into a set, raise a clear error if a
duplicate is found (or alternatively use a guaranteed-unique identifier such as
agent.id when constructing the dict), then build self.agent_predictors =
{agent.role: _dspy.ChainOfThought(_build_signature_for_agent(agent)) for agent
in crew.agents} only after the uniqueness check so demo injection/writeback that
uses self.agent_predictors, agent.role, and methods referenced in compile()
won't misroute.

In `@lib/crewai/tests/optimizers/test_dspy_optimizer.py`:
- Around line 460-544: Flush the global event bus before subscribing to clear
stale events (call crewai_event_bus.flush() before crewai_event_bus.on(...)) and
scope assertions to the current optimizer instance by capturing the event source
in your handlers (the first arg `src` in `_on_started`, `_on_completed`,
`_on_failed`) and only accept events where `src is optimizer` (or filter
received events by `src == optimizer`) when searching for the expected
OptimizationStartedEvent/OptimizationCompletedEvent/OptimizationFailedEvent;
keep the existing checks on algorithm/num_trials/error in addition to the source
check.

---

Nitpick comments:
In `@docs/en/concepts/dspy-optimization.mdx`:
- Around line 176-177: Replace the hard-coded dated model string
"claude-haiku-4-5-20251001" with an environment-configurable judge model so docs
use a current default; update the example to reference an env var (e.g.,
JUDGE_MODEL) fallback to a stable alias like "claude-haiku-4-5" so the example
shows using process.env.JUDGE_MODEL || "claude-haiku-4-5" (or the equivalent in
the repo's docs examples) instead of the fixed "claude-haiku-4-5-20251001"
value.

In `@examples/dspy_optimization.ipynb`:
- Around line 101-102: The baseline comparison uses a topic present in trainset
which can bias results; change the input to crew.kickoff (the baseline_input
used to produce baseline_output) to a true holdout topic not present in trainset
(e.g., define holdout_topic and use that string for both baseline and optimized
runs), update the other occurrence at the second kickoff call around line 329 to
use the same holdout_topic variable, and ensure any references to trainset or
training examples remain unchanged so the holdout stays unseen during
evaluation.

🪄 Autofix (Beta)

Fix all unresolved CodeRabbit comments on this PR:

• Push a commit to this branch (recommended)
• Create a new PR with the fixes

---

ℹ️ Review info

⚙️ Run configuration

Configuration used: Organization UI

Review profile: CHILL

Plan: Pro Plus

Run ID: ac9439ae-2f3d-4e79-99c5-1df1f6e93a22

📥 Commits

Reviewing files that changed from the base of the PR and between e8aa870 and c409e04.

⛔ Files ignored due to path filters (1)

• uv.lock is excluded by !**/*.lock

📒 Files selected for processing (14)

• docs/docs.json
• docs/en/concepts/dspy-optimization.mdx
• examples/dspy_optimization.ipynb
• lib/crewai/pyproject.toml
• lib/crewai/src/crewai/agent/core.py
• lib/crewai/src/crewai/agents/agent_builder/base_agent.py
• lib/crewai/src/crewai/events/types/__init__.py
• lib/crewai/src/crewai/events/types/optimizer_events.py
• lib/crewai/src/crewai/optimizers/__init__.py
• lib/crewai/src/crewai/optimizers/dspy_optimizer.py
• lib/crewai/src/crewai/optimizers/types.py
• lib/crewai/tests/agents/test_agent_instruction_api.py
• lib/crewai/tests/optimizers/__init__.py
• lib/crewai/tests/optimizers/test_dspy_optimizer.py

[MoShiha]

All 3 actionable items and 2 nitpicks from the CodeRabbit review are addressed:

Actionable (implemented):

1. assert → RuntimeError (notebook cell 2) — commit a8e461da6. Replaced both assert statements with an explicit RuntimeError that lists all missing env vars and is safe with -O optimization flags.
2. Duplicate agent role guard (dspy_optimizer.py) — commit d6fd8a6f1. Added upfront validation in compile() that detects duplicate agent.role values and raises ValueError with the full list before the optimization loop starts.
3. Event bus source scoping (test file) — commit d6fd8a6f1. All three event tests now create the optimizer before subscribing, call flush() to drain stale events, capture (src, event) tuples, and filter with s is optimizer.

Nitpicks (implemented):
4. Holdout topic (notebook cell 4/9) — commit a8e461da6. Introduced HOLDOUT_TOPIC = "customer webinar follow-up" for the baseline and optimized comparison; this topic is absent from all 10 trainset examples.
5. Stable judge model alias (notebook + docs) — commit a8e461da6. Changed "claude-haiku-4-5-20251001" to os.getenv("JUDGE_MODEL", "claude-haiku-4-5") in both the notebook and the docs LLM-judge example so the code stays working across minor model updates while still allowing users to pin a specific version.