Add custom evaluators tutorial and extend evaluation docs

[nadheesh]

Closes #583

Summary

• Add a new Custom Evaluators tutorial with step-by-step walkthrough for creating code and LLM-judge evaluators in the AMP Console
• Extend Evaluation concepts page with a Custom Evaluators section, tabbed evaluator type/level/built-in evaluator reference, and expanded Viewing Results (monitor dashboard + trace view)
• Extend Evaluation Monitors tutorial with score breakdown tables (by agent, by model) and trace view score visibility sections
• Add new screenshots for custom evaluator UI and evaluation trace view
• Update sidebar to include the new tutorial

Summary by CodeRabbit

• Documentation
  • Added comprehensive guide for creating and using custom evaluators with Code and LLM-Judge options
  • Enhanced evaluation documentation with improved structure and tabbed interfaces for easier navigation
  • Expanded evaluation monitoring tutorial with new score breakdown capabilities and per-level statistics
  • Added detailed guidance for viewing and tracking evaluation scores within trace details

[coderabbitai]

No actionable comments were generated in the recent review. 🎉

ℹ️ Recent review info

⚙️ Run configuration

Configuration used: Path: .coderabbit.yaml

Review profile: CHILL

Plan: Pro

Run ID: e8687703-94e2-4c63-b90d-fbffd856cbb8

📥 Commits

Reviewing files that changed from the base of the PR and between 09ed80a and 08862b2.

⛔ Files ignored due to path filters (9)

• website/docs/img/evaluation/custom-eval-basic-details.png is excluded by !**/*.png
• website/docs/img/evaluation/custom-eval-code-details.png is excluded by !**/*.png
• website/docs/img/evaluation/custom-eval-code-editor.png is excluded by !**/*.png
• website/docs/img/evaluation/custom-eval-list.png is excluded by !**/*.png
• website/docs/img/evaluation/custom-eval-llm-judge-editor.png is excluded by !**/*.png
• website/docs/img/evaluation/monitor-dashboard.png is excluded by !**/*.png
• website/docs/img/evaluation/run-logs.png is excluded by !**/*.png
• website/docs/img/evaluation/span-scores-tab.png is excluded by !**/*.png
• website/docs/img/evaluation/traces-table-scores.png is excluded by !**/*.png

📒 Files selected for processing (4)

• website/docs/concepts/evaluation.mdx
• website/docs/tutorials/custom-evaluators.mdx
• website/docs/tutorials/evaluation-monitors.mdx
• website/sidebars.ts

---

📝 Walkthrough

Walkthrough

This PR extends evaluation-related documentation by adding a new comprehensive tutorial for creating custom evaluators and reorganizing existing evaluation concepts and monitoring guides with tabbed interfaces, new sections, and enhanced UI guidance for score visibility and breakdowns.

Changes

Cohort / File(s)	Summary
Custom Evaluators Documentation website/docs/tutorials/custom-evaluators.mdx	New tutorial covering prerequisites, step-by-step creation of Code and LLM-Judge evaluators, configuration, tagging, and usage within monitors; includes detailed examples and UI navigation guidance.
Evaluation Concepts & Guides website/docs/concepts/evaluation.mdx, website/docs/tutorials/evaluation-monitors.mdx	Extends evaluation documentation with tabbed interfaces for Rule-Based vs. LLM-as-Judge evaluators, adds Custom Evaluators section with examples, reorganizes built-in evaluators table, and augments monitor dashboards with score breakdowns by agent/model and trace view score visibility.
Navigation website/sidebars.ts	Adds new custom-evaluators tutorial to sidebar Tutorials section.

Estimated code review effort

🎯 3 (Moderate) | ⏱️ ~20 minutes

Poem

🐰 With tabs and tables, docs now gleam,
Custom judges fulfill a dream,
From code to prompts, evaluators arise,
Score breakdowns shine before our eyes!
Documentation hops to life anew,
Guiding users in all they do! ✨

🚥 Pre-merge checks | ✅ 5

✅ Passed checks (5 passed)

Check name	Status	Explanation
Title check	✅ Passed	The title directly describes the main changes: adding a custom evaluators tutorial and extending evaluation documentation.
Description check	✅ Passed	The description includes a linked issue reference and clear summary of changes, though it lacks several template sections like Purpose, Goals, Approach, and others.
Linked Issues check	✅ Passed	All coding requirements from issue #583 are met: new Custom Evaluators tutorial added, Evaluation concepts page extended with tabbed sections, Evaluation Monitors tutorial enhanced, sidebar updated.
Out of Scope Changes check	✅ Passed	All changes are directly aligned with issue #583 objectives; no out-of-scope modifications detected in the documentation updates and sidebar configuration.
Docstring Coverage	✅ Passed	No functions found in the changed files to evaluate docstring coverage. Skipping docstring coverage check.

_{✏️ Tip: You can configure your own custom pre-merge checks in the settings.}

✨ Finishing Touches

🧪 Generate unit tests (beta)

• Create PR with unit tests

📝 Coding Plan

• Generate coding plan for human review comments

---

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

• X
• Mastodon
• Reddit
• LinkedIn

_{Comment @coderabbitai help to get the list of available commands and usage tips.}