Skip to content

docs: end-to-end adoption guide (workshop-ready, from zero to production) #95

@placerda

Description

@placerda

Summary

Create a sequential end-to-end adoption guide that walks a team from zero to production with AgentOps Toolkit. This document serves as the map — each step links to the detailed tutorial. It is also designed to be the base material for a 1-day hands-on workshop.

Motivation

Existing tutorials cover individual scenarios (model-direct, RAG, agent workflow, etc.) but no document covers the full adoption journey. A developer who just installed AgentOps has no single guide that shows the recommended path from first eval to CI/CD gating to production observability.

Proposed Content

Sequential walkthrough:

  1. Installpip install agentops-toolkit
  2. Init workspaceagentops init, understand .agentops/ structure
  3. Configure first eval — choose scenario, create run.yaml, select bundle + dataset
  4. Run local evalagentops eval run, understand exit codes
  5. Interpret resultsresults.json schema, report.md structure, threshold pass/fail
  6. Set up CI/CDagentops workflow generate, PR gating, branch strategy
  7. Enable tracingAGENTOPS_OTLP_ENDPOINT, AI Toolkit Trace Viewer, Jaeger
  8. Compare runsagentops eval compare, baseline strategy, regression detection
  9. Monitor quality — cadence recommendations, golden dataset maintenance, alerting

Each step should be self-contained but link to the detailed tutorial for that topic.

Workshop Mapping

This guide maps to a 1-day (7h) workshop:

Module Duration Guide Section
Intro and Setup 30 min Steps 1-2
First Eval 45 min Steps 3-4
Interpret Results 30 min Step 5
Scenarios (labs) 60 min Step 3 variants (RAG, tools, HTTP, safety)
CI/CD 45 min Step 6
Tracing 45 min Step 7
Compare and Regression 30 min Step 8
Continuous Operations 30 min Step 9

Acceptance Criteria

  • Document covers the full journey from install to production monitoring
  • Each step links to the relevant detailed tutorial
  • Workshop agenda table included with time estimates
  • Tested by following the guide from scratch on a clean environment

Context

  • Existing tutorials: tutorial-model-direct.md, tutorial-basic-foundry-agent.md, tutorial-rag.md, tutorial-http-agent.md, tutorial-agent-workflow.md, tutorial-conversational-agent.md
  • CI/CD guide: ci-github-actions.md
  • Tracing guide: docs/telemetry.md (PR feat: wire OTel telemetry into runner, add telemetry docs #86)

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions