Module 13 Agentic QA and Evaluation

🔬 Module 13 · Agentic QA and Evaluation

How you prove an agent works — and block the ones that do not.

Module 13 of 16 · track Ship · Estimated time: 5–6 hours · Prerequisites: Module 06 or 08. Module 11 recommended.

Non-deterministic systems need a different testing discipline. This module gives you golden sets, contract tests, multi-agent tests, a quality gate with real thresholds, and a CI step that actually blocks a bad deploy.

Open the module on GitHub ↗: the slides, notebooks, exercises and solutions are there. This page is the reading order, with the tutorial and the workbench alongside.

Read first, on the site

The lesson that frames this module is The Agentic PDLC for QA: How to Test Probabilistic Software (6 min): testing software that is right a share of the time. Then try the golden-set confidence calculator in the workbench: the same decision, with your numbers.

What you will be able to do

• Build a golden set that represents real usage, not happy paths
• Write contract tests for tool schemas and agent outputs
• Test multi-agent handoffs where the bug lives between agents
• Run a quality gate that exits non-zero and blocks promotion
• Debug a failing agent from CloudWatch traces

Concepts in this module

Portable GenAI concepts	AWS-specific surface
Golden sets	CloudWatch Logs Insights
Contract testing	AgentCore Observability
LLM-as-judge	Bedrock model invocation logging
Quality gates
Non-deterministic testing

Portable concepts transfer to any stack. The AWS column is where this module touches the cloud — see docs/concepts/portability-matrix.md.

Run it in this order

#		Step	What it is
1	📖	slides/Agentic_QA_Training.pptx	Agentic QA
2	📖	src/golden_set.jsonl	The golden set — study its shape
3	💻	notebooks/eval_harness.ipynb	Evaluation harness
4	📖	src/test_contracts.py	Contract tests
5	📖	src/test_multiagent.py	Multi-agent tests
6	📖	src/quality_gate.py	The quality gate — the file with teeth
7	💻	notebooks/debug_walkthrough.ipynb	Debugging walkthrough
8	🔖	src/cloudwatch_filters.md	CloudWatch filter patterns
9	✏️	exercises/QA_Exercise_TravelMind.md	QA exercise — TravelMind

📖 read   💻 run   ✏️ practise   📊 workbook   🔖 reference

There is no separate solutions/ folder — the exercise carries its own answer key, and the module's src/ is itself the worked reference: a real golden set, real contract tests, and a quality gate that exits non-zero. Read those after you have attempted the exercise, not before.

Common mistakes

• A gate that warns instead of failing. Teams learn to ignore it within a week.
• Golden sets built only from cases you already pass.

When it breaks

Every error anyone has hit in this course is in the Error Index, by the exact string. Model access and region problems are in Model and Region Notes. If it is a question rather than an error, check the answered ones first, then ask in Q&A.

Field guide for this module

Frameworks

• Evidence Ladder — Never claim above your rung
• Abstention Budget — The vital sign, and how to target it
• Failure Signature Catalog — Debug from evidence, not the prompt
• Silent Degradation Watchlist — 12 failures that never raise an error

Quick reference

• Observability — Ten log fields and Logs Insights queries

Recipes and procedures

• Build a quality gate — A gate that exits non-zero
• Validate an LLM judge — Calibrate before you trust

---

← Module 12 · A2A and A2UI: Agent Interoperability · All modules · Module 14 · End-to-End Production Pipeline →

_{✏️ This page is generated from modules/13-agentic-qa-and-evaluation/README.md. To change it, edit the module README: an edit made here is replaced at the next sync.}