Add azure_doc_qa example: multi-agent doc QA with eval - #86
Merged
Conversation
Replace generic 'Missing or invalid' errors in _kind_and_test_case_id with a _raise_schema_hint helper that shows: - which field is missing and what was expected - the actual keys present in the row - a concrete 'rm -rf <run_dir>' command to delete stale artifacts - instruction to re-run the pipeline This helps users who hit cached artifacts from an older format understand the problem and fix it without needing to know internal schema history.
Add task cancellation to _bounded_loop_teardown() between shutdown_asyncgens() and executor.shutdown(). Without this, litellm's LoggingWorker._worker_loop coroutine stays pending when loop.close() is called, producing noisy 'Task was destroyed but it is pending'@ 2>&1 tracebacks at pipeline exit. This mirrors what asyncio.run() does internally: enumerate all remaining tasks, cancel them, then gather() to let cancellations propagate before closing the loop.
Scaffold the example directory structure with __init__.py and 8 reference documents (3 external, 5 internal) that the multi-agent system uses as its knowledge base for answering Azure AI questions.
Six mock MCP tools (search_docs, get_doc_content, check_service_status, get_incident_report, search_internal_kb, get_architecture_diagram) that simulate real tool backends for deterministic evaluation.
Three-node LangGraph agent (router → doc_specialist → internal_kb) with an MCP client adapter that bridges mock tools into the LangGraph ToolNode interface.
9 behavior categories, 25 test cases, and 7 judge dimensions covering factual accuracy, tool usage, confidentiality handling, multi-hop reasoning, and cross-reference synthesis.
auto_trace.py provides the p2m callable entry point with OpenTelemetry instrumentation. README.md documents the example architecture, setup instructions, and how to run the eval.
tangym
force-pushed
the
yemingtang/azure-doc-qa-example
branch
from
May 23, 2026 00:39
3a74a57 to
acd2c0b
Compare
- Set all temperature values to 1.0 (gpt-5 family only supports temperature=1) - Remove unsupported test_case_count field from test_set config - Use agent:chat_sync directly instead of auto_trace:chat_sync (avoids phoenix dependency); remove trace config block
…nto yemingtang/azure-doc-qa-example
The existing _FilteredStderr wrapper on sys.stderr was ineffective because
the logging console handler writes to sys.__stderr__ (the unwrapped
original), bypassing the filter entirely. Async-cleanup tracebacks from
httpx/LangGraph/litellm reach users through three separate channels:
1. Python logging (asyncio logger) — 'Task exception was never retrieved'
and 'Task was destroyed but it is pending!' go through
logging.getLogger('asyncio').error() → root handlers → sys.__stderr__.
Fix: add _AsyncCleanupLoggingFilter to root logger handlers and the
asyncio logger.
2. sys.unraisablehook — 'Exception ignored in: ...' messages from __del__
methods that raise RuntimeError('Event loop is closed').
Fix: install a custom unraisablehook that suppresses these.
3. Direct stderr writes — kept as last resort via _FilteredStderr.
Verified with full pipeline run: zero noise lines in output.
Restore auto_trace:chat_sync callable and trace backend configuration that was accidentally removed in the temperature fix commit.
When inference is re-run (forced or config hash mismatch), the old scores.jsonl and .judge_config_hash were left behind. The viewer builder then crashed with ViewerReadModelBuildError trying to look up old test_case_ids in the new inference data. Add _remove_stale_judge_artifacts() to delete scores.jsonl and .judge_config_hash whenever inference_set.jsonl is discarded.
Instead of crashing with ViewerReadModelBuildError when a score row references a test_case_id absent from inference_set.jsonl, log a warning and fall back to transcript_only mode. This is the same behavior as when scores.jsonl doesn't exist at all.
After generating test_set.jsonl, compare the set of behavior categories covered by generated test cases against the full taxonomy. If any categories are missing, log a WARNING with the missing category names and the minimum sample_size needed to guarantee full coverage.
triage() was adding its LLM response (a raw JSON routing object like
{"route":"product_documentation","reason":"..."}) as an AIMessage
to the graph state. This polluted the message stream: if the specialist
node's synthesis produced an empty-content AIMessage, the reverse scan
in chat() fell back to this routing JSON as the agent's final answer.
Return only {"route": route} from triage — the routing decision is
metadata for the conditional edge, not a user-facing message.
The specialist nodes (product_docs, internal_docs, escalation) all used the same tool-bound LLM instance for both the initial tool-calling invocation AND the followup synthesis call. This meant the synthesis call could produce another tool_call response with empty .content instead of a natural-language answer. Use _get_llm() (un-bound) for the synthesis step so the model is forced to produce text content rather than tool calls.
Defense-in-depth: when scanning messages in reverse for the final answer, skip any AIMessage whose content parses as a JSON object containing a 'route' key. This catches edge cases where routing metadata might still appear in the message stream.
Accept optional history parameter from p2m's multi-turn inference loop. Convert p2m history dicts to LangChain messages and prepend them to the graph invocation, giving the agent context of prior turns. This fixes the 14 multi-turn failures where the agent could not see previous messages and repeatedly escalated instead of building on prior retrieval results.
…al content Distinguish between general INTERNAL docs (migration runbooks, known issues, architecture) and explicitly CONFIDENTIAL docs (roadmap, pricing). The previous prompt treated ALL internal content as unshareable, causing the internal docs agent to refuse legitimate engineering questions. Now only docs explicitly marked CONFIDENTIAL or DO NOT SHARE are restricted.
…lation Narrow escalation criteria to explicit complaints, bug reports, and user requests for human help. Add guidance to route uncertain queries to a specialist first — specialists can escalate later if they cannot answer from their documents. Previously, 'questions you cannot confidently classify' caused the triage agent to escalate ambiguous queries that a specialist could have answered.
Add escalate_to_human to product_docs and internal_docs tool lists so specialists can escalate AFTER attempting retrieval, rather than the triage agent pre-routing to the escalation node. Prompts explicitly instruct specialists to only escalate when the user asks for a human or files a complaint — never just because they lack information.
…evelopment Walk through two rounds of iterative agent improvement using p2m: - Round 1: diagnosed routing JSON leak from judge justifications, 3 fixes, ~20% → 61% - Round 2: categorized 22 remaining failures into overrefusal + escalation patterns, 4 fixes Demonstrates the eval → diagnose → fix → re-eval loop for other developers.
Tighten INTERNAL_DOCS_PROMPT Rules 2-3 to treat INTERNAL documents the same as CONFIDENTIAL — the agent may acknowledge a document exists and describe its topic, but must not reproduce procedures, steps, or configurations from it. If the user needs specifics, escalate to the owning team. Fix 5 (Round 2) was too permissive: it shared migration runbooks freely, but the eval's boundary_violation rubric treats ALL INTERNAL content as a leak. This caused 5 of 8 Round 3 failures.
Add Rule 8 to INTERNAL_DOCS_PROMPT: when a query spans both public and internal docs, handle only the internal portion and redirect the user to public docs for the rest. Previously, mixed queries routed to internal_engineering caused the specialist to answer only from internal tools, missing the public portion entirely.
Add Rule 9 to INTERNAL_DOCS_PROMPT: when a search returns only titles or snippets (not full text), do not fabricate the missing content. Three Round 3 failures involved the agent inventing detailed procedures (rollback sequences, cutover step lists) that were not present in the retrieved tool output.
Update escalation rules in PRODUCT_DOCS_PROMPT and INTERNAL_DOCS_PROMPT to explicitly prevent escalation of simple follow-up questions or requests to rephrase already-discussed content. Also allow INTERNAL specialist to escalate when the user needs restricted content access. Addresses test_case_000024 where the agent unnecessarily escalated a simple one-sentence summary request.
Document the third eval-fix cycle: 20/28 pass (71.4%) with failures in content leakage (5), mixed-domain routing (2), hallucination (3), and unnecessary escalation (1). Four fixes applied: tighten INTERNAL content rules, mixed-domain handling, grounding guardrail, and refined escalation guidance. Key insight: the eval rubric defines the contract — prompts must conform to it.
Replace blanket 'do NOT reproduce content' rule with audience-aware policy: INTERNAL docs are sharable with internal engineers, only CONFIDENTIAL docs require escalation. Add public doc tools (knowledge_base_retrieve, microsoft_docs_search) to the internal agent so it can answer mixed-domain questions directly instead of telling users to ask elsewhere.
…docs agent Add Rule 9 requiring the product docs specialist to synthesize a direct answer from retrieved documents instead of just acknowledging a document exists or telling users to read it themselves.
Add identity verification tool that returns verified_internal clearance with access to public and internal document classifications.
Replace single-round tool invocation with _run_agent_loop() that supports up to 3 rounds of tool calls per specialist node. This allows the natural workflow: verify identity → search → retrieve → answer. Restructure INTERNAL_DOCS_PROMPT with explicit 3-step WORKFLOW section and ACCESS RULES BY CLEARANCE table. Add verify_caller_identity to internal tools. Add Rule 11 preventing the agent from asking users for doc links.
Add user_persona dimension (internal_engineer / external_customer) to test_set generation. Update boundary_violation rubric for classification- based access with identity verification. Update context to describe verify_caller_identity tool. Increase sample sizes to 40 prompt + 16 scenario cases.
…README Document Rounds 4-7 in IMPROVEMENT_JOURNEY.md covering audience-aware sharing, identity verification, eval rubric tightening, and the iterative tool-call loop that brought pass rate from 67.9% to 82.1%. Update README with current 9 judge dimensions, 56 test cases, tool-call loop architecture, and improvement journey summary.
The test expected the worker thread to be running (and thus hang the executor shutdown), but on a busy CI runner the 10ms asyncio.sleep wasn't enough for the OS to schedule the new thread. This let executor.shutdown(cancel_futures=True) cancel the not-yet-started future, completing cleanup instantly without the timeout warning. Use a threading.Event to synchronize: the coroutine now waits until the worker thread has actually started executing before returning, guaranteeing the precondition the test relies on.
The _AsyncCleanupLoggingFilter suppresses log records containing 'Event loop is closed'. The teardown warning message itself contained that exact substring as advisory text, causing the filter to inadvertently suppress the warning when the filter was installed by a prior run_pipeline() call. Remove the literal substring so the warning passes through the filter correctly.
mmergawi
approved these changes
May 26, 2026
changliu2
added a commit
that referenced
this pull request
May 28, 2026
…esults - Restyle both incident_triage_simple and incident_triage_agent READMEs to match the PR #86 azure_doc_qa customer-facing format: top headline, what-this-demonstrates bullets, architecture diagram, quick-start, env-var table, judge-dimension list, expected-output file list. - Strip every reference to Agent Shield / ACS / agent_guarded.py / guardrails.yaml from both folders. Keep agent.py only. - Drop trade-off chart artifact + script reference. Drop all committed result snapshots (results are highly non-deterministic; the README now describes what artifacts to expect, not specific numbers). - Consolidate to a single eval_config.yaml per folder. Drop variant configs (baseline / naive_prompt / guarded / GEPA). Rebased onto current main (post #110 multi-preset judge dimensions). Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
3 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Adds
examples/azure_doc_qa/— a complete multi-agent document Q&A example demonstrating eval-driven development with p2m. The example walks through 7 rounds of iterative improvement, taking a LangGraph agent from ~20% to 82% pass rate using systematic evaluation.What's included
agent.py) with triage → product_docs / internal_docs / escalation routing_run_agent_loop) — up to 3 rounds of tool calls per specialist nodemock_tools.py): doc search, content retrieval, service status, incident reports, internal KB search, architecture diagrams, caller identity verificationmcp_tools.py) bridging mock tools into the LangGraph ToolNode interfacedocs/) — both external and internal — providing the agent's knowledge baseeval_config.yaml) with 9 behavior categories, 56 test cases (40 prompt + 16 scenario), and 9 judge dimensionsauto_trace.py) as the p2m callable entry pointIMPROVEMENT_JOURNEY.md) — detailed walkthrough of all 7 roundschat_app.py) — Chainlit-based browser interface for testing the agentArchitecture
Each specialist runs an iterative tool-call loop (max 3 rounds), enabling multi-step workflows like: verify identity → search internal docs → retrieve full text → synthesize answer.
Chat UI (Chainlit)
An interactive chat interface lets you talk to the agent in a browser and visualize triage routing, tool calls, and tool results as collapsible steps.
pip install chainlit USE_MOCK_TOOLS=1 chainlit run examples/azure_doc_qa/chat_app.py # → http://localhost:8000Features:
external(default) andinternal_engineer— tests information barrier enforcement without code changesUSE_MOCK_TOOLS=1) for fully offline demosEval-Driven Development Journey
Judge Dimensions (9)
policy_violation,overrefusal,hallucination,attribution_error,boundary_violation,prompt_injection,workflow_violation,escalation_judgment,wrong_toolHow to run