Add LangGraph Foundry hosted-agent example with Adaptive Eval

[ShilJain]

What this PR adds

A new self-contained example: examples/langgraph-foundry-hosted/ — a LangGraph travel-planner agent deployed as a Microsoft Foundry hosted agent (Responses protocol) and evaluated with Adaptive Eval (assert-ai).

It demonstrates the end-to-end loop for a hosted agent:

• A LangGraph StateGraph travel planner (intent classification → research → itinerary optimization / clarification) with simulated tools.
• Deployment to Foundry via azd (azure.yaml, agent.yaml, agent.manifest.yaml, Dockerfile).
• A callable eval target (auto_trace.py) that calls the live deployed agent over the Responses protocol, with the endpoint read from an environment variable.
• An eval_config.yaml that runs the Adaptive Eval pipeline (systematize → test_set → inference → judge) against the deployed agent.

Why

Existing examples cover local/callable and Prompt Agent targets. This adds a worked example for evaluating an agent that is already deployed as a Foundry hosted agent, which is a common production shape.

Notes / safety

• No secrets committed — all endpoints/identifiers are read from environment variables; only .env.example placeholders are included, and .env is gitignored.
• Synthetic/simulated tools only; no customer data.

Validation

Static sanity pass (deterministic)

Check	Result
YAML loads (agent.manifest.yaml, agent.yaml, azure.yaml, eval_config.yaml)	✅ all parsed
Python compiles (agent.py, main.py, auto_trace.py, export_foundry_traces.py)	✅ py_compile clean
Example deps install (requirements.txt)	✅ resolves (langgraph, langchain_azure_ai, azure-ai-projects, httpx, …)
LangGraph graph construction (agent.get_graph())	✅ CompiledStateGraph, nodes: intent_classifier, research, itinerary_optimizer, clarification
ResponsesHostServer import (main.py)	✅ imports in a clean example venv
Callable eval target (auto_trace:chat_sync)	✅ resolves, history param present (multi-turn support)

Live end-to-end run (deployed Foundry hosted agent)

Ran assert-ai run --config eval_config.yaml --force-stage inference --force-stage judge against the deployed hosted agent over the Responses protocol (endpoint via FOUNDRY_AGENT_ENDPOINT, AAD auth).

• Pipeline status: Completed — Inference ✅, Scoring ✅
• Suite / run: travel-planner-langgraph-v1 / demo-1
• Test cases: 10 generated (5 prompt + 5 scenario); 9 inference rows produced, 8 judged
• Token usage: 35 calls · 198.8K in / 9.7K out · 57.3% cached

Track	Total	Scored	Overrefusal	Policy violation	Judge failure
Prompt	4	4	0.0% (0/4)	50.0% (2/4)	0.0%
Scenario	5	4	25.0% (1/4)	100.0% (4/4)	20.0%

Artifacts written to artifacts/results/travel-planner-langgraph-v1/demo-1/ (scores.jsonl, inference_set.jsonl, metrics.json).

Known caveats (honest notes)

• 1 inference target error (HTTPStatusError, stop_reason=target_error) on a prompt case (demographic-stereotyping probe) — appears transient from the deployed agent, not a sample/code defect; rerunning that case is recommended.
• 1 judge failure on a scenario row (scenario judge-failure rate 20%), so 8 of 9 rows were scored.
• High policy-violation rates are expected signal from the adversarial suite (stereotyping / prompt-injection / sycophancy probes) — they flag behaviors to inspect in scores.jsonl, not a harness failure.
• On Windows, run evals from outside OneDrive (or pause OneDrive sync) — OneDrive file locks can break the final atomic artifacts.json write with PermissionError [WinError 5].

[jakepresent]

I took a pass and I think we should tighten a few things before merging, mostly because #252 landed now and this sample is still using the callable-wrapper path.

• Can we add a short note in the README explaining when to use this path vs the new native target.model: azure_ai/agents/<id> path from #252? My understanding is this example is still the right path for the Responses/v2/custom hosted-agent case, while azure_ai/agents/<asst_id> is now native for the v1 Assistants surface. Without that note, it may look like ASSERT still requires a wrapper for all Foundry hosted agents.
• The eval config uses azure/gpt-5.4 for systematize/test generation/tester/judge, so the README/env template should also document the ASSERT model auth required for those calls (AZURE_API_BASE plus key or AAD setup, depending on what was validated). Right now the README only lists the Foundry project/model/agent endpoint vars, while .env.example has an unexplained AZURE_API_BASE=.
• There are a few stale/mismatched sample values: agent.manifest.yaml still says get_current_time and calculator, the model default is gpt-5.1 in README/agent.py/manifest but gpt-5.4 in .env.example/azure.yaml/eval_config, and the viewer command should be cd ../../viewer && npm run dev if the user is following the previous cd examples/langgraph-foundry-hosted.
• The PR body points to a validation comment, but I don't see a comment or checks on the PR. Can you move that output into the PR body or add the comment so the validation is visible?

I did a local sanity pass: YAML loads, the Python files compile, the example deps install, the LangGraph/ResponsesHostServer imports and graph construction work, and the callable target resolves with history support. I didn't run the live end-to-end path since that needs a deployed Foundry agent endpoint.

[tangym]

• Can we add a short note in the README explaining when to use this path vs the new native target.model: azure_ai/agents/<id> path from feat(azure-auth): native AAD support for azure_ai/* targets (Foundry hosted agents) #252? My understanding is this example is still the right path for the Responses/v2/custom hosted-agent case, while azure_ai/agents/<asst_id> is now native for the v1 Assistants surface. Without that note, it may look like ASSERT still requires a wrapper for all Foundry hosted agents.

+1 to Jake's point. To add context: Responses v2 isn't supported by the native AAD path that landed in #252. I've filed issue #253 to track that gap. I would still like to see this PR (#250) land because it gives users an escape hatch for hosted agents: when the native target doesn't fit (Responses v2 today, other shapes tomorrow), they can always drop in a thin callable wrapper and keep evaluating.

[ShilJain]

Thanks for the thorough pass, Jake , Yeming! Addressed all four in 12fbffa:

• Callable vs native (feat(azure-auth): native AAD support for azure_ai/* targets (Foundry hosted agents) #252): Added a README note — this sample targets the Responses/v2 custom hosted-agent path, while target.model: azure_ai/agents/<asst_id> (feat(azure-auth): native AAD support for azure_ai/* targets (Foundry hosted agents) #252) is now native for the v1 Assistants surface. Clarified that ASSERT does not require a wrapper for all Foundry hosted agents.
• ASSERT model auth: README and .env.example now document the azure/gpt-5.4 pipeline credential — AZURE_API_BASE + AZURE_API_KEY, or the AAD path (ASSERT_AZURE_USE_AAD=1 with the Cognitive Services OpenAI User role). No more unexplained AZURE_API_BASE=.
• Stale/mismatched values: agent.manifest.yaml now lists the five travel tools (search_flights, search_hotels, check_weather, check_travel_advisories, validate_budget); standardized on gpt-5.4 across README/agent.py/manifest; viewer command is now cd ../../viewer && npm run dev.
• Validation visibility: Moved the output into the PR body — a static sanity pass plus a live end-to-end run against the deployed Foundry agent (suite travel-planner-langgraph-v1/demo-1). Noted the two transient errors (1 inference HTTP error, 1 judge failure) honestly rather than claiming an all-green pass.

Re-requesting your review.

[jakepresent]

Original comments are addressed. I re-ran the static pass: YAML loads, Python compiles, config loading works, the agent imports/graph construction work, and the callable resolves with history support.

One small PR-body cleanup before merge would be to remove export_foundry_traces.py from the validation line since that file is not in this PR, but that is not blocking from me.

[tangym]

Thanks Shilpa for the iteration and for contributing this example! All comments are addressed, merging now.