LangChain AI Agent Evaluation System

Build a refund support agent, trace its tool calls, and evaluate both its answers and its behavior with seven DeepEval metrics.

A correct final answer is only one part of a successful agent run. Did the agent identify the customer, check refund eligibility, select the right tools, and refund only the requested items?

This repository is a hands-on learning lab for answering those questions. It pairs a LangChain refund agent with eight evaluation scenarios, seven metrics, and small examples that isolate individual evaluation concepts. Start with the examples, follow the complete evaluation flow, then adapt it to your own agent.

• What you will learn
• How the agent works
• Quick start
• Run evaluations
• Evaluation metrics
• Evaluation dataset
• Read the results
• Try the chat demo
• Explore and extend
• Project map
• Limitations and troubleshooting

• Build a tool-using agent with a system prompt, conversation memory, and a structured response.
• Define evaluation cases with inputs, expected answers, and expected tools.
• Capture LangChain execution traces with DeepEval's CallbackHandler.
• Evaluate task completion, efficiency, planning, tool selection, tool arguments, and answer correctness.
• Use per-case explanations to decide whether to improve the agent, the dataset, or the evaluation itself.

The repository has two learning tracks:

The agent handles refund requests against the sample purchase records in agent/data.py. Its system prompt requires it to identify the customer, check that purchases are no more than 30 days old and are not in the future, and refund only the items requested.

The tools use in-memory sample data. Issuing a ticket returns a string; it does not contact a payment provider or create an external ticket. Eligibility checks are directed by the prompt rather than enforced inside the refund tools.

agent/agent.py creates the agent with an InMemorySaver checkpointer and the Pydantic Answer schema:

class Answer(BaseModel):
    answer: str
    conversation: bool = True

answer contains the user-facing reply. conversation indicates whether the agent needs another user response. Evaluation cases each receive a new thread ID to keep their conversation histories separate.

flowchart TD
    Dataset["Eight refund scenarios"] --> Runner["evaluations/evaluate_all.py"]
    Runner --> Agent["LangChain refund agent"]
    Agent <--> Tools["Purchase history · date · refund calculation · ticket"]
    Agent --> Response["Structured answer + messages and tool calls"]
    Agent -. "CallbackHandler" .-> Trace["Execution trace"]
    Trace --> TraceMetrics["Task completion · step efficiency<br/>plan quality · plan adherence"]
    Response --> TestCase["LLMTestCase + expected answer and tools"]
    TestCase --> OutputMetrics["Argument correctness · tool correctness<br/>answer correctness"]
    TraceMetrics --> Report["Scores, pass rates, and reasons"]
    OutputMetrics --> Report

You need Python 3.12+, uv, and an OpenAI API key with access to the models configured in the code.

git clone https://github.com/Mohamad-Hachem/LangChain_Agent_Evaluation_System.git
cd LangChain_Agent_Evaluation_System
uv sync --locked

Copy .env.example to .env in the repository root, then set:

OPENAI_API_KEY=your_openai_api_key_here

The .env file is ignored by Git. Keep your real key there.

The refund agent and its evaluator currently both use the model identifier gpt-6-luna. Check these two configuration files before running:

If that identifier is unavailable to your account, change each file to a model supported by its respective library and your API account. The standalone multiplication examples specify their own models inside their scripts.

Run commands from the repository root:

uv run --env-file .env python evaluations/evaluate_all.py

This runs the refund agent once per scenario, evaluates all seven metrics, and prints reports in the terminal. Agent generation and LLM-based judging make API calls and incur usage costs.

Check the dates before interpreting scores. The purchase records use fixed dates, while get_today_date reads your machine's current date. The expected answers are static. Align the sample dates and expected outcomes for your experiment; see Evaluation dataset.

evaluations/evaluate_all.py is the main entry point. It performs evaluation in two stages:

1. dataset.evals_iterator(...) runs four metrics against the execution trace captured by CallbackHandler.
2. The runner collects the same run's structured answer and tool calls into an LLMTestCase. evaluate(...) then applies the remaining three metrics.

Reusing the response avoids generating a different answer for each metric. The runner disables cache reads and writes and sets inspect_after_run=False, so it does not automatically launch the inspector. The terminal ends with:

Completed all 7 metrics for 8 test cases.

Completion means the evaluation finished; individual metrics can still fail. The script produces two evaluation reports, one for each stage, rather than a single combined export.

For small examples without the refund workflow:

uv run --env-file .env python understanding_metrics/tool_correctness.py
uv run --env-file .env python understanding_metrics/argument_correctness.py
uv run --env-file .env python understanding_metrics/all_metrics.py

For an individual metric applied to the refund agent:

uv run --env-file .env python evaluations/task_completion.py
uv run --env-file .env python evaluations/tool_correctness.py
uv run --env-file .env python evaluations/answer_correctness.py

Each individual script invokes the agent again. Its configuration can differ from the combined runner. In particular, evaluations/argument_correctness.py collects tool calls but omits them from its LLMTestCase; use the combined runner for refund argument evaluation.

These thresholds are defined in build_metrics() in evaluations/evaluate_all.py. Scores run from 0 to 1; a score at or above the threshold passes.

The refund dataset specifies expected tool names, without reference arguments. The combined runner also leaves exact matching and tool ordering at their defaults. Its tool correctness score therefore does not establish that arguments or execution order were correct. Inspect those separately.

The GEval rubric checks the logical conclusion against the reference answer. Read the reasons alongside the score to see whether details such as the refund amount were assessed as intended.

evaluations/dataset.py defines eight DeepEval Golden cases. A golden describes the desired behavior before the agent runs; an LLMTestCase adds the actual output and observed tool calls afterward.

These expected outcomes need a consistent date reference. Most sample purchases are dated October 2026. For example, on October 15, 2026, the October 22 book purchase is in the future, while its reference answer calls it more than 30 days old. A correct policy decision can therefore disagree with the reference answer.

For a meaningful experiment, align agent/data.py and the expected answers with the date you intend to test. For repeatable experiments, adapt the date tool to return a fixed reference date and record it with your results.

The screenshots below show an example run over eight cases. They illustrate how to diagnose failures; they are not a reproducible benchmark or a guarantee of current performance.

Average score and pass rate answer different questions. Argument correctness averages 0.62, but only two cases meet its 0.90 threshold. High step efficiency also does not establish that the agent made a good plan: inspect the plan quality failures and their traces together.

In this case, the agent reports that the customer could not be found. Tool correctness and answer correctness pass, while argument correctness fails. Read the captured arguments and the judge's reason before changing the agent: the failure may expose an agent mistake, missing evaluation evidence, or a rubric that needs refinement.

LLM judgments and agent responses can vary between runs. Keep the model choices, date reference, prompts, dataset, and thresholds with any results you compare.

agent/streamlit_ui.py provides a conversational refund interface with a reset button. Streamlit is not declared in pyproject.toml; add it for this command with uv:

uv run --env-file .env --with streamlit python -m streamlit run agent/streamlit_ui.py

Open the local URL printed in the terminal. The app begins with “I would like to request a refund.” Reply with a sample username and the items you want to return, such as I am user131. Please refund only my mouse. The conversation ends when the agent returns conversation=False.

The UI builds its own agent with three tools and does not include get_today_date. Use agent/agent.py and the evaluation runner when studying the four-tool agent's date checks. The UI displays chat messages; evaluation reports appear in the terminal.

A suggested reading order:

1. Read understanding_metrics/tool_correctness.py to see inputs, expected tools, actual tool calls, and scoring in one small example.
2. Explore agent/prompts.py, agent/tools.py, and agent/agent.py to understand the refund workflow.
3. Read evaluations/dataset.py to connect each business rule to an expected outcome.
4. Follow evaluations/evaluate_all.py from trace capture to test case construction and scoring.
5. Inspect a failed case, change one part of the system, and rerun with the same evaluation settings.

To evaluate your own agent, replace the sample data and tools, rewrite the system prompt, and define goldens for your task. Update SUCCESSFUL_TASK_DESCRIPTION and the GEval rubric in the combined runner so the judges assess your intended behavior.

Useful additions to the dataset include purchases exactly 30 days old, future purchases, mixed eligible and ineligible items, duplicate item requests, and attempts to bypass the refund policy. Add expected tool arguments when you want stronger reference checks, and enable order or exact-match checks only when those requirements are part of your task.

This is an educational prototype with a small, single-turn evaluation dataset. The interactive agent has memory, but the eight goldens do not test complete multi-turn conversations. Refund eligibility is prompt-driven, and the tools do not provide production validation or persistent storage. Unknown users or items passed directly to the calculation tool can raise lookup errors.

Start with the documented Python script commands. The package entry point declared in pyproject.toml points to a main function that is not currently implemented.