diff --git a/README.md b/README.md index 9ae5522..119406a 100644 --- a/README.md +++ b/README.md @@ -5,6 +5,26 @@ with [ThinkingBox](https://github.com/microsoft/thinkingbox). See the framework's [README](https://github.com/microsoft/thinkingbox#readme) for installation, configuration, and the `tb` CLI overview. +## ThinkingBox-Bench + +[ThinkingBox-Bench](releases/thinkingbox_bench_v1/README.md) is the primary +evaluation release in this repository. Version 1.0 contains 507 executable +tool-agent-user tasks across retail and e-commerce, travel and hospitality, +auto insurance, neobank support, and consulting IT/HR support. + +Each task runs in an isolated, stateful tool environment and is evaluated with +executable checks over the final backend state and side effects. Some tasks +also check required properties of the final response. + +If your goal is to run the published benchmark, follow the complete +[ThinkingBox-Bench v1.0 installation and run instructions](releases/thinkingbox_bench_v1/README.md#run-the-benchmark) +directly. The remaining setup and examples in this README are intended for +customized ThinkingBox development, individual scenarios, and smoke testing. + +The rest of this repository also contains individual datasets and development +fixtures that are not part of ThinkingBox-Bench. The benchmark's canonical task +set is defined by the test list linked from its release documentation. + ## Contents - **`dataset/`** — scenarios, agents, and test cases. @@ -13,7 +33,8 @@ installation, configuration, and the `tb` CLI overview. `tb mcp-start`. - **`support/`** — large data files used by some tools (embeddings, knowledge bases). Set `THINKINGBOX_DATA=` so tools can locate them. -- **`releases/`** — per-release dataset snapshots pinned to git tags. +- **[`releases/`](releases/README.md)** — supported benchmark releases and + their canonical test lists. ## Layout @@ -30,18 +51,13 @@ parent/ ## Setup Install the framework first (see the [thinkingbox -README](https://github.com/microsoft/thinkingbox#readme)). Then, still from -`thinkingbox/`, install the server packages from this repo into the same env: +README](https://github.com/microsoft/thinkingbox#readme)). For the smoke tests +below, install the `thinkingbox_tools` package into the same environment: ```bash uv pip install --config-settings editable-mode=compat -e ../thinkingbox-data/servers/thinkingbox_tools -uv pip install --config-settings editable-mode=compat -e ../thinkingbox-data/servers/tb_business_ops_servers_202606 ``` -Some tools also need extra services (e.g. Typesense, embeddings server) — see -[`tools_with_additional_setup.md`](https://github.com/microsoft/thinkingbox/blob/main/docs/tools_with_additional_setup.md) -in the framework repo. - ## Verify your setup Before running larger scenarios, sanity-check that the framework, the @@ -106,28 +122,12 @@ If `tb pp` shows a successful conversation or `tb agg` reports passing assertions, the framework, server packages, and LLM endpoint are all wired up. -## Run larger scenarios - -To run dataset_2602_external_retail and dataset_2603_sandbox_rl_zendesk, -you need to also start typesense server. - -All commands below assume you are in the `thinkingbox/` directory. +## Run ThinkingBox-Bench -``` -export THINKINGBOX_DATA="../thinkingbox-data/dataset" -export TB_MCP_START_SERVERS_FILE=../thinkingbox-data/servers/servers.yaml -./scripts/background_tasks.sh -``` - -```bash -uv run tb infer -c config/config_o4mini.yaml \ - --dataset ../thinkingbox-data/dataset --agent think \ - --test-list ../thinkingbox-data/releases/dataset_2602_external_retail/testlist_2602_external_retail_full100.yaml \ - --repeat 10 --batch-size 40 \ - --output output_sandbox_external_retail_10reps.jsonl -``` +Use the canonical +[ThinkingBox-Bench v1.0 installation and run instructions](releases/thinkingbox_bench_v1/README.md#run-the-benchmark). -### Re-run assertions on a saved test context +## Re-run assertions on a saved test context After decoding once, re-run just the test assertions (no LLM calls): @@ -145,7 +145,7 @@ uv run tb run-test -c config/config_o4mini.yaml \ --resultfile output.yaml --update ``` -### Interactive TUI +## Interactive TUI Chat with a scenario: diff --git a/releases/README.md b/releases/README.md new file mode 100644 index 0000000..ab8ff95 --- /dev/null +++ b/releases/README.md @@ -0,0 +1,13 @@ +# Releases + +This repository publishes two supported evaluation releases: + +| Release | Tasks | Description | +| - | -: | - | +| [ThinkingBox-Bench v1.0](thinkingbox_bench_v1/README.md) | 507 | Stateful business workflows across five domains | +| [Airline Tau-Bench 26-04](dataset_2604_airline_tau_bench/README.md) | 21 | Microsoft-adapted Tau-Bench airline evaluation | + +Each release directory documents its intended use, canonical test list, +evaluation method, prerequisites, and execution command. Other files under +`dataset/` support development or individual scenarios and are not separate +published releases. diff --git a/releases/dataset_2602_external_retail/README.md b/releases/dataset_2602_external_retail/README.md deleted file mode 100644 index a80400d..0000000 --- a/releases/dataset_2602_external_retail/README.md +++ /dev/null @@ -1,171 +0,0 @@ -# External Retail 26-02 - -## Repository - -Use the following tag of `thinkingbox-data` to use a stable version of this dataset and tools. - -| | | -| - | - | -| Tag | [ds-external-retail-2026-02-v1.0](https://github.com/microsoft/thinkingbox-data/releases/tag/ds-external-retail-2026-02-v1.0) | - - -## Overview - -**Dataset name**: External Retail 26-02 - -**Test cases**: 100 - -System: Zendesk - -Domain: Retail & E-commerce - -Business function: Customer Support & Service - -Agent name: B2C Ecommerce Electronics Support Agent (External) - -The agent operates as a customer service representative for **TechHome Direct**, a fictitious online electronics and appliances retailer. It handles consumer inquiries about orders, deliveries, returns, warranty claims, installation services, payment disputes, order modifications, and membership management through a Zendesk-based ticketing system. - -## Intended use - -This dataset is designed **exclusively for evaluation** of LLM agent capabilities. It provides an unbiased benchmark for measuring how well agents handle realistic, multi-step customer service workflows with tool use. - -**This dataset MUST NOT be used for:** -- Prompt tuning or prompt optimization -- Fine-tuning or training language models -- Reinforcement learning or reward model training -- Any form of optimization that uses test case content, expected outcomes, or golden tool interactions as a training signal - -Using this dataset for training or tuning would compromise its value as an independent evaluation benchmark. The dataset is intended to represent an unbiased assessment of agent performance; any model optimized against it would produce inflated scores that do not reflect genuine capability improvements. - -## What the dataset covers - -The 100 test cases simulate realistic B2C customer support interactions across the full lifecycle of an e-commerce customer relationship. Each test case provides a customer query, a simulated user persona with context, a pre-populated database state, and a golden expected outcome (the sequence of tool calls and their parameters that constitute a correct resolution). - -### Topic distribution - -| Category | Stories | Cases | What it tests | -|---|---|---|---| -| **Returns** | ST006, ST008, ST009, ST010 | 40 | Return eligibility by tier/window/product category, fee calculations (restocking, shipping, removal), RMA creation, returns with installation cancellation, ineligible return handling, pre-delivery return attempts | -| **Defects & Warranty** | ST007, ST012, ST013, ST014 | 14 | Warranty claim filing, routing between return and warranty paths, protection plan vs manufacturer warranty, minor defect partial refunds with score-based compensation | -| **Installation Services** | ST015, ST016, ST017, ST018 | 10 | Customer-requested rescheduling, weather delay rescheduling with compensation, workmanship warranty issues, installation cancellation with shipping cost charge | -| **Order Tracking & Delivery** | ST002, ST003 | 7 | Delayed shipment compensation (score-based), delivery exception handling, ticket reuse for follow-ups | -| **Undelivered Packages** | ST004, ST005 | 9 | Missing package investigation, courtesy replacements (tier/score-based), carrier investigation, returned-to-sender reship with fault determination | -| **Order Modifications** | ST025, ST026, ST028, ST029 | 11 | Order cancellation (with and without installation), shipping address changes within modification window, promo code after purchase | -| **Membership Management** | ST030, ST031 | 5 | Plus membership upgrades and cancellations | -| **Exchanges** | ST011 | 4 | Same-SKU exchanges for defective items, inventory availability checks | - -### Complexity dimensions - -The test cases vary along several dimensions that affect difficulty: - -- **Customer tiers** (Standard, Plus, VIP) determine return windows, fee waivers, shipping speeds, and ticket priority -- **Customer scores** (Regular, Opportunist, Bonus Hunter) determine compensation amounts and fee waivers, and must never be disclosed to the customer -- **Policy lookups**: the agent's system prompt references policies but does not contain specific fee amounts, return windows, or warranty periods. The agent must search the knowledge base at runtime to retrieve these values and apply them correctly. -- **Multi-step workflows**: most cases require multiple tool calls in sequence (e.g., look up customer profile, find or create a Zendesk ticket, retrieve order details, search policies, calculate fees, create an RMA, update ticket status) -- **Ticket management**: the agent must decide whether to create a new ticket, reopen a recently solved one, or reuse an existing open ticket, and must set the correct status, priority, and type based on the customer's tier and issue -- **User interaction**: 35 of the 100 test cases do not include all required information in the initial query. The agent must ask the customer for missing details (email, order ID, preferences) before proceeding. A simulated user with the required context is provided for each test case. -- **Edge cases**: first-time customer courtesy waivers, product category-specific return windows (computing, gaming, and wearables have reduced windows for Standard customers), installed appliance removal fees - -### Evaluation method - -Each test case is evaluated by comparing the final database state after the agent's execution against a golden expected state. This is a deterministic, hash-based comparison: the agent either produced exactly the correct set of side effects (created the right tickets, RMAs, refunds, etc. with the correct parameters) or it did not. There is no partial credit. - -## Scenario - -All test cases refer to scenario `dataset/scenario/sandbox_external_retail.yaml`, which provides the agent with: -- System instructions, defining the agent's role -- Knowledge base articles: 10 policy documents searchable at runtime - -The agent has access to 33 tools spanning several systems. - -## Test cases - -The dataset is made of 100 test cases, found in `dataset/test_case/sandbox_external_retail/`. - -Some interaction with the user is expected in most test cases, to retrieve missing information that is critical for the completion of the task. A prompt for a simulated user, including the required additional context, is provided for each test case. - -Two splits are provided as YAML files in this directory: - -| Split | File | Cases | Description | -|---|---|---|---| -| **full100** | `testlist_2602_external_retail_full100.yaml` | 100 | All test cases | -| **quick20** | `testlist_2602_external_retail_quick20.yaml` | 20 | A subset of 20 test cases | - -The quick20 split is designed to track the full100 performance, and to be used as proxy for faster iteration. - -## Evaluation results - -All results are pass@k over 20 repetitions. - -Agent: -- Temperature: 1.0 -- No seed -- Reasoning effort (reasoning models only): medium - -User simulator: GPT-5 Chat - -### full100 - -| Model | Orchestrator | pass@1 | pass@5 | -|---|---|---|---| -| Opus4.6 | ThinkingBox | 0.73 | 0.86 | -| BIC RL FT GPT-5 mini Medium Reasoning | ThinkingBox | 0.70 | 0.89 | -| GPT 5.2 reasoning medium | ThinkingBox | 0.69 | 0.85 | -| Sonnet4.6 | ThinkingBox | 0.68 | 0.84 | -| Opus4.7 | ThinkingBox | 0.61 | 0.76 | -| GPT 5 reasoning medium | ThinkingBox | 0.58 | 0.83 | -| Sonnet4.5 | ThinkingBox | 0.52 | 0.82 | -| GPT 5.2 reasoning medium | MCS | 0.47 | 0.73 | -| GPT-5 mini reasoning medium | ThinkingBox | 0.37 | 0.77 | -| GPT-4.1 | ThinkingBox | 0.16 | 0.42 | -| GPT-5-Chat | ThinkingBox | 0.03 | 0.11 | -| GPT-4.1 | MCS | 0.01 | 0.06 | - -### quick20 - -| Model | Orchestrator | pass@1 | pass@5 | -|---|---|---|---| -| GPT 5.2 reasoning medium | ThinkingBox | 0.69 | 0.86 | -| GPT-4.1 | ThinkingBox | 0.20 | 0.51 | -| GPT-5 mini reasoning medium | ThinkingBox | 0.39 | 0.82 | -| t11_24_25_5mini_rm medium (FT) | ThinkingBox | 0.74 | 0.92 | -| GPT-4.1 | MCS | 0.02 | 0.08 | -| GPT-5.2 reasoning medium | MCS | 0.38 | 0.72 | - -### Notes - -Results on `t11_24_25_5mini_rm` refer to a run on an internal deployment of the fine-tuned model. There is often a small difference when re-evaluating on the public deployment after publishing, which we cannot quantify at this time. - -MCS tests were run with the following parameters: - -- GPT-4.1: - - useModelKnowledge: false - - model: CurrentModels/GPT41, routed to AOAI GPT4.1, with overrides (temperature=1.0, seed=None) -- GPT-5.2 reasoning - - useModelKnowledge: false - - model: ReasoningExperimentalModels/GPT5Reasoning, routed to AOAI GPT-5.2, with overrides (temperature=1.0, reasoning=medium) - -## Running - -Check `thinkingbox/README.md` for installing ThinkingBox - -```bash -# Install the servers in the ThinkingBox virtual environment -uv pip install --config-settings editable-mode=compat -e servers/tb_business_ops_servers_202606 - -# Start typesense (default port, key: Fake) -mkdir -p /tmp/typesense/data && typesense-server --data-dir="/tmp/typesense/data" --api-key="Fake" --enable-cors - -# Start session proxy -THINKINGBOX_DATA=thinkingbox-data tb mcp-start --servers servers.yaml - -# Decode (full100, 20 repetitions) -tb infer -c config.yaml -d thinkingbox-data/dataset -a think \ - --inputs thinkingbox-data/dataset/test_case/sandbox_external_retail/ \ - --repeat 20 --batch-size 40 -o output_zendesk_external_retail_full100.jsonl - -# Decode (quick20, 5 repetitions) -tb infer -c config.yaml -d thinkingbox-data/dataset -a think \ - --test-list thinkingbox-data/releases/dataset_2602_external_retail/testlist_2602_external_retail_quick20.yaml \ - --repeat 5 --batch-size 40 -o output_zendesk_external_retail_quick20.jsonl -``` diff --git a/releases/dataset_2602_external_retail/testlist_2602_external_retail_full100.yaml b/releases/dataset_2602_external_retail/testlist_2602_external_retail_full100.yaml deleted file mode 100644 index 7a21e1f..0000000 --- a/releases/dataset_2602_external_retail/testlist_2602_external_retail_full100.yaml +++ /dev/null @@ -1,98 +0,0 @@ -- sandbox_external_retail_group1.py:test_case_ST002_001 -- sandbox_external_retail_group1.py:test_case_ST002_003 -- sandbox_external_retail_group1.py:test_case_ST002_004 -- sandbox_external_retail_group1.py:test_case_ST003_003 -- sandbox_external_retail_group1.py:test_case_ST003_004 -- sandbox_external_retail_group1.py:test_case_ST003_006 -- sandbox_external_retail_group1.py:test_case_ST004_001 -- sandbox_external_retail_group1.py:test_case_ST004_003 -- sandbox_external_retail_group1.py:test_case_ST004_004 -- sandbox_external_retail_group1.py:test_case_ST004_005 -- sandbox_external_retail_group1.py:test_case_ST004_007 -- sandbox_external_retail_group1.py:test_case_ST004_008 -- sandbox_external_retail_group1.py:test_case_ST004_009 -- sandbox_external_retail_group1.py:test_case_ST005_001 -- sandbox_external_retail_group1.py:test_case_ST005_005 -- sandbox_external_retail_group1.py:test_case_ST006_001 -- sandbox_external_retail_group1.py:test_case_ST006_002 -- sandbox_external_retail_group1.py:test_case_ST006_003 -- sandbox_external_retail_group1.py:test_case_ST006_004 -- sandbox_external_retail_group1.py:test_case_ST006_005 -- sandbox_external_retail_group1.py:test_case_ST006_006 -- sandbox_external_retail_group1.py:test_case_ST006_008 -- sandbox_external_retail_group1.py:test_case_ST006_010 -- sandbox_external_retail_group1.py:test_case_ST006_011 -- sandbox_external_retail_group1.py:test_case_ST006_015 -- sandbox_external_retail_group1.py:test_case_ST006_016 -- sandbox_external_retail_group1.py:test_case_ST006_017 -- sandbox_external_retail_group1.py:test_case_ST006_019 -- sandbox_external_retail_group1.py:test_case_ST006_021 -- sandbox_external_retail_group1.py:test_case_ST006_022 -- sandbox_external_retail_group1.py:test_case_ST006_023 -- sandbox_external_retail_group1.py:test_case_ST006_024 -- sandbox_external_retail_group1.py:test_case_ST006_025 -- sandbox_external_retail_group1.py:test_case_ST006_026 -- sandbox_external_retail_group1.py:test_case_ST006_028 -- sandbox_external_retail_group1.py:test_case_ST006_030 -- sandbox_external_retail_group1.py:test_case_ST006_031 -- sandbox_external_retail_group1.py:test_case_ST006_032 -- sandbox_external_retail_group1.py:test_case_ST006_033 -- sandbox_external_retail_group1.py:test_case_ST006_034 -- sandbox_external_retail_group1.py:test_case_ST006_037 -- sandbox_external_retail_group1.py:test_case_ST006_039 -- sandbox_external_retail_group1.py:test_case_ST006_040 -- sandbox_external_retail_group1.py:test_case_ST006_041 -- sandbox_external_retail_group1.py:test_case_ST006_042 -- sandbox_external_retail_group1.py:test_case_ST006_043 -- sandbox_external_retail_group1.py:test_case_ST006_044 -- sandbox_external_retail_group1.py:test_case_ST006_045 -- sandbox_external_retail_group1.py:test_case_ST006_047 -- sandbox_external_retail_group1.py:test_case_ST006_049 -- sandbox_external_retail_group1.py:test_case_ST007_003 -- sandbox_external_retail_group1.py:test_case_ST007_005 -- sandbox_external_retail_group1.py:test_case_ST007_006 -- sandbox_external_retail_group1.py:test_case_ST007_008 -- sandbox_external_retail_group1.py:test_case_ST008_003 -- sandbox_external_retail_group1.py:test_case_ST009_001 -- sandbox_external_retail_group1.py:test_case_ST009_004 -- sandbox_external_retail_group1.py:test_case_ST009_005 -- sandbox_external_retail_group1.py:test_case_ST010_001 -- sandbox_external_retail_group1.py:test_case_ST011_002 -- sandbox_external_retail_group1.py:test_case_ST011_006 -- sandbox_external_retail_group1.py:test_case_ST011_007 -- sandbox_external_retail_group1.py:test_case_ST011_009 -- sandbox_external_retail_group1.py:test_case_ST012_003 -- sandbox_external_retail_group1.py:test_case_ST013_001 -- sandbox_external_retail_group1.py:test_case_ST013_003 -- sandbox_external_retail_group1.py:test_case_ST013_004 -- sandbox_external_retail_group1.py:test_case_ST013_005 -- sandbox_external_retail_group1.py:test_case_ST013_006 -- sandbox_external_retail_group1.py:test_case_ST013_007 -- sandbox_external_retail_group1.py:test_case_ST013_008 -- sandbox_external_retail_group1.py:test_case_ST014_002 -- sandbox_external_retail_group1.py:test_case_ST014_005 -- sandbox_external_retail_group1.py:test_case_ST015_001 -- sandbox_external_retail_group1.py:test_case_ST016_004 -- sandbox_external_retail_group1.py:test_case_ST016_006 -- sandbox_external_retail_group1.py:test_case_ST017_001 -- sandbox_external_retail_group1.py:test_case_ST017_002 -- sandbox_external_retail_group1.py:test_case_ST017_003 -- sandbox_external_retail_group1.py:test_case_ST017_005 -- sandbox_external_retail_group1.py:test_case_ST018_002 -- sandbox_external_retail_group1.py:test_case_ST018_003 -- sandbox_external_retail_group1.py:test_case_ST018_005 -- sandbox_external_retail_group1.py:test_case_ST025_001 -- sandbox_external_retail_group1.py:test_case_ST025_002 -- sandbox_external_retail_group1.py:test_case_ST025_003 -- sandbox_external_retail_group1.py:test_case_ST025_004 -- sandbox_external_retail_group1.py:test_case_ST025_006 -- sandbox_external_retail_group1.py:test_case_ST026_002 -- sandbox_external_retail_group1.py:test_case_ST026_003 -- sandbox_external_retail_group1.py:test_case_ST028_003 -- sandbox_external_retail_group1.py:test_case_ST029_001 -- sandbox_external_retail_group1.py:test_case_ST029_002 -- sandbox_external_retail_group1.py:test_case_ST030_001 -- sandbox_external_retail_group1.py:test_case_ST030_002 -- sandbox_external_retail_group1.py:test_case_ST030_003 -- sandbox_external_retail_group1.py:test_case_ST031_001 -- sandbox_external_retail_group1.py:test_case_ST031_002 diff --git a/releases/dataset_2602_external_retail/testlist_2602_external_retail_quick20.yaml b/releases/dataset_2602_external_retail/testlist_2602_external_retail_quick20.yaml deleted file mode 100755 index 38796b3..0000000 --- a/releases/dataset_2602_external_retail/testlist_2602_external_retail_quick20.yaml +++ /dev/null @@ -1,19 +0,0 @@ -- sandbox_external_retail_group1.py:test_case_ST002_003 -- sandbox_external_retail_group1.py:test_case_ST025_004 -- sandbox_external_retail_group1.py:test_case_ST003_006 -- sandbox_external_retail_group1.py:test_case_ST026_003 -- sandbox_external_retail_group1.py:test_case_ST006_033 -- sandbox_external_retail_group1.py:test_case_ST009_001 -- sandbox_external_retail_group1.py:test_case_ST025_002 -- sandbox_external_retail_group1.py:test_case_ST013_005 -- sandbox_external_retail_group1.py:test_case_ST008_003 -- sandbox_external_retail_group1.py:test_case_ST006_008 -- sandbox_external_retail_group1.py:test_case_ST006_019 -- sandbox_external_retail_group1.py:test_case_ST029_001 -- sandbox_external_retail_group1.py:test_case_ST006_023 -- sandbox_external_retail_group1.py:test_case_ST005_001 -- sandbox_external_retail_group1.py:test_case_ST006_025 -- sandbox_external_retail_group1.py:test_case_ST013_004 -- sandbox_external_retail_group1.py:test_case_ST029_002 -- sandbox_external_retail_group1.py:test_case_ST030_001 -- sandbox_external_retail_group1.py:test_case_ST005_005 diff --git a/releases/dataset_2603_sandbox_rl_zendesk/README.md b/releases/dataset_2603_sandbox_rl_zendesk/README.md deleted file mode 100644 index 5451061..0000000 --- a/releases/dataset_2603_sandbox_rl_zendesk/README.md +++ /dev/null @@ -1,126 +0,0 @@ -# Sandbox RL Zendesk 26-03 - -## Repository - -Use the following tag of `thinkingbox-data` to use a stable version of this dataset and tools. - -| | | -| - | - | -| Tag | [ds-sandbox-rl-zendesk-2026-03-v1.0](https://github.com/microsoft/thinkingbox-data/releases/tag/ds-sandbox-rl-zendesk-2026-03-v1.0) | - - -## Overview - -**Dataset name**: Sandbox RL Zendesk 26-03 - -**Test cases**: 516 - -**Agents**: 5 - -| Agent | System | Domain | Business function | -| - | - | - | - | -| a01 | Zendesk | Retail & E-commerce | Customer Support & Service | -| a02r | Zendesk | Travel & Hospitality | Customer Support & Service | -| a03 | Zendesk | Insurance | Customer Support & Service | -| a04r | Zendesk | Banking & Finance | IT & Technical Support | -| a05 | Zendesk | Consulting Services | IT & Technical Support | - -### Agent descriptions - -**a01** - -- Name: B2C Ecommerce Electronics Support Agent (External) -- Num Tests: 100 -- Description: Handles consumer inquiries about orders, deliveries, returns, and warranty - -**a02r** - -- Name: Hotel Booking Support & Reservation Management Agent (External) -- Num Tests: 104 -- Description: Assists guests with reservations, modifications, cancellations, and billing - -**a03** - -- Name: Car Insurance Policy & Claims Service Agent (External) -- Num Tests: 101 -- Description: Supports policy changes, FNOL intake, claim status inquiries, and documentation - -**a04r** - -- Name: Internal Neobank IT Support Desk (IT Service Agent) -- Num Tests: 104 -- Description: Handles internal employee how-to requests, access and end-user hardware providing, and system issues - -**a05** - -- Name: Internal Consulting Helpdesk Agent (IT, HR, Expense Support) -- Num Tests: 107 -- Description: Supports consultants with equipment issues, system access, travel/expense inquiries - - -## Intended use - -This dataset is designed **exclusively for evaluation** of LLM agent capabilities. It provides an unbiased benchmark for measuring how well agents handle realistic, multi-step customer service workflows with tool use. - -**This dataset MUST NOT be used for:** -- Prompt tuning or prompt optimization -- Fine-tuning or training language models -- Reinforcement learning or reward model training -- Any form of optimization that uses test case content, expected outcomes, or golden tool interactions as a training signal - -Using this dataset for training or tuning would compromise its value as an independent evaluation benchmark. The dataset is intended to represent an unbiased assessment of agent performance; any model optimized against it would produce inflated scores that do not reflect genuine capability improvements. - -### Evaluation method - -Each test case is evaluated by comparing the final database state after the agent's execution against a golden expected state. This is a deterministic, hash-based comparison: the agent either produced exactly the correct set of side effects or it did not. - -In addition, 30 out of 516 test cases have simple rubrics evaluated by a LLM judge, in the form of yes/no questions. - -Tests pass if the final state is correct and all rubrics evaluate to the expected result. There is no partial credit. - -## Evaluation results - -All results are pass@k over 20 repetitions. - -Agent: -- Temperature: 1.0 -- No seed -- Reasoning effort (reasoning models only): medium - -User simulator and Judge: GPT-5 Chat - -### ds-sandbox-rl-zendesk-2026-03-v1.0 - -| Model | Orchestrator | pass@1 | pass@5 | -|---|---|---|---| -| GPT o4-mini reasoning medium | ThinkingBox | 0.09 | 0.28 | -| t11_24_25_varset_3_rm (o4-mini FT) reasoning medium | ThinkingBox | 0.37 | 0.71 | -| GPT 5-mini reasoning medium | ThinkingBox | 0.19 | 0.41 | -| GPT 5.2 reasoning medium | ThinkingBox | 0.46 | 0.71 | -| GPT 5.4 reasoning medium | ThinkingBox | 0.64 | 0.83 | -| Claude Sonnet 4.6 reasoning medium | ThinkingBox | 0.57 | 0.78 | -| Claude Opus 4.6 reasoning medium | ThinkingBox | 0.38 | 0.57 | - -### Notes - -Results on `t11_24_25_varset_3_rm` refer to a run on the PPE CAPI deployment of the fine-tuned model. - -## Running - -Check `thinkingbox/README.md` for installing ThinkingBox - -```bash -# Install the Sandbox RL servers in the ThinkingBox virtual environment -uv pip install --config-settings editable-mode=compat -e servers/tb_business_ops_servers_202606 - -# Start typesense (default port, key: Fake) -mkdir -p /tmp/typesense/data && typesense-server --data-dir="/tmp/typesense/data" --api-key="Fake" --enable-cors - -# Start session proxy -THINKINGBOX_DATA=thinkingbox-data tb mcp-start --servers thinkingbox-data/servers/servers.yaml - -# Decode (full dataset, 5 repetitions) -tb infer -c config.yaml -d thinkingbox-data/dataset -a think \ - --test-list thinkingbox-data/releases/dataset_2603_sandbox_rl_zendesk/testlist_2603_sandbox_rl_zendesk.yaml \ - --repeat 5 --batch-size 40 -o output_2603_sandbox_rl_zendesk.jsonl -``` diff --git a/releases/dataset_2604_airline_tau_bench/README.md b/releases/dataset_2604_airline_tau_bench/README.md index e6d66f6..d439836 100644 --- a/releases/dataset_2604_airline_tau_bench/README.md +++ b/releases/dataset_2604_airline_tau_bench/README.md @@ -6,7 +6,7 @@ Use the following tag of `thinkingbox-data` to use a stable version of this data | | | | - | - | -| Tag | [ds-airline-tau-bench-2026-04-v1.0](https://github.com/microsoft/thinkingbox-data/releases/tag/ds-airline-tau-bench-2026-04-v1.0) | +| Tag | [`ds-airline-tau-bench-2026-04-v1.0`](https://github.com/microsoft/thinkingbox-data/releases/tag/ds-airline-tau-bench-2026-04-v1.0) | ## Overview @@ -78,16 +78,20 @@ Some interaction with the user is expected in most test cases, to retrieve missi ## Running Check `thinkingbox/README.md` for installing ThinkingBox. +The commands below assume `thinkingbox/` and `thinkingbox-data/` are cloned +side-by-side and are run from the `thinkingbox/` directory. ```bash # Install the airline tau-bench server in the ThinkingBox virtual environment -uv pip install --config-settings editable-mode=compat -e servers/thinkingbox_tools +uv pip install --config-settings editable-mode=compat \ + -e ../thinkingbox-data/servers/thinkingbox_tools # Start session proxy -THINKINGBOX_DATA=thinkingbox-data tb mcp-start --servers thinkingbox-data/servers/servers.yaml +THINKINGBOX_DATA=../thinkingbox-data \ + tb mcp-start --servers ../thinkingbox-data/servers/servers.yaml # Decode (full dataset, 5 repetitions) -tb infer -c config.yaml -d thinkingbox-data/dataset -a think \ - --test-list thinkingbox-data/releases/dataset_2604_airline_tau_bench/testlist_2604_airline_tau_bench.yaml \ +tb infer -c config.yaml -d ../thinkingbox-data/dataset -a think \ + --test-list ../thinkingbox-data/releases/dataset_2604_airline_tau_bench/testlist_2604_airline_tau_bench.yaml \ --repeat 5 --batch-size 40 -o output_2604_airline_tau_bench.jsonl ``` diff --git a/releases/thinkingbox_bench_v1/README.md b/releases/thinkingbox_bench_v1/README.md new file mode 100644 index 0000000..d75a864 --- /dev/null +++ b/releases/thinkingbox_bench_v1/README.md @@ -0,0 +1,163 @@ +# ThinkingBox-Bench v1.0 + +## Repository + +Use the following `thinkingbox-data` tag for a stable version of the benchmark, +tool servers, and supporting data: + +| | | +| - | - | +| Tag | [`thinkingbox-bench-v1.0`](https://github.com/microsoft/thinkingbox-data/releases/tag/thinkingbox-bench-v1.0) | + +ThinkingBox-Bench is an executable benchmark for evaluating whether tool-using +LLM agents can reliably complete stateful business workflows. Version 1.0 +contains 507 tool-agent-user tasks across five domains. + +Each task provides an initial backend state, a user goal and simulated-user +context, domain tools, policy constraints, and executable checks. An attempt +passes only when all required checks over the final state, side effects, and +designated dialogue properties pass. + +## Release contents + +| Domain | Canonical definitions | Tasks | +| - | - | -: | +| Retail and e-commerce | `sandbox_external_retail_group1.py` | 98 | +| Travel and hospitality | `external_booking_v1_group1.py` (89) and `external_booking_v1_group1_rubrics_yesno.py` (15) | 104 | +| Auto insurance | `sandbox_auto_insurance_group1.py` | 100 | +| Neobank support | `sandbox_neobank_support_v1_group1.py` (89) and `sandbox_neobank_support_v1_group1_rubrics_yesno.py` (15) | 104 | +| Consulting IT/HR support | `sandbox_consulting_group1.py` | 101 | +| **Total** | | **507** | + +The canonical task set is +[`testlist_thinkingbox_bench_v1.yaml`](testlist_thinkingbox_bench_v1.yaml). +The test list, rather than every test definition under `dataset/test_case/`, +defines the release. It selects rubric-enhanced alternatives for 15 travel +tasks and 15 neobank tasks in place of their corresponding base definitions, +so these alternatives do not increase the task count. + +Other datasets and tests in this repository are not part of ThinkingBox-Bench +v1.0. + +## Evaluation method + +Each task compares the final backend state with its golden expected state using +deterministic, hash-based checks. Thirty tasks additionally use simple yes/no +rubrics evaluated by an LLM judge. A task passes only when its final state is +correct and every applicable rubric returns the expected result; there is no +partial credit. + +## Intended use + +ThinkingBox-Bench v1.0 is intended exclusively for evaluation. Do not use its +task content, expected outcomes, golden state, or tool trajectories for prompt +optimization, fine-tuning, reinforcement learning, reward-model training, or +other model optimization. + +## Run the benchmark + +ThinkingBox-Bench requires: + +- Python 3.12 and [`uv`](https://docs.astral.sh/uv/) +- a Linux or WSL environment +- ThinkingBox and `thinkingbox-data` cloned side-by-side +- Typesense 30.1, installed by the ThinkingBox installation script + +On Ubuntu or WSL Ubuntu, install the required system tools: + +```bash +sudo apt-get update +sudo apt-get install -y git curl tar coreutils procps +``` + +Install `uv` if it is not already available: + +```bash +curl -LsSf https://astral.sh/uv/install.sh | sh +source "$HOME/.local/bin/env" +``` + +It does **not** require the embeddings server, downloaded Hugging Face models, +or pre-indexed Typesense snapshots. Each benchmark scenario initializes its +own Typesense collections from the sources in its scenario definition. + +Clone and install ThinkingBox: + +```bash +git clone https://github.com/microsoft/thinkingbox.git +git clone https://github.com/microsoft/thinkingbox-data.git + +cd thinkingbox-data +git checkout thinkingbox-bench-v1.0 +cd ../thinkingbox + +uv venv --python 3.12 +uv sync --group dev +source .venv/bin/activate +``` + +Install the benchmark's MCP server package into the same environment: + +```bash +uv pip install --config-settings editable-mode=compat \ + -e ../thinkingbox-data/servers/tb_business_ops_servers_202606 +``` + +Install Typesense into the active ThinkingBox virtual environment: + +```bash +./scripts/install_typesense.sh +typesense-server --version +``` + +Configure an LLM endpoint in `config/config_o4mini.yaml` or another ThinkingBox +configuration file. See the +[ThinkingBox LLM configuration guide](https://github.com/microsoft/thinkingbox/blob/main/docs/llm_endpoint_config.md) +for the supported providers and fields. + +From `thinkingbox/`, start Typesense and the MCP Session Proxy in one terminal: + +```bash +export THINKINGBOX_DATA="../thinkingbox-data" +export TB_MCP_START_SERVERS_FILE="../thinkingbox-data/servers/servers.yaml" +./scripts/background_tasks.sh +``` + +Wait for the script to print `All processes are running`. It uses the +`TYPESENSE_API_KEY=Fake` default expected by the benchmark server +configuration. Keep this terminal running. + +In another terminal, enter `thinkingbox/`, activate the same environment, and +run all 507 tasks: + +```bash +source .venv/bin/activate + +uv run tb infer -c config/config_o4mini.yaml \ + --dataset ../thinkingbox-data/dataset --agent think \ + --test-list ../thinkingbox-data/releases/thinkingbox_bench_v1/testlist_thinkingbox_bench_v1.yaml \ + --repeat 20 --batch-size 20 \ + --output output_thinkingbox_bench_v1.jsonl +``` + +## Analyze the results + +The `tb infer` command writes one JSON object per trial to +`output_thinkingbox_bench_v1.jsonl`. Use `tb agg` to compute the aggregate +metrics: + +```bash +uv run tb agg output_thinkingbox_bench_v1.jsonl +``` + +For a JSONL containing 20 attempts for every task, the aggregate output +includes pass@1, pass@20, and pass^20. pass@20 measures whether at least one of +20 attempts succeeds; pass^20 estimates whether all 20 attempts succeed. These +metrics are omitted if tasks have unequal attempt counts. + +Press Ctrl+C in the background-services terminal to stop Typesense and the MCP +Session Proxy. + +For a reproducible published result, record the exact `thinkingbox` and +`thinkingbox-data` commits, ThinkingBox configuration, model deployment, +inference parameters, user-simulator model, and judge model used for the run. diff --git a/releases/dataset_2603_sandbox_rl_zendesk/testlist_2603_sandbox_rl_zendesk.yaml b/releases/thinkingbox_bench_v1/testlist_thinkingbox_bench_v1.yaml similarity index 100% rename from releases/dataset_2603_sandbox_rl_zendesk/testlist_2603_sandbox_rl_zendesk.yaml rename to releases/thinkingbox_bench_v1/testlist_thinkingbox_bench_v1.yaml