From e16e71ec7fe91b767009218169a623eb176d4080 Mon Sep 17 00:00:00 2001
From: Liang-Chun Tsai <ltsai@mirosoft.com>
Date: Mon, 17 Aug 2026 11:02:01 -0700
Subject: [PATCH 01/11] Document ThinkingBox-Bench v1 release

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Copilot-Session: f71fc0b5-25dc-43ae-8eea-04d0e45f8de9
---
 README.md                                     | 43 ++++++++--
 .../dataset_2603_sandbox_rl_zendesk/README.md | 38 +++++----
 releases/thinkingbox_bench_v1/README.md       | 84 +++++++++++++++++++
 3 files changed, 141 insertions(+), 24 deletions(-)
 create mode 100644 releases/thinkingbox_bench_v1/README.md

diff --git a/README.md b/README.md
index 9ae5522..4280d0e 100644
--- a/README.md
+++ b/README.md
@@ -5,6 +5,25 @@ with [ThinkingBox](https://github.com/microsoft/thinkingbox). See the
 framework's [README](https://github.com/microsoft/thinkingbox#readme) for
 installation, configuration, and the `tb` CLI overview.
 
+## ThinkingBox-Bench
+
+[ThinkingBox-Bench](releases/thinkingbox_bench_v1/README.md) is the primary
+evaluation release in this repository. Version 1.0 contains 507 executable
+tool-agent-user tasks across retail and e-commerce, travel and hospitality,
+auto insurance, neobank support, and consulting IT/HR support.
+
+Each task runs in an isolated, stateful tool environment and is evaluated with
+executable checks over the final backend state and side effects. Some tasks
+also check required properties of the final response.
+
+| Version | Tasks | Domains | Release |
+| - | -: | -: | - |
+| ThinkingBox-Bench v1.0 | 507 | 5 | [Documentation and usage](releases/thinkingbox_bench_v1/README.md) |
+
+The rest of this repository also contains individual datasets and development
+fixtures that are not part of ThinkingBox-Bench. The benchmark's canonical task
+set is defined by the test list linked from its release documentation.
+
 ## Contents
 
 - **`dataset/`** — scenarios, agents, and test cases.
@@ -13,7 +32,7 @@ installation, configuration, and the `tb` CLI overview.
   `tb mcp-start`.
 - **`support/`** — large data files used by some tools (embeddings, knowledge
   bases). Set `THINKINGBOX_DATA=<path-to-this-repo>` so tools can locate them.
-- **`releases/`** — per-release dataset snapshots pinned to git tags.
+- **`releases/`** — benchmark releases and versioned dataset snapshots.
 
 ## Layout
 
@@ -106,25 +125,33 @@ If `tb pp` shows a successful conversation or `tb agg` reports passing
 assertions, the framework, server packages, and LLM endpoint are all wired
 up.
 
-## Run larger scenarios
+## Run ThinkingBox-Bench
 
-To run dataset_2602_external_retail and dataset_2603_sandbox_rl_zendesk, 
-you need to also start typesense server. 
+ThinkingBox-Bench requires the business-operations server package and its
+background services. Install the prerequisites described in
+[`tools_with_additional_setup.md`](https://github.com/microsoft/thinkingbox/blob/main/docs/tools_with_additional_setup.md),
+then see the
+[v1.0 release documentation](releases/thinkingbox_bench_v1/README.md) for the
+benchmark composition and evaluation details.
 
 All commands below assume you are in the `thinkingbox/` directory.
 
+In one terminal, start the background services:
+
 ```
-export THINKINGBOX_DATA="../thinkingbox-data/dataset"
+export THINKINGBOX_DATA="../thinkingbox-data"
 export TB_MCP_START_SERVERS_FILE=../thinkingbox-data/servers/servers.yaml
 ./scripts/background_tasks.sh
 ```
 
+In another terminal, run the benchmark:
+
 ```bash
 uv run tb infer -c config/config_o4mini.yaml \
     --dataset ../thinkingbox-data/dataset --agent think \
-    --test-list ../thinkingbox-data/releases/dataset_2602_external_retail/testlist_2602_external_retail_full100.yaml \
-    --repeat 10 --batch-size 40 \
-    --output output_sandbox_external_retail_10reps.jsonl
+    --test-list ../thinkingbox-data/releases/dataset_2603_sandbox_rl_zendesk/testlist_2603_sandbox_rl_zendesk.yaml \
+    --repeat 5 --batch-size 40 \
+    --output output_thinkingbox_bench_v1.jsonl
 ```
 
 ### Re-run assertions on a saved test context
diff --git a/releases/dataset_2603_sandbox_rl_zendesk/README.md b/releases/dataset_2603_sandbox_rl_zendesk/README.md
index 5451061..ed56aab 100644
--- a/releases/dataset_2603_sandbox_rl_zendesk/README.md
+++ b/releases/dataset_2603_sandbox_rl_zendesk/README.md
@@ -1,19 +1,16 @@
 # Sandbox RL Zendesk 26-03
 
-## Repository
-
-Use the following tag of `thinkingbox-data` to use a stable version of this dataset and tools.
-
-|  |  |
-| - | - |
-| Tag | [ds-sandbox-rl-zendesk-2026-03-v1.0](https://github.com/microsoft/thinkingbox-data/releases/tag/ds-sandbox-rl-zendesk-2026-03-v1.0) |
-
+> [!NOTE]
+> This is the source dataset snapshot for
+> [ThinkingBox-Bench v1.0](../thinkingbox_bench_v1/README.md). The snapshot
+> name is retained for compatibility; use the ThinkingBox-Bench release
+> documentation as the canonical benchmark entry point.
 
 ## Overview
 
 **Dataset name**: Sandbox RL Zendesk 26-03
 
-**Test cases**: 516
+**Test cases**: 507
 
 **Agents**: 5
 
@@ -30,7 +27,7 @@ Use the following tag of `thinkingbox-data` to use a stable version of this data
 **a01**
 
 - Name: B2C Ecommerce Electronics Support Agent (External)
-- Num Tests: 100
+- Num Tests: 98
 - Description: Handles consumer inquiries about orders, deliveries, returns, and warranty
 
 **a02r**
@@ -42,7 +39,7 @@ Use the following tag of `thinkingbox-data` to use a stable version of this data
 **a03**
 
 - Name: Car Insurance Policy & Claims Service Agent (External)
-- Num Tests: 101
+- Num Tests: 100
 - Description: Supports policy changes, FNOL intake, claim status inquiries, and documentation
 
 **a04r**
@@ -54,7 +51,7 @@ Use the following tag of `thinkingbox-data` to use a stable version of this data
 **a05**
 
 - Name: Internal Consulting Helpdesk Agent (IT, HR, Expense Support)
-- Num Tests: 107
+- Num Tests: 101
 - Description: Supports consultants with equipment issues, system access, travel/expense inquiries
 
 
@@ -74,11 +71,19 @@ Using this dataset for training or tuning would compromise its value as an indep
 
 Each test case is evaluated by comparing the final database state after the agent's execution against a golden expected state. This is a deterministic, hash-based comparison: the agent either produced exactly the correct set of side effects or it did not.
 
-In addition, 30 out of 516 test cases have simple rubrics evaluated by a LLM judge, in the form of yes/no questions.
+In addition, 30 out of 507 test cases have simple rubrics evaluated by an LLM
+judge in the form of yes/no questions. For these tasks, the canonical test list
+selects rubric-enhanced definitions in place of the corresponding base
+definitions; they are not additional tasks.
 
 Tests pass if the final state is correct and all rubrics evaluate to the expected result. There is no partial credit.
 
-## Evaluation results
+## Historical evaluation results
+
+The results below were produced on the original 516-task snapshot. Nine tasks
+with zero pass rate were subsequently removed, producing the current 507-task
+test list. These results are retained for provenance but must not be interpreted
+as results on ThinkingBox-Bench v1.0.
 
 All results are pass@k over 20 repetitions.
 
@@ -89,7 +94,7 @@ Agent:
 
 User simulator and Judge: GPT-5 Chat
 
-### ds-sandbox-rl-zendesk-2026-03-v1.0
+### Original 516-task snapshot
 
 | Model | Orchestrator | pass@1 | pass@5 |
 |---|---|---|---|
@@ -107,7 +112,8 @@ Results on `t11_24_25_varset_3_rm` refer to a run on the PPE CAPI deployment of
 
 ## Running
 
-Check `thinkingbox/README.md` for installing ThinkingBox
+For the canonical benchmark instructions, see
+[ThinkingBox-Bench v1.0](../thinkingbox_bench_v1/README.md).
 
 ```bash
 # Install the Sandbox RL servers in the ThinkingBox virtual environment
diff --git a/releases/thinkingbox_bench_v1/README.md b/releases/thinkingbox_bench_v1/README.md
new file mode 100644
index 0000000..30d6d19
--- /dev/null
+++ b/releases/thinkingbox_bench_v1/README.md
@@ -0,0 +1,84 @@
+# ThinkingBox-Bench v1.0
+
+ThinkingBox-Bench is an executable benchmark for evaluating whether tool-using
+LLM agents can reliably complete stateful business workflows. Version 1.0
+contains 507 tool-agent-user tasks across five domains.
+
+Each task provides an initial backend state, a user goal and simulated-user
+context, domain tools, policy constraints, and executable checks. An attempt
+passes only when all required checks over the final state, side effects, and
+designated dialogue properties pass.
+
+## Release contents
+
+| Domain | Canonical definitions | Tasks |
+| - | - | -: |
+| Retail and e-commerce | `sandbox_external_retail_group1.py` | 98 |
+| Travel and hospitality | `external_booking_v1_group1.py` (89) and `external_booking_v1_group1_rubrics_yesno.py` (15) | 104 |
+| Auto insurance | `sandbox_auto_insurance_group1.py` | 100 |
+| Neobank support | `sandbox_neobank_support_v1_group1.py` (89) and `sandbox_neobank_support_v1_group1_rubrics_yesno.py` (15) | 104 |
+| Consulting IT/HR support | `sandbox_consulting_group1.py` | 101 |
+| **Total** | | **507** |
+
+The canonical task set is
+[`testlist_2603_sandbox_rl_zendesk.yaml`](../dataset_2603_sandbox_rl_zendesk/testlist_2603_sandbox_rl_zendesk.yaml).
+The test list, rather than every test definition under `dataset/test_case/`,
+defines the release. It selects rubric-enhanced alternatives for 15 travel
+tasks and 15 neobank tasks in place of their corresponding base definitions,
+so these alternatives do not increase the task count.
+
+The source snapshot retains its original
+[`dataset_2603_sandbox_rl_zendesk`](../dataset_2603_sandbox_rl_zendesk/README.md)
+name for compatibility. Other datasets and tests in this repository are not
+part of ThinkingBox-Bench v1.0.
+
+## Intended use
+
+ThinkingBox-Bench v1.0 is intended exclusively for evaluation. Do not use its
+task content, expected outcomes, golden state, or tool trajectories for prompt
+optimization, fine-tuning, reinforcement learning, reward-model training, or
+other model optimization.
+
+## Run the benchmark
+
+The commands below assume `thinkingbox/` and `thinkingbox-data/` are cloned
+side-by-side and are run from the `thinkingbox/` directory.
+
+Install the benchmark's server package in the ThinkingBox environment:
+
+```bash
+uv pip install --config-settings editable-mode=compat \
+    -e ../thinkingbox-data/servers/tb_business_ops_servers_202606
+```
+
+Install the additional service prerequisites described in
+[`tools_with_additional_setup.md`](https://github.com/microsoft/thinkingbox/blob/main/docs/tools_with_additional_setup.md).
+ThinkingBox-Bench requires Typesense.
+
+In one terminal, start the required background services:
+
+```bash
+export THINKINGBOX_DATA="../thinkingbox-data"
+export TB_MCP_START_SERVERS_FILE="../thinkingbox-data/servers/servers.yaml"
+./scripts/background_tasks.sh
+```
+
+In another terminal, run all 507 tasks:
+
+```bash
+uv run tb infer -c config/config_o4mini.yaml \
+    --dataset ../thinkingbox-data/dataset --agent think \
+    --test-list ../thinkingbox-data/releases/dataset_2603_sandbox_rl_zendesk/testlist_2603_sandbox_rl_zendesk.yaml \
+    --repeat 5 --batch-size 40 \
+    --output output_thinkingbox_bench_v1.jsonl
+```
+
+Aggregate pass rates:
+
+```bash
+uv run tb agg output_thinkingbox_bench_v1.jsonl
+```
+
+For a reproducible published result, record the exact `thinkingbox-data` commit
+and the ThinkingBox configuration, model deployment, inference parameters,
+user-simulator model, and judge model used for the run.

From 763f1b0979a7fb6cc5c4d5250a6c926f582b3b44 Mon Sep 17 00:00:00 2001
From: Liang-Chun Tsai <ltsai@mirosoft.com>
Date: Mon, 17 Aug 2026 11:09:04 -0700
Subject: [PATCH 02/11] Consolidate supported benchmark releases

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Copilot-Session: f71fc0b5-25dc-43ae-8eea-04d0e45f8de9
---
 README.md                                     |   5 +-
 releases/README.md                            |  13 ++
 .../dataset_2602_external_retail/README.md    | 171 ------------------
 ...testlist_2602_external_retail_full100.yaml |  98 ----------
 ...testlist_2602_external_retail_quick20.yaml |  19 --
 .../dataset_2603_sandbox_rl_zendesk/README.md | 132 --------------
 .../dataset_2604_airline_tau_bench/README.md  |  12 +-
 releases/thinkingbox_bench_v1/README.md       |  18 +-
 .../testlist_thinkingbox_bench_v1.yaml}       |   0
 9 files changed, 36 insertions(+), 432 deletions(-)
 create mode 100644 releases/README.md
 delete mode 100644 releases/dataset_2602_external_retail/README.md
 delete mode 100644 releases/dataset_2602_external_retail/testlist_2602_external_retail_full100.yaml
 delete mode 100755 releases/dataset_2602_external_retail/testlist_2602_external_retail_quick20.yaml
 delete mode 100644 releases/dataset_2603_sandbox_rl_zendesk/README.md
 rename releases/{dataset_2603_sandbox_rl_zendesk/testlist_2603_sandbox_rl_zendesk.yaml => thinkingbox_bench_v1/testlist_thinkingbox_bench_v1.yaml} (100%)

diff --git a/README.md b/README.md
index 4280d0e..b9c6147 100644
--- a/README.md
+++ b/README.md
@@ -32,7 +32,8 @@ set is defined by the test list linked from its release documentation.
   `tb mcp-start`.
 - **`support/`** — large data files used by some tools (embeddings, knowledge
   bases). Set `THINKINGBOX_DATA=<path-to-this-repo>` so tools can locate them.
-- **`releases/`** — benchmark releases and versioned dataset snapshots.
+- **[`releases/`](releases/README.md)** — supported benchmark releases and
+  their canonical test lists.
 
 ## Layout
 
@@ -149,7 +150,7 @@ In another terminal, run the benchmark:
 ```bash
 uv run tb infer -c config/config_o4mini.yaml \
     --dataset ../thinkingbox-data/dataset --agent think \
-    --test-list ../thinkingbox-data/releases/dataset_2603_sandbox_rl_zendesk/testlist_2603_sandbox_rl_zendesk.yaml \
+    --test-list ../thinkingbox-data/releases/thinkingbox_bench_v1/testlist_thinkingbox_bench_v1.yaml \
     --repeat 5 --batch-size 40 \
     --output output_thinkingbox_bench_v1.jsonl
 ```
diff --git a/releases/README.md b/releases/README.md
new file mode 100644
index 0000000..ab8ff95
--- /dev/null
+++ b/releases/README.md
@@ -0,0 +1,13 @@
+# Releases
+
+This repository publishes two supported evaluation releases:
+
+| Release | Tasks | Description |
+| - | -: | - |
+| [ThinkingBox-Bench v1.0](thinkingbox_bench_v1/README.md) | 507 | Stateful business workflows across five domains |
+| [Airline Tau-Bench 26-04](dataset_2604_airline_tau_bench/README.md) | 21 | Microsoft-adapted Tau-Bench airline evaluation |
+
+Each release directory documents its intended use, canonical test list,
+evaluation method, prerequisites, and execution command. Other files under
+`dataset/` support development or individual scenarios and are not separate
+published releases.
diff --git a/releases/dataset_2602_external_retail/README.md b/releases/dataset_2602_external_retail/README.md
deleted file mode 100644
index a80400d..0000000
--- a/releases/dataset_2602_external_retail/README.md
+++ /dev/null
@@ -1,171 +0,0 @@
-# External Retail 26-02
-
-## Repository
-
-Use the following tag of `thinkingbox-data` to use a stable version of this dataset and tools.
-
-|  |  |
-| - | - |
-| Tag | [ds-external-retail-2026-02-v1.0](https://github.com/microsoft/thinkingbox-data/releases/tag/ds-external-retail-2026-02-v1.0) |
-
-
-## Overview
-
-**Dataset name**: External Retail 26-02
-
-**Test cases**: 100
-
-System: Zendesk
-
-Domain: Retail & E-commerce
-
-Business function: Customer Support & Service
-
-Agent name: B2C Ecommerce Electronics Support Agent (External)
-
-The agent operates as a customer service representative for **TechHome Direct**, a fictitious online electronics and appliances retailer. It handles consumer inquiries about orders, deliveries, returns, warranty claims, installation services, payment disputes, order modifications, and membership management through a Zendesk-based ticketing system.
-
-## Intended use
-
-This dataset is designed **exclusively for evaluation** of LLM agent capabilities. It provides an unbiased benchmark for measuring how well agents handle realistic, multi-step customer service workflows with tool use.
-
-**This dataset MUST NOT be used for:**
-- Prompt tuning or prompt optimization
-- Fine-tuning or training language models
-- Reinforcement learning or reward model training
-- Any form of optimization that uses test case content, expected outcomes, or golden tool interactions as a training signal
-
-Using this dataset for training or tuning would compromise its value as an independent evaluation benchmark. The dataset is intended to represent an unbiased assessment of agent performance; any model optimized against it would produce inflated scores that do not reflect genuine capability improvements.
-
-## What the dataset covers
-
-The 100 test cases simulate realistic B2C customer support interactions across the full lifecycle of an e-commerce customer relationship. Each test case provides a customer query, a simulated user persona with context, a pre-populated database state, and a golden expected outcome (the sequence of tool calls and their parameters that constitute a correct resolution).
-
-### Topic distribution
-
-| Category | Stories | Cases | What it tests |
-|---|---|---|---|
-| **Returns** | ST006, ST008, ST009, ST010 | 40 | Return eligibility by tier/window/product category, fee calculations (restocking, shipping, removal), RMA creation, returns with installation cancellation, ineligible return handling, pre-delivery return attempts |
-| **Defects & Warranty** | ST007, ST012, ST013, ST014 | 14 | Warranty claim filing, routing between return and warranty paths, protection plan vs manufacturer warranty, minor defect partial refunds with score-based compensation |
-| **Installation Services** | ST015, ST016, ST017, ST018 | 10 | Customer-requested rescheduling, weather delay rescheduling with compensation, workmanship warranty issues, installation cancellation with shipping cost charge |
-| **Order Tracking & Delivery** | ST002, ST003 | 7 | Delayed shipment compensation (score-based), delivery exception handling, ticket reuse for follow-ups |
-| **Undelivered Packages** | ST004, ST005 | 9 | Missing package investigation, courtesy replacements (tier/score-based), carrier investigation, returned-to-sender reship with fault determination |
-| **Order Modifications** | ST025, ST026, ST028, ST029 | 11 | Order cancellation (with and without installation), shipping address changes within modification window, promo code after purchase |
-| **Membership Management** | ST030, ST031 | 5 | Plus membership upgrades and cancellations |
-| **Exchanges** | ST011 | 4 | Same-SKU exchanges for defective items, inventory availability checks |
-
-### Complexity dimensions
-
-The test cases vary along several dimensions that affect difficulty:
-
-- **Customer tiers** (Standard, Plus, VIP) determine return windows, fee waivers, shipping speeds, and ticket priority
-- **Customer scores** (Regular, Opportunist, Bonus Hunter) determine compensation amounts and fee waivers, and must never be disclosed to the customer
-- **Policy lookups**: the agent's system prompt references policies but does not contain specific fee amounts, return windows, or warranty periods. The agent must search the knowledge base at runtime to retrieve these values and apply them correctly.
-- **Multi-step workflows**: most cases require multiple tool calls in sequence (e.g., look up customer profile, find or create a Zendesk ticket, retrieve order details, search policies, calculate fees, create an RMA, update ticket status)
-- **Ticket management**: the agent must decide whether to create a new ticket, reopen a recently solved one, or reuse an existing open ticket, and must set the correct status, priority, and type based on the customer's tier and issue
-- **User interaction**: 35 of the 100 test cases do not include all required information in the initial query. The agent must ask the customer for missing details (email, order ID, preferences) before proceeding. A simulated user with the required context is provided for each test case.
-- **Edge cases**: first-time customer courtesy waivers, product category-specific return windows (computing, gaming, and wearables have reduced windows for Standard customers), installed appliance removal fees
-
-### Evaluation method
-
-Each test case is evaluated by comparing the final database state after the agent's execution against a golden expected state. This is a deterministic, hash-based comparison: the agent either produced exactly the correct set of side effects (created the right tickets, RMAs, refunds, etc. with the correct parameters) or it did not. There is no partial credit.
-
-## Scenario
-
-All test cases refer to scenario `dataset/scenario/sandbox_external_retail.yaml`, which provides the agent with:
-- System instructions, defining the agent's role
-- Knowledge base articles: 10 policy documents searchable at runtime
-
-The agent has access to 33 tools spanning several systems.
-
-## Test cases
-
-The dataset is made of 100 test cases, found in `dataset/test_case/sandbox_external_retail/`.
-
-Some interaction with the user is expected in most test cases, to retrieve missing information that is critical for the completion of the task. A prompt for a simulated user, including the required additional context, is provided for each test case.
-
-Two splits are provided as YAML files in this directory:
-
-| Split | File | Cases | Description |
-|---|---|---|---|
-| **full100** | `testlist_2602_external_retail_full100.yaml` | 100 | All test cases |
-| **quick20** | `testlist_2602_external_retail_quick20.yaml` | 20 | A subset of 20 test cases |
-
-The quick20 split is designed to track the full100 performance, and to be used as proxy for faster iteration.
-
-## Evaluation results
-
-All results are pass@k over 20 repetitions.
-
-Agent:
-- Temperature: 1.0
-- No seed
-- Reasoning effort (reasoning models only): medium
-
-User simulator: GPT-5 Chat
-
-### full100
-
-| Model | Orchestrator | pass@1 | pass@5 |
-|---|---|---|---|
-| Opus4.6 | ThinkingBox | 0.73 | 0.86 |
-| BIC RL FT GPT-5 mini Medium Reasoning | ThinkingBox | 0.70 | 0.89 |
-| GPT 5.2 reasoning medium | ThinkingBox | 0.69 | 0.85 |
-| Sonnet4.6 | ThinkingBox | 0.68 | 0.84 |
-| Opus4.7 | ThinkingBox | 0.61 | 0.76 |
-| GPT 5 reasoning medium | ThinkingBox | 0.58 | 0.83 |
-| Sonnet4.5 | ThinkingBox | 0.52 | 0.82 |
-| GPT 5.2 reasoning medium | MCS | 0.47 | 0.73 |
-| GPT-5 mini reasoning medium | ThinkingBox | 0.37 | 0.77 |
-| GPT-4.1 | ThinkingBox | 0.16 | 0.42 |
-| GPT-5-Chat | ThinkingBox | 0.03 | 0.11 |
-| GPT-4.1 | MCS | 0.01 | 0.06 |
-
-### quick20
-
-| Model | Orchestrator | pass@1 | pass@5 |
-|---|---|---|---|
-| GPT 5.2 reasoning medium | ThinkingBox | 0.69 | 0.86 |
-| GPT-4.1 | ThinkingBox | 0.20 | 0.51 |
-| GPT-5 mini reasoning medium | ThinkingBox | 0.39 | 0.82 |
-| t11_24_25_5mini_rm medium (FT) | ThinkingBox | 0.74 | 0.92 |
-| GPT-4.1 | MCS | 0.02 | 0.08 |
-| GPT-5.2 reasoning medium | MCS | 0.38 | 0.72 |
-
-### Notes
-
-Results on `t11_24_25_5mini_rm` refer to a run on an internal deployment of the fine-tuned model. There is often a small difference when re-evaluating on the public deployment after publishing, which we cannot quantify at this time.
-
-MCS tests were run with the following parameters:
-
-- GPT-4.1:
-    - useModelKnowledge: false
-    - model: CurrentModels/GPT41, routed to AOAI GPT4.1, with overrides (temperature=1.0, seed=None)
-- GPT-5.2 reasoning
-    - useModelKnowledge: false
-    - model: ReasoningExperimentalModels/GPT5Reasoning, routed to AOAI GPT-5.2, with overrides (temperature=1.0, reasoning=medium)
-
-## Running
-
-Check `thinkingbox/README.md` for installing ThinkingBox
-
-```bash
-# Install the servers in the ThinkingBox virtual environment
-uv pip install --config-settings editable-mode=compat -e servers/tb_business_ops_servers_202606
-
-# Start typesense (default port, key: Fake)
-mkdir -p /tmp/typesense/data && typesense-server --data-dir="/tmp/typesense/data" --api-key="Fake" --enable-cors
-
-# Start session proxy
-THINKINGBOX_DATA=thinkingbox-data tb mcp-start --servers servers.yaml
-
-# Decode (full100, 20 repetitions)
-tb infer -c config.yaml -d thinkingbox-data/dataset -a think \
-    --inputs thinkingbox-data/dataset/test_case/sandbox_external_retail/ \
-    --repeat 20 --batch-size 40 -o output_zendesk_external_retail_full100.jsonl
-
-# Decode (quick20, 5 repetitions)
-tb infer -c config.yaml -d thinkingbox-data/dataset -a think \
-    --test-list thinkingbox-data/releases/dataset_2602_external_retail/testlist_2602_external_retail_quick20.yaml \
-    --repeat 5 --batch-size 40 -o output_zendesk_external_retail_quick20.jsonl
-```
diff --git a/releases/dataset_2602_external_retail/testlist_2602_external_retail_full100.yaml b/releases/dataset_2602_external_retail/testlist_2602_external_retail_full100.yaml
deleted file mode 100644
index 7a21e1f..0000000
--- a/releases/dataset_2602_external_retail/testlist_2602_external_retail_full100.yaml
+++ /dev/null
@@ -1,98 +0,0 @@
-- sandbox_external_retail_group1.py:test_case_ST002_001
-- sandbox_external_retail_group1.py:test_case_ST002_003
-- sandbox_external_retail_group1.py:test_case_ST002_004
-- sandbox_external_retail_group1.py:test_case_ST003_003
-- sandbox_external_retail_group1.py:test_case_ST003_004
-- sandbox_external_retail_group1.py:test_case_ST003_006
-- sandbox_external_retail_group1.py:test_case_ST004_001
-- sandbox_external_retail_group1.py:test_case_ST004_003
-- sandbox_external_retail_group1.py:test_case_ST004_004
-- sandbox_external_retail_group1.py:test_case_ST004_005
-- sandbox_external_retail_group1.py:test_case_ST004_007
-- sandbox_external_retail_group1.py:test_case_ST004_008
-- sandbox_external_retail_group1.py:test_case_ST004_009
-- sandbox_external_retail_group1.py:test_case_ST005_001
-- sandbox_external_retail_group1.py:test_case_ST005_005
-- sandbox_external_retail_group1.py:test_case_ST006_001
-- sandbox_external_retail_group1.py:test_case_ST006_002
-- sandbox_external_retail_group1.py:test_case_ST006_003
-- sandbox_external_retail_group1.py:test_case_ST006_004
-- sandbox_external_retail_group1.py:test_case_ST006_005
-- sandbox_external_retail_group1.py:test_case_ST006_006
-- sandbox_external_retail_group1.py:test_case_ST006_008
-- sandbox_external_retail_group1.py:test_case_ST006_010
-- sandbox_external_retail_group1.py:test_case_ST006_011
-- sandbox_external_retail_group1.py:test_case_ST006_015
-- sandbox_external_retail_group1.py:test_case_ST006_016
-- sandbox_external_retail_group1.py:test_case_ST006_017
-- sandbox_external_retail_group1.py:test_case_ST006_019
-- sandbox_external_retail_group1.py:test_case_ST006_021
-- sandbox_external_retail_group1.py:test_case_ST006_022
-- sandbox_external_retail_group1.py:test_case_ST006_023
-- sandbox_external_retail_group1.py:test_case_ST006_024
-- sandbox_external_retail_group1.py:test_case_ST006_025
-- sandbox_external_retail_group1.py:test_case_ST006_026
-- sandbox_external_retail_group1.py:test_case_ST006_028
-- sandbox_external_retail_group1.py:test_case_ST006_030
-- sandbox_external_retail_group1.py:test_case_ST006_031
-- sandbox_external_retail_group1.py:test_case_ST006_032
-- sandbox_external_retail_group1.py:test_case_ST006_033
-- sandbox_external_retail_group1.py:test_case_ST006_034
-- sandbox_external_retail_group1.py:test_case_ST006_037
-- sandbox_external_retail_group1.py:test_case_ST006_039
-- sandbox_external_retail_group1.py:test_case_ST006_040
-- sandbox_external_retail_group1.py:test_case_ST006_041
-- sandbox_external_retail_group1.py:test_case_ST006_042
-- sandbox_external_retail_group1.py:test_case_ST006_043
-- sandbox_external_retail_group1.py:test_case_ST006_044
-- sandbox_external_retail_group1.py:test_case_ST006_045
-- sandbox_external_retail_group1.py:test_case_ST006_047
-- sandbox_external_retail_group1.py:test_case_ST006_049
-- sandbox_external_retail_group1.py:test_case_ST007_003
-- sandbox_external_retail_group1.py:test_case_ST007_005
-- sandbox_external_retail_group1.py:test_case_ST007_006
-- sandbox_external_retail_group1.py:test_case_ST007_008
-- sandbox_external_retail_group1.py:test_case_ST008_003
-- sandbox_external_retail_group1.py:test_case_ST009_001
-- sandbox_external_retail_group1.py:test_case_ST009_004
-- sandbox_external_retail_group1.py:test_case_ST009_005
-- sandbox_external_retail_group1.py:test_case_ST010_001
-- sandbox_external_retail_group1.py:test_case_ST011_002
-- sandbox_external_retail_group1.py:test_case_ST011_006
-- sandbox_external_retail_group1.py:test_case_ST011_007
-- sandbox_external_retail_group1.py:test_case_ST011_009
-- sandbox_external_retail_group1.py:test_case_ST012_003
-- sandbox_external_retail_group1.py:test_case_ST013_001
-- sandbox_external_retail_group1.py:test_case_ST013_003
-- sandbox_external_retail_group1.py:test_case_ST013_004
-- sandbox_external_retail_group1.py:test_case_ST013_005
-- sandbox_external_retail_group1.py:test_case_ST013_006
-- sandbox_external_retail_group1.py:test_case_ST013_007
-- sandbox_external_retail_group1.py:test_case_ST013_008
-- sandbox_external_retail_group1.py:test_case_ST014_002
-- sandbox_external_retail_group1.py:test_case_ST014_005
-- sandbox_external_retail_group1.py:test_case_ST015_001
-- sandbox_external_retail_group1.py:test_case_ST016_004
-- sandbox_external_retail_group1.py:test_case_ST016_006
-- sandbox_external_retail_group1.py:test_case_ST017_001
-- sandbox_external_retail_group1.py:test_case_ST017_002
-- sandbox_external_retail_group1.py:test_case_ST017_003
-- sandbox_external_retail_group1.py:test_case_ST017_005
-- sandbox_external_retail_group1.py:test_case_ST018_002
-- sandbox_external_retail_group1.py:test_case_ST018_003
-- sandbox_external_retail_group1.py:test_case_ST018_005
-- sandbox_external_retail_group1.py:test_case_ST025_001
-- sandbox_external_retail_group1.py:test_case_ST025_002
-- sandbox_external_retail_group1.py:test_case_ST025_003
-- sandbox_external_retail_group1.py:test_case_ST025_004
-- sandbox_external_retail_group1.py:test_case_ST025_006
-- sandbox_external_retail_group1.py:test_case_ST026_002
-- sandbox_external_retail_group1.py:test_case_ST026_003
-- sandbox_external_retail_group1.py:test_case_ST028_003
-- sandbox_external_retail_group1.py:test_case_ST029_001
-- sandbox_external_retail_group1.py:test_case_ST029_002
-- sandbox_external_retail_group1.py:test_case_ST030_001
-- sandbox_external_retail_group1.py:test_case_ST030_002
-- sandbox_external_retail_group1.py:test_case_ST030_003
-- sandbox_external_retail_group1.py:test_case_ST031_001
-- sandbox_external_retail_group1.py:test_case_ST031_002
diff --git a/releases/dataset_2602_external_retail/testlist_2602_external_retail_quick20.yaml b/releases/dataset_2602_external_retail/testlist_2602_external_retail_quick20.yaml
deleted file mode 100755
index 38796b3..0000000
--- a/releases/dataset_2602_external_retail/testlist_2602_external_retail_quick20.yaml
+++ /dev/null
@@ -1,19 +0,0 @@
-- sandbox_external_retail_group1.py:test_case_ST002_003
-- sandbox_external_retail_group1.py:test_case_ST025_004
-- sandbox_external_retail_group1.py:test_case_ST003_006
-- sandbox_external_retail_group1.py:test_case_ST026_003
-- sandbox_external_retail_group1.py:test_case_ST006_033
-- sandbox_external_retail_group1.py:test_case_ST009_001
-- sandbox_external_retail_group1.py:test_case_ST025_002
-- sandbox_external_retail_group1.py:test_case_ST013_005
-- sandbox_external_retail_group1.py:test_case_ST008_003
-- sandbox_external_retail_group1.py:test_case_ST006_008
-- sandbox_external_retail_group1.py:test_case_ST006_019
-- sandbox_external_retail_group1.py:test_case_ST029_001
-- sandbox_external_retail_group1.py:test_case_ST006_023
-- sandbox_external_retail_group1.py:test_case_ST005_001
-- sandbox_external_retail_group1.py:test_case_ST006_025
-- sandbox_external_retail_group1.py:test_case_ST013_004
-- sandbox_external_retail_group1.py:test_case_ST029_002
-- sandbox_external_retail_group1.py:test_case_ST030_001
-- sandbox_external_retail_group1.py:test_case_ST005_005
diff --git a/releases/dataset_2603_sandbox_rl_zendesk/README.md b/releases/dataset_2603_sandbox_rl_zendesk/README.md
deleted file mode 100644
index ed56aab..0000000
--- a/releases/dataset_2603_sandbox_rl_zendesk/README.md
+++ /dev/null
@@ -1,132 +0,0 @@
-# Sandbox RL Zendesk 26-03
-
-> [!NOTE]
-> This is the source dataset snapshot for
-> [ThinkingBox-Bench v1.0](../thinkingbox_bench_v1/README.md). The snapshot
-> name is retained for compatibility; use the ThinkingBox-Bench release
-> documentation as the canonical benchmark entry point.
-
-## Overview
-
-**Dataset name**: Sandbox RL Zendesk 26-03
-
-**Test cases**: 507
-
-**Agents**: 5
-
-| Agent | System | Domain | Business function |
-| - | - | - | - |
-| a01 | Zendesk | Retail & E-commerce | Customer Support & Service |
-| a02r | Zendesk | Travel & Hospitality | Customer Support & Service |
-| a03 | Zendesk | Insurance | Customer Support & Service |
-| a04r | Zendesk | Banking & Finance | IT & Technical Support |
-| a05 | Zendesk | Consulting Services | IT & Technical Support |
-
-### Agent descriptions
-
-**a01**
-
-- Name: B2C Ecommerce Electronics Support Agent (External)
-- Num Tests: 98
-- Description: Handles consumer inquiries about orders, deliveries, returns, and warranty
-
-**a02r**
-
-- Name: Hotel Booking Support & Reservation Management Agent (External)
-- Num Tests: 104
-- Description: Assists guests with reservations, modifications, cancellations, and billing
-
-**a03**
-
-- Name: Car Insurance Policy & Claims Service Agent (External)
-- Num Tests: 100
-- Description: Supports policy changes, FNOL intake, claim status inquiries, and documentation
-
-**a04r**
-
-- Name: Internal Neobank IT Support Desk (IT Service Agent)
-- Num Tests: 104
-- Description: Handles internal employee how-to requests, access and end-user hardware providing, and system issues
-
-**a05**
-
-- Name: Internal Consulting Helpdesk Agent (IT, HR, Expense Support)
-- Num Tests: 101
-- Description: Supports consultants with equipment issues, system access, travel/expense inquiries
-
-
-## Intended use
-
-This dataset is designed **exclusively for evaluation** of LLM agent capabilities. It provides an unbiased benchmark for measuring how well agents handle realistic, multi-step customer service workflows with tool use.
-
-**This dataset MUST NOT be used for:**
-- Prompt tuning or prompt optimization
-- Fine-tuning or training language models
-- Reinforcement learning or reward model training
-- Any form of optimization that uses test case content, expected outcomes, or golden tool interactions as a training signal
-
-Using this dataset for training or tuning would compromise its value as an independent evaluation benchmark. The dataset is intended to represent an unbiased assessment of agent performance; any model optimized against it would produce inflated scores that do not reflect genuine capability improvements.
-
-### Evaluation method
-
-Each test case is evaluated by comparing the final database state after the agent's execution against a golden expected state. This is a deterministic, hash-based comparison: the agent either produced exactly the correct set of side effects or it did not.
-
-In addition, 30 out of 507 test cases have simple rubrics evaluated by an LLM
-judge in the form of yes/no questions. For these tasks, the canonical test list
-selects rubric-enhanced definitions in place of the corresponding base
-definitions; they are not additional tasks.
-
-Tests pass if the final state is correct and all rubrics evaluate to the expected result. There is no partial credit.
-
-## Historical evaluation results
-
-The results below were produced on the original 516-task snapshot. Nine tasks
-with zero pass rate were subsequently removed, producing the current 507-task
-test list. These results are retained for provenance but must not be interpreted
-as results on ThinkingBox-Bench v1.0.
-
-All results are pass@k over 20 repetitions.
-
-Agent:
-- Temperature: 1.0
-- No seed
-- Reasoning effort (reasoning models only): medium
-
-User simulator and Judge: GPT-5 Chat
-
-### Original 516-task snapshot
-
-| Model | Orchestrator | pass@1 | pass@5 |
-|---|---|---|---|
-| GPT o4-mini reasoning medium | ThinkingBox | 0.09 | 0.28 |
-| t11_24_25_varset_3_rm (o4-mini FT) reasoning medium | ThinkingBox | 0.37 | 0.71 |
-| GPT 5-mini reasoning medium | ThinkingBox | 0.19 | 0.41 |
-| GPT 5.2 reasoning medium | ThinkingBox | 0.46 | 0.71 |
-| GPT 5.4 reasoning medium | ThinkingBox | 0.64 | 0.83 |
-| Claude Sonnet 4.6 reasoning medium | ThinkingBox | 0.57 | 0.78 |
-| Claude Opus 4.6 reasoning medium | ThinkingBox | 0.38 | 0.57 |
-
-### Notes
-
-Results on `t11_24_25_varset_3_rm` refer to a run on the PPE CAPI deployment of the fine-tuned model.
-
-## Running
-
-For the canonical benchmark instructions, see
-[ThinkingBox-Bench v1.0](../thinkingbox_bench_v1/README.md).
-
-```bash
-# Install the Sandbox RL servers in the ThinkingBox virtual environment
-uv pip install --config-settings editable-mode=compat -e servers/tb_business_ops_servers_202606
-
-# Start typesense (default port, key: Fake)
-mkdir -p /tmp/typesense/data && typesense-server --data-dir="/tmp/typesense/data" --api-key="Fake" --enable-cors
-
-# Start session proxy
-THINKINGBOX_DATA=thinkingbox-data tb mcp-start --servers thinkingbox-data/servers/servers.yaml
-
-# Decode (full dataset, 5 repetitions)
-tb infer -c config.yaml -d thinkingbox-data/dataset -a think \
-    --test-list thinkingbox-data/releases/dataset_2603_sandbox_rl_zendesk/testlist_2603_sandbox_rl_zendesk.yaml \
-    --repeat 5 --batch-size 40 -o output_2603_sandbox_rl_zendesk.jsonl
-```
diff --git a/releases/dataset_2604_airline_tau_bench/README.md b/releases/dataset_2604_airline_tau_bench/README.md
index e6d66f6..f0a1940 100644
--- a/releases/dataset_2604_airline_tau_bench/README.md
+++ b/releases/dataset_2604_airline_tau_bench/README.md
@@ -78,16 +78,20 @@ Some interaction with the user is expected in most test cases, to retrieve missi
 ## Running
 
 Check `thinkingbox/README.md` for installing ThinkingBox.
+The commands below assume `thinkingbox/` and `thinkingbox-data/` are cloned
+side-by-side and are run from the `thinkingbox/` directory.
 
 ```bash
 # Install the airline tau-bench server in the ThinkingBox virtual environment
-uv pip install --config-settings editable-mode=compat -e servers/thinkingbox_tools
+uv pip install --config-settings editable-mode=compat \
+    -e ../thinkingbox-data/servers/thinkingbox_tools
 
 # Start session proxy
-THINKINGBOX_DATA=thinkingbox-data tb mcp-start --servers thinkingbox-data/servers/servers.yaml
+THINKINGBOX_DATA=../thinkingbox-data \
+    tb mcp-start --servers ../thinkingbox-data/servers/servers.yaml
 
 # Decode (full dataset, 5 repetitions)
-tb infer -c config.yaml -d thinkingbox-data/dataset -a think \
-    --test-list thinkingbox-data/releases/dataset_2604_airline_tau_bench/testlist_2604_airline_tau_bench.yaml \
+tb infer -c config.yaml -d ../thinkingbox-data/dataset -a think \
+    --test-list ../thinkingbox-data/releases/dataset_2604_airline_tau_bench/testlist_2604_airline_tau_bench.yaml \
     --repeat 5 --batch-size 40 -o output_2604_airline_tau_bench.jsonl
 ```
diff --git a/releases/thinkingbox_bench_v1/README.md b/releases/thinkingbox_bench_v1/README.md
index 30d6d19..c5d4bf8 100644
--- a/releases/thinkingbox_bench_v1/README.md
+++ b/releases/thinkingbox_bench_v1/README.md
@@ -21,16 +21,22 @@ designated dialogue properties pass.
 | **Total** | | **507** |
 
 The canonical task set is
-[`testlist_2603_sandbox_rl_zendesk.yaml`](../dataset_2603_sandbox_rl_zendesk/testlist_2603_sandbox_rl_zendesk.yaml).
+[`testlist_thinkingbox_bench_v1.yaml`](testlist_thinkingbox_bench_v1.yaml).
 The test list, rather than every test definition under `dataset/test_case/`,
 defines the release. It selects rubric-enhanced alternatives for 15 travel
 tasks and 15 neobank tasks in place of their corresponding base definitions,
 so these alternatives do not increase the task count.
 
-The source snapshot retains its original
-[`dataset_2603_sandbox_rl_zendesk`](../dataset_2603_sandbox_rl_zendesk/README.md)
-name for compatibility. Other datasets and tests in this repository are not
-part of ThinkingBox-Bench v1.0.
+Other datasets and tests in this repository are not part of ThinkingBox-Bench
+v1.0.
+
+## Evaluation method
+
+Each task compares the final backend state with its golden expected state using
+deterministic, hash-based checks. Thirty tasks additionally use simple yes/no
+rubrics evaluated by an LLM judge. A task passes only when its final state is
+correct and every applicable rubric returns the expected result; there is no
+partial credit.
 
 ## Intended use
 
@@ -68,7 +74,7 @@ In another terminal, run all 507 tasks:
 ```bash
 uv run tb infer -c config/config_o4mini.yaml \
     --dataset ../thinkingbox-data/dataset --agent think \
-    --test-list ../thinkingbox-data/releases/dataset_2603_sandbox_rl_zendesk/testlist_2603_sandbox_rl_zendesk.yaml \
+    --test-list ../thinkingbox-data/releases/thinkingbox_bench_v1/testlist_thinkingbox_bench_v1.yaml \
     --repeat 5 --batch-size 40 \
     --output output_thinkingbox_bench_v1.jsonl
 ```
diff --git a/releases/dataset_2603_sandbox_rl_zendesk/testlist_2603_sandbox_rl_zendesk.yaml b/releases/thinkingbox_bench_v1/testlist_thinkingbox_bench_v1.yaml
similarity index 100%
rename from releases/dataset_2603_sandbox_rl_zendesk/testlist_2603_sandbox_rl_zendesk.yaml
rename to releases/thinkingbox_bench_v1/testlist_thinkingbox_bench_v1.yaml

From e8078008e8e3a332c9c1840c24b1be213406299c Mon Sep 17 00:00:00 2001
From: Liang-Chun Tsai <ltsai@mirosoft.com>
Date: Mon, 17 Aug 2026 11:11:30 -0700
Subject: [PATCH 03/11] Link supported release tags

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Copilot-Session: f71fc0b5-25dc-43ae-8eea-04d0e45f8de9
---
 releases/dataset_2604_airline_tau_bench/README.md | 2 +-
 releases/thinkingbox_bench_v1/README.md           | 9 +++++++++
 2 files changed, 10 insertions(+), 1 deletion(-)

diff --git a/releases/dataset_2604_airline_tau_bench/README.md b/releases/dataset_2604_airline_tau_bench/README.md
index f0a1940..ca0429f 100644
--- a/releases/dataset_2604_airline_tau_bench/README.md
+++ b/releases/dataset_2604_airline_tau_bench/README.md
@@ -6,7 +6,7 @@ Use the following tag of `thinkingbox-data` to use a stable version of this data
 
 |  |  |
 | - | - |
-| Tag | [ds-airline-tau-bench-2026-04-v1.0](https://github.com/microsoft/thinkingbox-data/releases/tag/ds-airline-tau-bench-2026-04-v1.0) |
+| Tag | [`ds-airline-tau-bench-2026-04-v1.0`](https://github.com/microsoft/thinkingbox-data/tree/ds-airline-tau-bench-2026-04-v1.0) |
 
 
 ## Overview
diff --git a/releases/thinkingbox_bench_v1/README.md b/releases/thinkingbox_bench_v1/README.md
index c5d4bf8..a419012 100644
--- a/releases/thinkingbox_bench_v1/README.md
+++ b/releases/thinkingbox_bench_v1/README.md
@@ -1,5 +1,14 @@
 # ThinkingBox-Bench v1.0
 
+## Repository
+
+Use the following `thinkingbox-data` tag for a stable version of the benchmark,
+tool servers, and supporting data:
+
+| | |
+| - | - |
+| Tag | [`thinkingbox-bench-v1.0`](https://github.com/microsoft/thinkingbox-data/tree/thinkingbox-bench-v1.0) |
+
 ThinkingBox-Bench is an executable benchmark for evaluating whether tool-using
 LLM agents can reliably complete stateful business workflows. Version 1.0
 contains 507 tool-agent-user tasks across five domains.

From 6ccb2949a1ac5e0579d34691e870c43d6faf3c83 Mon Sep 17 00:00:00 2001
From: Liang-Chun Tsai <ltsai@mirosoft.com>
Date: Mon, 17 Aug 2026 13:04:22 -0700
Subject: [PATCH 04/11] Link GitHub release pages

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Copilot-Session: f71fc0b5-25dc-43ae-8eea-04d0e45f8de9
---
 releases/dataset_2604_airline_tau_bench/README.md | 2 +-
 releases/thinkingbox_bench_v1/README.md           | 2 +-
 2 files changed, 2 insertions(+), 2 deletions(-)

diff --git a/releases/dataset_2604_airline_tau_bench/README.md b/releases/dataset_2604_airline_tau_bench/README.md
index ca0429f..d439836 100644
--- a/releases/dataset_2604_airline_tau_bench/README.md
+++ b/releases/dataset_2604_airline_tau_bench/README.md
@@ -6,7 +6,7 @@ Use the following tag of `thinkingbox-data` to use a stable version of this data
 
 |  |  |
 | - | - |
-| Tag | [`ds-airline-tau-bench-2026-04-v1.0`](https://github.com/microsoft/thinkingbox-data/tree/ds-airline-tau-bench-2026-04-v1.0) |
+| Tag | [`ds-airline-tau-bench-2026-04-v1.0`](https://github.com/microsoft/thinkingbox-data/releases/tag/ds-airline-tau-bench-2026-04-v1.0) |
 
 
 ## Overview
diff --git a/releases/thinkingbox_bench_v1/README.md b/releases/thinkingbox_bench_v1/README.md
index a419012..9e62375 100644
--- a/releases/thinkingbox_bench_v1/README.md
+++ b/releases/thinkingbox_bench_v1/README.md
@@ -7,7 +7,7 @@ tool servers, and supporting data:
 
 | | |
 | - | - |
-| Tag | [`thinkingbox-bench-v1.0`](https://github.com/microsoft/thinkingbox-data/tree/thinkingbox-bench-v1.0) |
+| Tag | [`thinkingbox-bench-v1.0`](https://github.com/microsoft/thinkingbox-data/releases/tag/thinkingbox-bench-v1.0) |
 
 ThinkingBox-Bench is an executable benchmark for evaluating whether tool-using
 LLM agents can reliably complete stateful business workflows. Version 1.0

From c0d9ec9635220d03165f517d6ef2ed3663f88b80 Mon Sep 17 00:00:00 2001
From: Liang-Chun Tsai <ltsai@mirosoft.com>
Date: Mon, 17 Aug 2026 13:41:35 -0700
Subject: [PATCH 05/11] Consolidate ThinkingBox-Bench setup instructions

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Copilot-Session: f71fc0b5-25dc-43ae-8eea-04d0e45f8de9
---
 README.md                               | 45 ++++-----------
 releases/thinkingbox_bench_v1/README.md | 77 +++++++++++++++++++++----
 2 files changed, 76 insertions(+), 46 deletions(-)

diff --git a/README.md b/README.md
index b9c6147..13a7b96 100644
--- a/README.md
+++ b/README.md
@@ -49,19 +49,18 @@ parent/
 
 ## Setup
 
+For ThinkingBox-Bench, use the complete
+[v1.0 installation and run instructions](releases/thinkingbox_bench_v1/README.md#run-the-benchmark).
+The commands below are only for general repository development and smoke tests.
+
 Install the framework first (see the [thinkingbox
-README](https://github.com/microsoft/thinkingbox#readme)). Then, still from
-`thinkingbox/`, install the server packages from this repo into the same env:
+README](https://github.com/microsoft/thinkingbox#readme)). For the smoke tests
+below, install the `thinkingbox_tools` package into the same environment:
 
 ```bash
 uv pip install --config-settings editable-mode=compat -e ../thinkingbox-data/servers/thinkingbox_tools
-uv pip install --config-settings editable-mode=compat -e ../thinkingbox-data/servers/tb_business_ops_servers_202606
 ```
 
-Some tools also need extra services (e.g. Typesense, embeddings server) — see
-[`tools_with_additional_setup.md`](https://github.com/microsoft/thinkingbox/blob/main/docs/tools_with_additional_setup.md)
-in the framework repo.
-
 ## Verify your setup
 
 Before running larger scenarios, sanity-check that the framework, the
@@ -128,34 +127,10 @@ up.
 
 ## Run ThinkingBox-Bench
 
-ThinkingBox-Bench requires the business-operations server package and its
-background services. Install the prerequisites described in
-[`tools_with_additional_setup.md`](https://github.com/microsoft/thinkingbox/blob/main/docs/tools_with_additional_setup.md),
-then see the
-[v1.0 release documentation](releases/thinkingbox_bench_v1/README.md) for the
-benchmark composition and evaluation details.
-
-All commands below assume you are in the `thinkingbox/` directory.
-
-In one terminal, start the background services:
-
-```
-export THINKINGBOX_DATA="../thinkingbox-data"
-export TB_MCP_START_SERVERS_FILE=../thinkingbox-data/servers/servers.yaml
-./scripts/background_tasks.sh
-```
-
-In another terminal, run the benchmark:
-
-```bash
-uv run tb infer -c config/config_o4mini.yaml \
-    --dataset ../thinkingbox-data/dataset --agent think \
-    --test-list ../thinkingbox-data/releases/thinkingbox_bench_v1/testlist_thinkingbox_bench_v1.yaml \
-    --repeat 5 --batch-size 40 \
-    --output output_thinkingbox_bench_v1.jsonl
-```
+Use the canonical
+[ThinkingBox-Bench v1.0 installation and run instructions](releases/thinkingbox_bench_v1/README.md#run-the-benchmark).
 
-### Re-run assertions on a saved test context
+## Re-run assertions on a saved test context
 
 After decoding once, re-run just the test assertions (no LLM calls):
 
@@ -173,7 +148,7 @@ uv run tb run-test -c config/config_o4mini.yaml \
     --resultfile output.yaml --update
 ```
 
-### Interactive TUI
+## Interactive TUI
 
 Chat with a scenario:
 
diff --git a/releases/thinkingbox_bench_v1/README.md b/releases/thinkingbox_bench_v1/README.md
index 9e62375..aba0b31 100644
--- a/releases/thinkingbox_bench_v1/README.md
+++ b/releases/thinkingbox_bench_v1/README.md
@@ -56,21 +56,66 @@ other model optimization.
 
 ## Run the benchmark
 
-The commands below assume `thinkingbox/` and `thinkingbox-data/` are cloned
-side-by-side and are run from the `thinkingbox/` directory.
+ThinkingBox-Bench requires:
 
-Install the benchmark's server package in the ThinkingBox environment:
+- Python 3.12 and [`uv`](https://docs.astral.sh/uv/)
+- a Linux or WSL environment
+- ThinkingBox and `thinkingbox-data` cloned side-by-side
+- Typesense 30.1, installed by the ThinkingBox installation script
+
+On Ubuntu or WSL Ubuntu, install the required system tools:
+
+```bash
+sudo apt-get update
+sudo apt-get install -y git curl tar coreutils procps
+```
+
+Install `uv` if it is not already available:
+
+```bash
+curl -LsSf https://astral.sh/uv/install.sh | sh
+source "$HOME/.local/bin/env"
+```
+
+It does **not** require the embeddings server, downloaded Hugging Face models,
+or pre-indexed Typesense snapshots. Each benchmark scenario initializes its
+own Typesense collections from the sources in its scenario definition.
+
+Clone and install ThinkingBox:
+
+```bash
+git clone https://github.com/microsoft/thinkingbox.git
+git clone https://github.com/microsoft/thinkingbox-data.git
+
+cd thinkingbox-data
+git checkout thinkingbox-bench-v1.0
+cd ../thinkingbox
+
+uv venv --python 3.12
+uv sync --group dev
+source .venv/bin/activate
+```
+
+Install the benchmark's MCP server package into the same environment:
 
 ```bash
 uv pip install --config-settings editable-mode=compat \
     -e ../thinkingbox-data/servers/tb_business_ops_servers_202606
 ```
 
-Install the additional service prerequisites described in
-[`tools_with_additional_setup.md`](https://github.com/microsoft/thinkingbox/blob/main/docs/tools_with_additional_setup.md).
-ThinkingBox-Bench requires Typesense.
+Install Typesense into the active ThinkingBox virtual environment:
 
-In one terminal, start the required background services:
+```bash
+./scripts/install_typesense.sh
+typesense-server --version
+```
+
+Configure an LLM endpoint in `config/config_o4mini.yaml` or another ThinkingBox
+configuration file. See the
+[ThinkingBox LLM configuration guide](https://github.com/microsoft/thinkingbox/blob/main/docs/llm_endpoint_config.md)
+for the supported providers and fields.
+
+From `thinkingbox/`, start Typesense and the MCP Session Proxy in one terminal:
 
 ```bash
 export THINKINGBOX_DATA="../thinkingbox-data"
@@ -78,9 +123,16 @@ export TB_MCP_START_SERVERS_FILE="../thinkingbox-data/servers/servers.yaml"
 ./scripts/background_tasks.sh
 ```
 
-In another terminal, run all 507 tasks:
+Wait for the script to print `All processes are running`. It uses the
+`TYPESENSE_API_KEY=Fake` default expected by the benchmark server
+configuration. Keep this terminal running.
+
+In another terminal, enter `thinkingbox/`, activate the same environment, and
+run all 507 tasks:
 
 ```bash
+source .venv/bin/activate
+
 uv run tb infer -c config/config_o4mini.yaml \
     --dataset ../thinkingbox-data/dataset --agent think \
     --test-list ../thinkingbox-data/releases/thinkingbox_bench_v1/testlist_thinkingbox_bench_v1.yaml \
@@ -94,6 +146,9 @@ Aggregate pass rates:
 uv run tb agg output_thinkingbox_bench_v1.jsonl
 ```
 
-For a reproducible published result, record the exact `thinkingbox-data` commit
-and the ThinkingBox configuration, model deployment, inference parameters,
-user-simulator model, and judge model used for the run.
+Press Ctrl+C in the background-services terminal to stop Typesense and the MCP
+Session Proxy.
+
+For a reproducible published result, record the exact `thinkingbox` and
+`thinkingbox-data` commits, ThinkingBox configuration, model deployment,
+inference parameters, user-simulator model, and judge model used for the run.

From 68a2f7845b3513e1f07452d0d98760080f810a6d Mon Sep 17 00:00:00 2001
From: Liang-Chun Tsai <ltsai@mirosoft.com>
Date: Mon, 17 Aug 2026 14:42:37 -0700
Subject: [PATCH 06/11] Use published benchmark run parameters

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Copilot-Session: f71fc0b5-25dc-43ae-8eea-04d0e45f8de9
---
 releases/thinkingbox_bench_v1/README.md | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/releases/thinkingbox_bench_v1/README.md b/releases/thinkingbox_bench_v1/README.md
index aba0b31..b1f5aad 100644
--- a/releases/thinkingbox_bench_v1/README.md
+++ b/releases/thinkingbox_bench_v1/README.md
@@ -136,7 +136,7 @@ source .venv/bin/activate
 uv run tb infer -c config/config_o4mini.yaml \
     --dataset ../thinkingbox-data/dataset --agent think \
     --test-list ../thinkingbox-data/releases/thinkingbox_bench_v1/testlist_thinkingbox_bench_v1.yaml \
-    --repeat 5 --batch-size 40 \
+    --repeat 20 --batch-size 20 \
     --output output_thinkingbox_bench_v1.jsonl
 ```
 

From 9c663b98e1a3a769d8200ce3d6dfc4fe585a1266 Mon Sep 17 00:00:00 2001
From: Liang-Chun Tsai <ltsai@mirosoft.com>
Date: Mon, 17 Aug 2026 14:48:33 -0700
Subject: [PATCH 07/11] Lead with benchmark run instructions

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Copilot-Session: f71fc0b5-25dc-43ae-8eea-04d0e45f8de9
---
 README.md | 9 +++++----
 1 file changed, 5 insertions(+), 4 deletions(-)

diff --git a/README.md b/README.md
index 13a7b96..385b873 100644
--- a/README.md
+++ b/README.md
@@ -16,6 +16,11 @@ Each task runs in an isolated, stateful tool environment and is evaluated with
 executable checks over the final backend state and side effects. Some tasks
 also check required properties of the final response.
 
+If your goal is to run the published benchmark, follow the complete
+[ThinkingBox-Bench v1.0 installation and run instructions](releases/thinkingbox_bench_v1/README.md#run-the-benchmark)
+directly. The remaining setup and examples in this README are intended for
+customized ThinkingBox development, individual scenarios, and smoke testing.
+
 | Version | Tasks | Domains | Release |
 | - | -: | -: | - |
 | ThinkingBox-Bench v1.0 | 507 | 5 | [Documentation and usage](releases/thinkingbox_bench_v1/README.md) |
@@ -49,10 +54,6 @@ parent/
 
 ## Setup
 
-For ThinkingBox-Bench, use the complete
-[v1.0 installation and run instructions](releases/thinkingbox_bench_v1/README.md#run-the-benchmark).
-The commands below are only for general repository development and smoke tests.
-
 Install the framework first (see the [thinkingbox
 README](https://github.com/microsoft/thinkingbox#readme)). For the smoke tests
 below, install the `thinkingbox_tools` package into the same environment:

From feaf1c300edb4439e9332623240c70d6921d44a0 Mon Sep 17 00:00:00 2001
From: Liang-Chun Tsai <ltsai@mirosoft.com>
Date: Mon, 17 Aug 2026 14:50:21 -0700
Subject: [PATCH 08/11] Remove redundant benchmark release table

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Copilot-Session: f71fc0b5-25dc-43ae-8eea-04d0e45f8de9
---
 README.md | 4 ----
 1 file changed, 4 deletions(-)

diff --git a/README.md b/README.md
index 385b873..119406a 100644
--- a/README.md
+++ b/README.md
@@ -21,10 +21,6 @@ If your goal is to run the published benchmark, follow the complete
 directly. The remaining setup and examples in this README are intended for
 customized ThinkingBox development, individual scenarios, and smoke testing.
 
-| Version | Tasks | Domains | Release |
-| - | -: | -: | - |
-| ThinkingBox-Bench v1.0 | 507 | 5 | [Documentation and usage](releases/thinkingbox_bench_v1/README.md) |
-
 The rest of this repository also contains individual datasets and development
 fixtures that are not part of ThinkingBox-Bench. The benchmark's canonical task
 set is defined by the test list linked from its release documentation.

From 1d3cd21deeb6e7aea10c25aa365341a4bb156dd2 Mon Sep 17 00:00:00 2001
From: Liang-Chun Tsai <ltsai@mirosoft.com>
Date: Mon, 17 Aug 2026 15:55:08 -0700
Subject: [PATCH 09/11] Link benchmark result analysis guide

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Copilot-Session: f71fc0b5-25dc-43ae-8eea-04d0e45f8de9
---
 releases/thinkingbox_bench_v1/README.md | 13 ++++++++++++-
 1 file changed, 12 insertions(+), 1 deletion(-)

diff --git a/releases/thinkingbox_bench_v1/README.md b/releases/thinkingbox_bench_v1/README.md
index b1f5aad..114e806 100644
--- a/releases/thinkingbox_bench_v1/README.md
+++ b/releases/thinkingbox_bench_v1/README.md
@@ -140,12 +140,23 @@ uv run tb infer -c config/config_o4mini.yaml \
     --output output_thinkingbox_bench_v1.jsonl
 ```
 
-Aggregate pass rates:
+## Analyze the results
+
+The `tb infer` command writes one JSON object per trial to
+`output_thinkingbox_bench_v1.jsonl`. Use `tb agg` to compute the aggregate
+metrics:
 
 ```bash
 uv run tb agg output_thinkingbox_bench_v1.jsonl
 ```
 
+See [Analyzing inference results](https://github.com/microsoft/thinkingbox/blob/main/docs/analyzing_results.md)
+for the JSONL artifact schema, table and JSON output modes, pass@1, pass@20,
+and pass^20 definitions, per-domain filtering, inspecting individual
+trajectories with `tb pp`, rerunning assertions, and resuming interrupted runs.
+The guide also explains that the paper's trajectory failure categories require
+manual or external classification; `tb agg` does not assign those categories.
+
 Press Ctrl+C in the background-services terminal to stop Typesense and the MCP
 Session Proxy.
 

From 9172be44275661e56214b8ca69205693b4552563 Mon Sep 17 00:00:00 2001
From: Liang-Chun Tsai <ltsai@mirosoft.com>
Date: Mon, 17 Aug 2026 16:12:47 -0700
Subject: [PATCH 10/11] Simplify benchmark analysis reference

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Copilot-Session: f71fc0b5-25dc-43ae-8eea-04d0e45f8de9
---
 releases/thinkingbox_bench_v1/README.md | 7 ++-----
 1 file changed, 2 insertions(+), 5 deletions(-)

diff --git a/releases/thinkingbox_bench_v1/README.md b/releases/thinkingbox_bench_v1/README.md
index 114e806..7c31b5c 100644
--- a/releases/thinkingbox_bench_v1/README.md
+++ b/releases/thinkingbox_bench_v1/README.md
@@ -151,11 +151,8 @@ uv run tb agg output_thinkingbox_bench_v1.jsonl
 ```
 
 See [Analyzing inference results](https://github.com/microsoft/thinkingbox/blob/main/docs/analyzing_results.md)
-for the JSONL artifact schema, table and JSON output modes, pass@1, pass@20,
-and pass^20 definitions, per-domain filtering, inspecting individual
-trajectories with `tb pp`, rerunning assertions, and resuming interrupted runs.
-The guide also explains that the paper's trajectory failure categories require
-manual or external classification; `tb agg` does not assign those categories.
+for the command used to obtain pass@1, pass@20, and pass^20 from the result
+JSONL.
 
 Press Ctrl+C in the background-services terminal to stop Typesense and the MCP
 Session Proxy.

From fcaba4c1a9debec42fda7f15bf29fe6d6b46c431 Mon Sep 17 00:00:00 2001
From: Liang-Chun Tsai <ltsai@mirosoft.com>
Date: Mon, 17 Aug 2026 16:15:13 -0700
Subject: [PATCH 11/11] Document benchmark pass-rate aggregation

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Copilot-Session: f71fc0b5-25dc-43ae-8eea-04d0e45f8de9
---
 releases/thinkingbox_bench_v1/README.md | 7 ++++---
 1 file changed, 4 insertions(+), 3 deletions(-)

diff --git a/releases/thinkingbox_bench_v1/README.md b/releases/thinkingbox_bench_v1/README.md
index 7c31b5c..d75a864 100644
--- a/releases/thinkingbox_bench_v1/README.md
+++ b/releases/thinkingbox_bench_v1/README.md
@@ -150,9 +150,10 @@ metrics:
 uv run tb agg output_thinkingbox_bench_v1.jsonl
 ```
 
-See [Analyzing inference results](https://github.com/microsoft/thinkingbox/blob/main/docs/analyzing_results.md)
-for the command used to obtain pass@1, pass@20, and pass^20 from the result
-JSONL.
+For a JSONL containing 20 attempts for every task, the aggregate output
+includes pass@1, pass@20, and pass^20. pass@20 measures whether at least one of
+20 attempts succeeds; pass^20 estimates whether all 20 attempts succeed. These
+metrics are omitted if tasks have unequal attempt counts.
 
 Press Ctrl+C in the background-services terminal to stop Typesense and the MCP
 Session Proxy.