DataSpace evaluates data agents on analytical questions over self-contained, task-local workspaces. A workspace may combine CSV, JSON, SQLite, Markdown, PDF, and video artifacts. The agent must discover and integrate the relevant evidence, then return the complete result as a verifiable table.
DataSpace is also the official benchmark of the KDD Cup 2026 Data Agent Track.
| 410 tasks | All task questions and workspaces are publicly available |
| 6 modalities | CSV, JSON, SQLite, Markdown, PDF, and video |
| Cross-language workspaces | Chinese and English may occur across both questions and data artifacts |
| Exact, semantic evaluation | Header-invariant column alignment with type-aware value and row comparison |
| 60 public references | Gold results and frozen configs for local end-to-end evaluation |
| 15.0 GB of context | 7,439 files spanning structured data, long documents, and multimedia |
Tasks cover evidence discovery, structured extraction, joins across representations, filtering, aggregation, ranking, temporal reasoning, and document or video understanding.
The complete release is hosted on Hugging Face. Install the Hub client and download the versioned archive:
pip install -U huggingface_hub
hf download HKUSTDial/DataSpace \
release/DataSpace-Benchmark.zip \
release/DataSpace-Benchmark.zip.sha256 \
--repo-type dataset \
--local-dir .Verify the archive before extraction:
cd release
sha256sum --check DataSpace-Benchmark.zip.sha256
unzip DataSpace-Benchmark.zipEvery task exposes one question and one heterogeneous workspace:
DataSpace-Benchmark/
├── input/
│ └── task_N/
│ ├── task.json
│ └── context/
├── output/ # gold.csv for 60 public-reference tasks
│ └── task_N/gold.csv
└── evaluation/
└── configs/task_N.json # frozen configs for the same 60 tasks
An agent should write one prediction per task:
predictions/
└── task_N/
└── prediction.csv
All 410 inputs are public. The release includes reference answers and evaluation configurations for 60 representative tasks; the remaining 350 references are withheld for official full-benchmark evaluation.
The official evaluator uses only the Python standard library:
python3 evaluation/evaluate.py \
--prediction-root /path/to/predictions \
--gold-root /path/to/DataSpace-Benchmark/output \
--config-root /path/to/DataSpace-Benchmark/evaluation/configs \
--output /path/to/evaluation_summary.jsonThe primary metric is Task Accuracy. A task is correct only when the predicted table matches the complete gold result under its frozen configuration. Prediction headers and column order are not scored; values are compared using task-specific types, numeric precision, units, and row-order semantics.
See evaluation/README.md for the complete protocol,
configuration schema, and test suite.
baseline/ contains:
- DataSpace-Agent, a controlled ReAct-style agent with three generic tools;
- a six-backbone comparison under the same agent implementation;
- a unified harness comparison across DataSpace-Agent, Smolagents, Codex, Claude Code, and Grok Build;
- the shared offline Data Workbench Runtime used for fair local execution.
Install the baseline package:
cd baseline
python3 -m venv .venv
.venv/bin/pip install -e .The baseline documentation covers runtime construction, model configuration,
single-task and full-benchmark execution, sandboxing, resume semantics, and
reproducibility checks. See baseline/README.md.
.
├── evaluation/ # official Task Accuracy evaluator and synthetic tests
├── baseline/ # agents, harness adapters, runtime, configs, and launchers
├── LICENSE
└── README.md
The benchmark data, public reference results, and frozen task configurations are distributed separately through Hugging Face.
The code and dataset are released under the MIT License.
@misc{dataspace2027,
title = {DataSpace: Benchmarking Data Agents for Verifiable Analytics
over Heterogeneous Workspaces},
author = {Boyan Li and Zhuowen Liang and Yupeng Xie and Xiaotian Lin and
Tianqi Luo and Xinyu Liu and Yizhang Zhu and Zhangyang Peng and
Yuan Li and Zhengxuan Zhang and Jiayi Zhang and Nan Tang and
Guoliang Li and Yuyu Luo},
year = {2027},
howpublished = {Hugging Face dataset},
url = {https://huggingface.co/datasets/HKUSTDial/DataSpace}
}