RAA Dataset and Evaluation

English | 简体中文

RAA (Referent Anchoring Accuracy) evaluates whether a full-duplex speech model can anchor its follow-up response to the item the user actually heard after interrupting a list-style answer.

This repository provides:

• 108 synthetic speech evaluation inputs with interruption metadata;
• Baseline inference and P1 played-audio context injection implementations;
• ASR transcription, RAA Judge, and offline data-integrity validation tools;
• Dataset construction scripts and reproduction instructions.

Repository Structure

raa_bench/
  data/                         # 108 RAA samples and their metadata
  scripts/                      # Minimal inference and evaluation pipeline
  .env.example                  # External service configuration template
  requirements.txt

Quick Start

python -m venv .venv
source .venv/bin/activate
pip install -r raa_bench/requirements.txt
cp raa_bench/.env.example raa_bench/.env

First, run the data validation, which requires neither network access nor API keys:

python raa_bench/scripts/validate_open_source_data.py

See raa_bench/README.md for the complete end-to-end workflow and raa_bench/data/README.md for the data format.

Dataset Size

The dataset contains 108 samples, combining 12 selected scenarios, three interruption delays (12, 16, and 20 seconds), and three interruption intents (elaborate, next, and confirm). Each sample includes a 16 kHz mono input.wav file and an interrupt.json metadata file.

External Services

The end-to-end inference, TTS, ASR, and Judge pipeline uses DashScope/Qwen services by default. Users must obtain their own API keys, cover any usage costs, and comply with the applicable service terms. Offline data validation does not call any external service.

License

The code and data are licensed under the Apache License 2.0. Copyright 2026 Taobao.