The DREAM dataset originates from the DREAM program - Development of Robot-Enhanced therapy for children with Autism spectrum disorders. This large-scale European research program aims to advance research into Autism Spectrum Disorder (ASD), one of the most common childhood developmental disorders, and to develop innovative methods and tools to support the diagnosis and therapy of children with ASD.
The dataset contains behavioral data from over 3,000 therapy sessions involving 61 children diagnosed with ASD, aged between 3 and 6 years. These sessions include an initial diagnosis, six therapeutic interventions, and a final diagnosis. Among the 61 children, 41 completed the entire therapy process, while the remaining participants completed an average of 4.5 interventions. Each session was recorded using three RGB cameras and two RGBD (Kinect) cameras, providing detailed information on the children’s behavior during therapy.
This dataset is particularly valuable for studying child-robot interactions, focusing on how children with ASD respond differently to robots compared to human partners. By analyzing these behaviors, researchers can fine-tune robotic interventions for enhanced therapeutic outcomes.
Skeleton visualization videos were generated for each diagnosis session, with body-upper arm and upper arm-lower arm angles included in the video.
User.10_3_diagnosis.abilities_20170508_112106.830000_vis.mp4
A CSV file was created as a foundation for symmetry analysis. This file contains metadata for all diagnosis sessions and the calculated symmetry metrics.
Additionally, distribution plots and linear regression plots were generated from the CSV data to examine the potential correlation between ADOS scores and symmetry metrics. We plan to explore this relationship further in future analysis.
- Set up the environment.
conda env create -f environment.yml
conda activate dream-analysis-env- Download the raw dataset using
Makecommand.
make download- Execute skeleton visualization and output mp4 format videos in
results/visualizationfolder.
# Visualize all sessions
make visualization
# Visualize specific session
make visualization <session_name.json>- Execute symmetry analysis and output a csv format result in
results/analysisfolder.
make analysisAll data in the dataset is stored in JSON format, with a detailed structure presented below.
{
"$id": "User 37_18_Intervention 2_20171102_123242.369000.json",
"$schema": "https://raw.githubusercontent.com/dream2020/data/master/specification/dream.1.1.json",
"ados": {
"preTest": {
"communication": 2,
"interaction": 5,
"module": 1.0,
"play": 1,
"protocol": "ADOS-G",
"socialCommunicationQuestionnaire": 23,
"stereotype": 0,
"total": 7
}
},
"condition": "RET",
"date": "2017-11-02T12:32:42.369000",
"eye_gaze": {"rx": [],"ry": [],"rz": []},
"frame_rate": 25.1,
"head_gaze": {"rx": [],"ry": [],"rz": []},
"participant": {"ageInMonths": 47,"gender": "male","id": 37},
"skeleton": {
"elbow_left": {"confidence": [],"x": [],"y": [],"z": []},
"elbow_right": {"confidence": [],"x": [],"y": [],"z": []},
"hand_left": {"confidence": [],"x": [],"y": [],"z": []},
"hand_right": {"confidence": [],"x": [],"y": [],"z": []},
"head": {"confidence": [],"x": [],"y": [],"z": []},
"sholder_center": {"confidence": [],"x": [],"y": [],"z": []},
"sholder_left": {"confidence": [],"x": [],"y": [],"z": []},
"sholder_right": {"confidence": [],"x": [],"y": [],"z": []},
"wrist_left": {"confidence": [],"x": [],"y": [],"z": []},
"wrist_right": {"confidence": [],"x": [],"y": [],"z": []}
},
"task": "TT"
}The dataset includes variables related to specific tasks, such as imitation, joint attention, and turn-taking. These tasks assess the child's ability to mimic the robot’s actions, follow visual cues, and interact reciprocally in structured scenarios.
This task evaluates how accurately a child can mimic the robot's movements, emotional expressions, and sounds. Interactive imitation games are conducted to teach motor behaviors and improve imitation skills. The robot provides verbal instructions for the child to replicate its actions, with the imitation tasks organized by various levels of difficulty:
- Level 1 (Imitation with Objects): The child imitates actions involving objects, such as moving a toy car or pretending to drink from a cup.
- Level 2 (Imitation of Gestures): The child imitates meaningful movements, such as waving and saying "bye-bye."
- Level 3 (Imitation of Non-meaningful Movements): The child imitates abstract movements without an inherent meaning.
This task captures behaviors such as gaze direction, gestures, and verbalizations aimed at directing shared attention to specific objects.
The task begins with a brief explanation provided to the child, such as: “Now we will play another game. In this game, I will show you the objects I’ve seen in an office.” Two pictures are then displayed simultaneously on a large touch-screen table, one on the left and one on the right. The child must look at the picture indicated by the robot. The complexity of the task increases across three levels:
- Level 1: The robot uses a combination of verbal action (e.g., “Look!”), gestural action (e.g., pointing to a picture), and gaze action (e.g., making eye contact and then shifting its gaze to the picture).
- Level 2: The robot omits verbal cues and only uses gestural and gaze actions to direct the child’s attention.
- Level 3: The robot relies solely on gaze cues, omitting both verbal and gestural actions.
This task involves tracking the child’s responses, including contingent utterances, emotional expressions, and rational or maladaptive behaviors during interactions with the robot. The child and robot take turns in a series of structured activities presented on a touch-screen tablet (Sandtray). These activities include sharing information, categorizing items, and continuing repeating patterns. Each sub-task begins with an instruction or question from the robot, followed by the child’s response and the robot’s feedback based on the behavior.
1. Sharing information:
- Five pictures are displayed on the tablet. The child selects a picture during their turn and waits while the robot selects a picture during its turn.
2. Categories:
- Level 1: Three pictures are displayed (two representing categories and one item to be categorized). The child categorizes the item during their turn and waits while the robot categorizes during its turn. The categories are simple and age-appropriate (e.g., fruits vs. vegetables), and items appear one by one.
- Level 2: Ten pictures are displayed (two categories and eight items to be categorized). The child categorizes one item at a time during their turn and waits for the robot to do the same. The categories are more complex (e.g., ground vehicles vs. water vehicles).
3. Patterns:
- Level 1: Six pictures are displayed, including a repetitive pattern of two or three items. The child continues the pattern during their turn and waits for the robot to add to the pattern during its turn. The task focuses on simple criteria like geometric shapes (e.g., rectangle, rectangle, triangle, rectangle, ...).
- Level 2: Ten pictures are displayed, with a more complex pattern consisting of four repeating items. The child categorizes based on two criteria—geometric shape and color (e.g., green square, star, orange square, circle, ...).
- The DREAM Dataset: Supporting a data-driven study of autism spectrum disorder and robot enhanced therapy
- DREAM Dataset
- Open Source DREAM Project
- DREAM_Deliverable_D1.1 Intervention Definition
- DREAM_Deliverable_D1.4 Manual of best practice in robot enhanced therapy for autism spectrum disorder
- DREAM_Deliverable_D4.1 Sensorized Therapy Room Design and Algorithms for Data Sensing and Interpretation
- DREAM Data Visualizer
- Sensing-enhanced Therapy System for Assessing Children with Autism Spectrum Disorders: A Feasibility Study
- Computer Vision to the Rescue: Infant Postural Symmetry Estimation from Incongruent Annotations
- Two-Eye Model-Based Gaze Estimation from A Kinect Sensor