Automatic real-time hand tracking enhances adolescents’ spatial learning by eliminating haptic feedback

Spatial skills underlie how humans acquire, represent, organise, manipulate, and navigate their environment, and therefore are fundamental for survival and proper function. Spatial skills require the integration of sensory-motor information with higher-order cognitive representations. Yet, the role of sensory modality in enhancing these skills through training is limited. Here, we tested a new application for real-time machine-learning-based hand tracking as a manipulator of sensory modality that enhances spatial skills in late childhood. To that end, child students completed an 8-session school training programme to improve their spatial skills. Children were randomly assigned into one of two training regimens: Hapto-Visual (HV; students constructed a physical 2D polydron net from different perspectives of a 3D cube) or Visual-Only (VO; students constructed a digitised version of the 2D polydron net without tactile information). The efficacy of the training was assessed using traditional spatial skill tests (Modified Mental Rotation and Mental Paper-Folding Tasks) administered before and after the training programme. We found significant improvement in both groups, with a significantly enhanced performance by the VO group compared to the HV group. Our findings suggest that omitting haptic feedback during spatial training compels reliance on mental representations, thereby bolstering spatial skills more effectively. These findings present a new application for real-time hand-tracking technology in educational settings and demonstrate its cost-effective potential to advance spatial abilities in young students.