To Read or To Rotate? Comparing the Effects of Technical Reading Training and Spatial Skills Training on Novice Programming Ability

Overview

This repository contains the replication materials for the paper To Read or To Rotate? Comparing the Effects of Technical Reading Training and Spatial Skills Training on Novice Programming Ability as accepted at ESEC/FSE 2021.

Paper Abstract:

Understanding how to best support and train novice programmers is a critical component of producing better and more diverse software engineers. In this paper, we present the results of a controlled 11-week longitudinal study with 57 CS1 students comparing two skill-based interventions to improve programming performance. The first intervention we consider involves spatial training, an established baseline that is known to be helpful in engineering contexts. The second intervention we consider is a novel computer science focused technical reading training.

In our reading training, we teach strategies for summarizing scientific papers and understanding scientific charts and figures; most of the covered readings were CS1-accessible portions of computer science research papers. For the spatial training, we use a standardized training curriculum previously found to improve programming skills by focusing on spatial ability, the manipulation of shapes in imagination. We first replicate findings that both reading ability and spatial ability are correlated with programming success. Significantly, however, we find that those in our reading training exhibit larger programming ability gains than those in the standard spatial training (p = 0.02, f² = 0.10). We also find that reading training participants perform particularly well on programming problems that require tracing through code (p = 0.03, f² = 0.10). Our results suggest that this sort of technical reading training could be a good choice for novices who have not completed a CS1 class. Finally, we present a discussion on the implications of our results on future CS1 interventions, the possibilities for non-programming based training positively impacting software developers, and future directions for software engineering research.

Paper Authors:

• Madeline Endres: endremad@umich.edu
• Madison Fansher: mfansher@umich.edu
• Priti Shah: priti@umich.edu
• Westley Weimer: weimerw@umich.edu

Paper Pre-print:

The preprint of the paper can be found here on gitHub.

Subfolder directories:

• Reading Training: This contains a list of all lecture topics and associated CS readings. It also contains slide decks for each reading training session.
• Participant Forms: This has the qualitative and quantitative assessments given during the pre and post-tests that we can share.
• Data Analysis: Contains the scripts used to analyze our data.
• Recruitment Information: Contains recruitment flyers and pre-screening form

Accessing Participant Data:

We cannot publish our participant data on GitHub. If interested in accessing participant data, please contact Madeline Endres at endremad@umich.edu and we can work on sharing the data with the permission of our IRB.

Non Author Materials:

For this study, we used several validated assessments that we are unable to post here due to the assessments’ policies. For more information on the SCS1 please see, this link or this link. For the PSVT:R, please contact So Yoon Yoon. Finally, we are unable to share the standardized Sorby spatial training materials. If interested, see https://www.higheredservices.org/ for more information about the spatial training materials.