This is the public-facing code repository for the paper, "Gauging Engagement: Measuring Student Response to a Large-Scale College Advising Field Experiment," by Brian Heseung Kim, Katharine Meyer, and Alice Choe.
Interactive, text message-based advising programs have become an increasingly common strategy to support college access and success for underrepresented student populations. Despite the proliferation of these programs, we know relatively little about how students engage in these text-based advising opportunities and whether that relates to stronger student outcomes – factors that could help explain why we’ve seen relatively mixed evidence about their efficacy to date. In this paper, we use data from a large-scale, two-way text advising experiment focused on improving college completion to explore variation in student engagement using nuanced interaction metrics and automated text analysis techniques (i.e., natural language processing). We then explore whether student engagement patterns are associated with key outcomes including persistence, GPA, credit accumulation, and degree completion. Our results reveal substantial variation in engagement measures across students, indicating the importance of analyzing engagement as a multi-dimensional construct. We moreover find that many of these nuanced engagement measures have strong correlations with student outcomes, even after controlling for student baseline characteristics and academic performance. Especially as virtual advising interventions proliferate across higher education institutions, we show the value of applying a more codified, comprehensive lens for examining student engagement in these programs and chart a path to potentially improving the efficacy of these programs in the future.
Keywords: Text as data, college persistence, data science, natural language process, nudge, higher education, field experiment
This repository is for demonstration and illustration purposes for other interested researchers, rather than full replication; while we would ideally be able to publicly release all requisite data files and allow for complete reproduction by any interested parties, the data files contain PII and thus the analysis is not actually reproducible by the public. Even so, we have released our code with only minor adjustments (to protect the identities of partner institutions and other individuals) for future reference, as the code may nonetheless be useful for researchers attempting to examine trends and engagement patterns within their own mass texting contexts. We hope others find this to be a useful building block for similar analyses going forward, and welcome suggestions, feedback, or other constructive collaboration.
Note that segments of this analysis are conducted in Stata (for general data analyses), R (for some data analyses, visualization, and topic modeling), and Python (for sentiment analysis). We recognize this will present difficulties for others referencing the code, and are happy to support others where possible in recreating similar analyses in their chosen statistical languages.
More information is available in our public working paper. If you would like to know more, have questions, or would otherwise like to get in touch, please send an email to Brian Heseung Kim at brhkim@gmail.com