This is the code repo for the forthcoming book, Computational Psychometrics, edited by Alina A. von Davier, Robert J. Mislevy and Jiangang Hao. Please note that only chapter 7 to chapter 14 contain code examples and the other chapters do not have any code examples. Click here to visit the Springer page for the book.
- Python tutorials: https://docs.python.org/3/tutorial/
- R tutorials: https://cran.r-project.org/doc/manuals/r-release/R-intro.html
“The future is already here—it’s just not very evenly distributed.”—attributed to W. Gibson
Digitally based learning and assessment systems generate complex data that record learners' interaction with the tasks or items at finer time granularity than in traditional settings. These rich process data can provide new opportunities for validating assessments, improving measurement precision, revealing response patterns/styles, uncovering group differences, detecting unintended behaviors, identifying new constructs, and providing more actionable feedback to learners and other stakeholders. But these benefits do not come for free. The significantly increased volume, velocity, and variety of data pose new challenges to researchers in psychometrics to handle, analyze, and interpret them in order to materialize the value of these rich data.
The techniques needed to handle these complex data are not well-covered in most existing treatments of psychometric methods. It has become necessary to extend psychometric methodology to new techniques, such as those for creating quantitative representations of the complex responses (such as video, audio, and keystroke) and those for relating the complex evidentiary representations to the targeted constructs.
A concise new term would be beneficial to facilitate the exchanges among researchers on this new front of psychometrics. However, coming up with a concise name to encompass a discipline with such a vast intension is challenging. After years of back and forth, we noted that despite their widespread differences the new methodologies share one common feature: computational models. We thus arrived at the term Computational Psychometrics (von Davier, 2015). We want to capture the essential feature of this new discipline, encompassing data-driven computational algorithms, while at the same time, establishing its alignment to fundamental concepts of psychometrics. This may not be a perfect name, as always in the history of science, but it is a concise one that highlights the key features of this new discipline.
Computational psychometrics thus aims at integrating techniques from data science and machine learning into psychometrics, guided by well-established psychometric principles in measurement science. Just as psychometrics is not a simple collection of statistical methods for educational measurement, computational psychometrics is more than a simple aggregation of data science and machine learning methods for use in measurement contexts. Insights into educational measurement are essential for realizing the potential of applying the new techniques in those contexts.
By drawing examples from real-world use cases, this edited volume is intended to further define the scope of this new discipline and to serve as a steppingstone for students and researchers in psychometrics prepare for the design, development, and analysis of the learning and assessment systems with their increasingly big and complex data. A strength of the volume resides in its GitHub-site companion, which provides a repository for the code —in R or Python—for all of the methodological chapters.
This volume mirrors the societal changes, advances in technology and computational power, and the wide-spread adoption of digital learning and assessments. It is our goal that the methods provided in this book will enable researchers and developers to create systems and tools for better access, more affordability, broader inclusion, and higher quality education for everyone, everywhere.
Alina A. von Davier, Newton, MA
Robert Mislevy, Severna Park, MD
Jiangang Hao, Princeton, NJ
Chapter 1. Introduction to Computational Psychometrics: towards a principled integration of data science and machine learning techniques into psychometrics
Authors: Alina A. von Davier, Robert J. Mislevy and Jiangang Hao
Abstract: In this chapter we articulate what is computational psychometrics, why we need a volume focused on it, and how this book contributes to the expansion of psychometric toolbox to include methodologies from machine learning and data science in order to address the complexities of big data collected from virtual learning and assessment systems. We also discuss here the structure of the edited volume, how each chapter contributes to enhancing the psychometrics science and our recommendations for further readings.
Chapter 2. Next generation learning and assessment: what, why and how
Author: Robert J. Mislevy
Abstract: Computational psychometrics is a blend of stochastic processes theory, computer science-based methods, and theory-based psychometric approaches that may aid the analyses of complex data from performance assessments. This chapter discusses the grounds for using complex performance assessments, the design of such assessments so that useful evidence about targeted abilities will be present in the data to be analysed, and roles that computational psychometric ideas and methods can play. It first provides background on a situative, sociocognitive, perspective on human capabilities and how we develop them and use them—a perspective we believe is necessary to synthesize the methodologies. Next it reviews the form of evidentiary argument that underlies the evidence-centered approach to design, interpretation, and use of educational assessments. It then points out junctures in extensions of the argument form where computational psychometric methods can carry out vital roles in assessment of more advanced constructs, from more complex data, in new forms and contexts of assessment. It concludes by reflecting on how one reconceives and extends the notions of validity, reliability, comparability, fairness, and generalizability to more complex assessments and analytic methods.
Chapter 3. Computational psychometrics
Authors: Alina A. von Davier, Kristen DiCerbo and Josine Verhagen
Abstract: In recent years the advances in technology provided affordances for learning and assessments opportunities. In this chapter we first describe computational psychometrics as a framework for the measurement of learners’ skills, knowledge, and abilities. We discuss the changes in educational measurement that led to the need for expanding the psychometrics toolbox and describe the properties of psychometric data. We then give an example of a class of models, the Dynamic Bayesian Models that encompass many traditional psychometric models and machine-learning algorithms. We conclude by emphasizing that model complexity and power need to be balanced with the responsibility for transparency and fairness towards stakeholders.
Chapter 4. Virtual performance-based assessments
Authors: Jessica Andrews Todd, Robert J. Mislevy, Michelle LaMar and Sebastiaan de Klerk
Abstract: Virtual performance-based assessments (VPBAs) are environments for test takers to interact with systems, sometimes including other persons or agents, in order to provide evidence about their knowledge, skills, or other attributes. Examples include tasks based on interactive simulations, games, branching scenarios, and collaboration among students communicating through digital chats. They may be used for summative purposes, as in certification examinations, or for other purposes, as in intelligent tutoring systems and exploratory learning environments. They afford opportunities to obtain direct evidence about capabilities that inherently involve interaction, such as inquiry and collaboration. Our focus here is digital, usually with regard to the environment but always with regard to the form of data. Digital data capture makes it possible to acquire rich details about students’ actions and the evolving situations in which they occur. The challenges they pose to psychometrics lie in designing VPBAs to optimally evoke the targeted capabilities, providing students with affordances that evidence that cognition, capturing the relevant aspects of the performances, identifying meaningful patterns in performances that constitute evidence about the targeted capabilities, and providing an inferential framework for synthesizing the evidence and characterizing its properties. This chapter provides an introduction to VPBAs and psychometric considerations in VPBA design and analysis.
Chapter 5. Knowledge Inference Models Used in Adaptive Learning
Authors: Maria Ofelia, M.O.Z. San Pedro and Ryan S. Baker
Abstract: This chapter provides an overview of adaptive learning and examines the student model component used in adaptive learning systems. Established and more recent approaches to student modeling that infer student knowledge (i.e. what students know at any given moment during the learning experience) are discussed, as student knowledge is the most common learner characteristic widely assessed in large-scale adaptive systems. This chapter concludes with a discussion of the limitations of the current generation of adaptive learning systems, and areas of potential for future progress.
Chapter 6. Concepts and models from Psychometrics
Authors: Robert J. Mislevy and Maria Bolsinova
Abstract: The concepts and methods of psychometrics originated under trait and behavioral psychology, with relatively simple data, used mainly for purposes of prediction and selection. Ideas emerged over time that nevertheless hold value for the new psychological perspectives, contexts of use, and forms of data and analytic tools we are now seeing. In this chapter we review some fundamental models and ideas from psychometrics that can be profitably reconceived, extended, and augmented in the new world of assessment. Methods we address include classical test theory, generalizability theory, item response theory, latent class models, cognitive diagnosis models, factor analysis, hierarchical models, and Bayesian networks. Key concepts are these: (1) The essential nature of psychometric models (observations, constructs, latent variables, and probability-based reasoning). (2) The interplay of design and discovery in assessment. (3) Understanding themeasurement issues of validity, reliability, comparability, generalizability, and fairness as social values that pertain even as forms of data, analysis, context, and purpose evolve.
Chapter 7. Bayesian Inference in Large-Scale Computational Psychometrics
Authors: Gunter Maris, Timo Bechger and Maarten Marsman
Abstract: This chapter provides an introduction to Bayesian inference using Markov Chain Monte Carlo (MCMC) methods. We focus on two popular MCMC methods: Metropolis-Hastings and the Gibbs sampler. A Metropolis-Hastings algorithm developed by Marsman et al. (Sci Rep 5:9050, 1–7, 2015) will be used to illustrate how MCMC can be done for a wide range of models in computational statistics.
Chapter 8. Data science perspectives
Authors: Jiangang Hao and Robert J. Mislevy
Abstract: Digitally based learning and assessment systems generate large volumes of complex process data. The next generation psychometricians need to acquire new data science skills to meet the data challenge. In this chapter, we summarize data science skills and identify the subset that psychometricians need to prioritize. We introduce an evidence identification centered data design (EICDD) process during the task design, as an important way to address the data challenges from digitally based assessments. We describe some specific data techniques to parse and process complex process data with example codes in Python programming language. We also outline the general methodological strategies when dealing with process data from digitally based assessments.
Chapter 9. Supervised machine learning
Author: Jiangang Hao
Abstract: Machine learning refers to a set of methodologies that allow computers to “learn” the relationship among numerical representations of data. In this Chapter, we focus on an important branch of machine learning, supervised machine learning, and introduce three widely used supervised learning methods, the Support Vector Machine, Random forest, and Gradient Boosting Machine. Python codes examples are included to show how to use these methods in practice.
Chapter 10. Unsupervised machine learning
Author Pak Chunk Wong
Abstract: The chapter introduces the concept of machine learning with an emphasis on unsupervised learning algorithms and applications. The discussion starts with a brief background on machine learning and then a high-level discussion on the differences between supervised and unsupervised learning algorithms. We present three categories of unsupervised machine learning techniques that include clustering, outlier detection, and dimension reduction; five prevailing unsupervised learning algorithms that include K-means, agglomerative clustering, DBSCAN, principal component analysis, and multidimensional scaling; and five Python programming examples that demonstrate the learning concepts and results using psychometric assessment data collected from an online collaborative problem-solving environment. This chapter demonstrates the potential of machine learning and highlights the opportunities it presents in psychometric research and development.
Chapter 11. AI and deep learning for educational research
Authors: Yuchi Huang and Saad M. Khan
Abstract: There is a growing need for assessment and learning tools that capture a broad range of learner behavior necessary for the evaluation of skills such as problem solving, communication and collaboration. In these real-world applications student data is captured with a high degree of granularity, variety of temporal scales and in a multitude of modalities. Unfortunately such complex, noisy and unstructured data limit the applicability of traditional models of measurement and psychometrics designed to extract evidence of competency from item response data. In this chapter, we present recent advances in AI and Machine Learning that can be utilized for measurement of a variety of complex constructs and competencies. These models include frameworks such as deep neural networks and adversarial generative networks that enable us to harness concept hierarchies and the latent structure within data to learn increasingly complex representations and make powerful predictions.
Chapter 12. Time series and stochastic processes
Authors: Peter Halpin, Lu Ou and Michelle LaMar
Abstract: This chapter addresses some statistical modeling approaches for time series data and discusses their potential for psychometric applications. We adopt a broad conceptualization of time series, including under this rubric any type of data that involves serial statistical dependence. Such dependence may be represented in continuous time, discrete time, or in a purely sequential manner. This chapter begins by discussing the relationships among these three representations and offers some general advice on when each might prove useful. We then provide an overview of three modeling frameworks that exemplify the different representations of statistical dependence: Markov decision processes, state-space modeling, and temporal point processes. For each modeling framework, we discuss its specification, its psychometric interpretation, and provide a brief numeric example including R code.
Chapter 13. Social network analysis
Author: Mengxiao Zhu
Abstract: Supported by advances in technology, simulation-, scenario- and game-based assessments (DiCerbo & Behrens, 2012; Mislevy et al., 2014) provide opportunities for the students to interact with complex tasks. Rich process data can be collected during the assessment, such as log data of student response actions (e.g., Zhu, Shu, & von Davier, 2016), keystroke data (e.g., Almond, Deane, Quinlan, Wagner, & Sydorenko, 2012), and eye-tracking data (e.g., Tai, Loehr, & Brigham, 2006). Process data record the series of activities conducted by students during problem-solving processes and contain information not represented in the final answers. One useful direction in which to study process data is to explore how students transit from one action to other actions, or from one state to other states. In this chapter, we introduce the basic concepts and methods of Social Network Analysis (SNA) and discuss related applications in visualizing and analyzing process data using SNA to understand the transitions in response process data.
Chapter 14. Text mining and automated scoring
Authors: Michael Flor and Jiangang Hao
Abstract: Natural Language Processing (NLP) is playing an increasingly important role in learning and assessments. Some typical applications of NLP in education include automated scoring, automated item generation, conversation-based assessments, writing assistants, text mining for education, and so on. In this chapter, we aim at introducing some basics of NLP through two typical applications in educational contexts, text mining and automated scoring. We hope readers can get an overall picture of NLP and get familiarized with some basic tools for handling natural language data, which may serve as stepping stones for their future work with NLP.
After the book was published, we received much feedback from the learning and assessment community. We want to thank all the readers for their commendations, suggestions, and comments on the book. Meanwhile, we realized that it might be more beneficial to share additional thoughts on how we embarked on creating this edited volume to introduce computational psychometrics. Three main reasons pushed us to this effort.
First of all, as we briefly mentioned in the preface and the chapter 1, there is an intrinsic need to expand the existing psychometric methodologies to include new methods from, e.g., data science and machine learning, to address the new challenges of learning and assessment in the digital age. When many new methods are included, introducing a new term to encompass these new features will be more convenient and effective for communication in the community. Historically, this is the general process of how new disciplines emerge, such as how psychometrics has emerged from statistics.
Second, we intended to help address the practical challenge of preparing the workforce. Over the past few years, we all went through some painstaking efforts to hire people with the right combination of skills to meet the challenge of digital learning and assessment. We observed that many applicants from psychometrics programs do not have the needed data science/machine learning skills (and mindsets) to process and model complex data from digital tasks. In contrast, applicants with data science/machine learning skills from other disciplines, such as computer science, generally know very little about the core values of psychometrics. In practice, hiring people who do not know the core values of the substantive area poses a big retention challenge for organizations, as they may quickly move on if they find they are not interested in the area at all after a few months. Therefore, we feel it is imperative to prioritize a set of new methodologies and integrate them with the core values of psychometrics in a principled manner to help prepare a stable workforce for digital learning and assessment in the future.
Finally, we noticed a lack of a bridge for people from other quantitative disciplines (such as computer science, applied mathematics, physics, and others) to digital learning and assessment. There are many talents with superb technical skills but know little about the values and principles of learning and assessment. We believe that providing a concise coverage of psychometrics' established values and methods could help them better understand how to apply their skills to join forces to promote learning and assessment in a digital age.
As such, we decided to create an edited volume with carefully selected topics contributed by experts we can reach. We hope the book could help readers who have been or will be working in the exciting areas of digital learning and assessments to better understand the methods and principles, and communicate them efficiently under the name of computational psychometrics.