Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
27 commits
Select commit Hold shift + click to select a range
b4809d3
rescaled all figs to fit an A4 page, adjusted any text smaller than 6…
paxtonfitzpatrick Nov 24, 2025
3d3b6e9
add .fff files (LaTeX endfloat package) to gitignore
paxtonfitzpatrick Nov 24, 2025
5f6ce63
remove keywords, reorder sections, embed references, move figs to end
paxtonfitzpatrick Nov 24, 2025
1ee4c51
remove non-essential quotes and italics
paxtonfitzpatrick Nov 25, 2025
017c7fd
remove all italics except lecture names
paxtonfitzpatrick Nov 25, 2025
81a7a82
fixed capitalization in HallGree08 book title
paxtonfitzpatrick Nov 25, 2025
0d838b0
WIP Nature Comms author checklist
paxtonfitzpatrick Nov 25, 2025
28860c2
ignore MS Word temp files
paxtonfitzpatrick Nov 25, 2025
fb84d4f
re-linked corrmat source file to update
paxtonfitzpatrick Nov 25, 2025
d04b1f8
recompiled with updated supp figs
paxtonfitzpatrick Nov 25, 2025
997caa2
supplementary materials -> supplementary information
paxtonfitzpatrick Nov 25, 2025
ad991ed
Supplementary results -> Supplementary discussion
paxtonfitzpatrick Nov 25, 2025
f70c4a0
fixed a typo in a docstring
paxtonfitzpatrick Nov 26, 2025
9fe038e
note sample sizes and measures of center in captions, add khan academ…
paxtonfitzpatrick Nov 26, 2025
d0c1408
update fig 5 to display individual points
paxtonfitzpatrick Nov 28, 2025
264cb3f
note stats results in fig 6 caption
paxtonfitzpatrick Nov 28, 2025
ab73c14
add some required notes to Methods > Participants section
paxtonfitzpatrick Dec 1, 2025
e2d3aab
rephrase fig 6 interpretation
paxtonfitzpatrick Dec 1, 2025
fe1cb53
add statement re: sex/gender considerations
paxtonfitzpatrick Dec 1, 2025
c42e1b5
update title
paxtonfitzpatrick Dec 1, 2025
ae9d7c6
removed khan academy lecture screenshot background from fig 1
paxtonfitzpatrick Dec 1, 2025
5475f36
updated author checklist and reporting summary
paxtonfitzpatrick Dec 1, 2025
e637d46
final recompile
paxtonfitzpatrick Dec 1, 2025
0b95b3c
updated individual knowledge & learning maps captions
paxtonfitzpatrick Dec 1, 2025
7906f97
re-add quotes around English in survey response section
paxtonfitzpatrick Dec 1, 2025
ad9d581
final updates before creating zenodo DOI
paxtonfitzpatrick Dec 1, 2025
04b30a5
Merge pull request #114 from paxtonfitzpatrick/revision-4
paxtonfitzpatrick Dec 1, 2025
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
4 changes: 4 additions & 0 deletions .gitignore
Original file line number Diff line number Diff line change
Expand Up @@ -37,6 +37,7 @@ exp/server.log
# latex extensions
paper/**/*.aux
paper/**/*.fdb_latexmk
paper/**/*.fff
paper/**/*.fls
paper/**/*.log
paper/**/*.out
Expand All @@ -45,3 +46,6 @@ paper/**/*.blg
paper/**/*.bbl
texput.log
paper/.#compile.sh

# MS Word temp files
paper/**/~$*
8 changes: 4 additions & 4 deletions README.md
Original file line number Diff line number Diff line change
@@ -1,4 +1,4 @@
# Text embedding models yield high-resolution insights into conceptual knowledge from short multiple-choice quizzes
# Text embedding models yield detailed conceptual knowledge maps derived from short multiple-choice quizzes

<p align="center">
<a href="https://psyarxiv.com/dh3q2">
Expand All @@ -7,8 +7,8 @@
</p>

This repository contains all data and code used to produce the paper
"[_Text embedding models yield high-resolution insights into conceptual
knowledge from short multiple-choice quizzes_](https://psyarxiv.com/dh3q2)" by
"[_Text embedding models yield detailed conceptual
knowledge maps derived from short multiple-choice quizzes_](https://psyarxiv.com/dh3q2)" by
Paxton C. Fitzpatrick, Andrew C. Heusser, and Jeremy R. Manning.

We also include reproducible environments for running our experiment and
Expand All @@ -17,7 +17,7 @@ analyses via [Docker](https://www.docker.com/).

## Table of Contents

- [Text embedding models yield high-resolution insights into conceptual knowledge from short multiple-choice quizzes](#text-embedding-models-yield-high-resolution-insights-into-conceptual-knowledge-from-short-multiple-choice-quizzes)
- [Text embedding models yield detailed conceptual knowledge maps derived from short multiple-choice quizzes](#text-embedding-models-yield-detailed-conceptual-knowledge-maps-derived-from-short-multiple-choice-quizzes)
- [Table of Contents](#table-of-contents)
- [Repo Organization](#repo-organization)
- [Installing Docker](#installing-docker)
Expand Down
97 changes: 58 additions & 39 deletions code/notebooks/main/4_reconstructing-knowledge.ipynb

Large diffs are not rendered by default.

4 changes: 2 additions & 2 deletions code/notebooks/main/6_knowledge-smoothness.ipynb
Original file line number Diff line number Diff line change
Expand Up @@ -121,7 +121,7 @@
" A `pandas.Series` whose values are iterables of length 2 \n",
" representing lower and upper confidence interval bounds for the \n",
" accross-participants mean overall/raw p(correct) for each quiz.\n",
" interp_func : float, optional\n",
" interp_freq : float, optional\n",
" If provided, interpolate the series of by-distance p(correct)\n",
" values for each quiz & accuracy of reference question to the \n",
" given frequency before computing the intersections.\n",
Expand Down Expand Up @@ -1296,7 +1296,7 @@
"name": "python",
"nbconvert_exporter": "python",
"pygments_lexer": "ipython3",
"version": "3.9.7"
"version": "3.9.18"
}
},
"nbformat": 4,
Expand Down
Binary file not shown.
Binary file modified paper/admin/nr-reporting-summary.pdf
Binary file not shown.
Binary file modified paper/figs/active-topics.pdf
Binary file not shown.
Binary file modified paper/figs/bos-qcorrs-peaks.pdf
Binary file not shown.
Binary file modified paper/figs/content-mastery.pdf
Binary file not shown.
Binary file modified paper/figs/experiment.pdf
Binary file not shown.
Binary file modified paper/figs/forces-qcorrs-peaks.pdf
Binary file not shown.
Binary file modified paper/figs/individual-knowledge-maps-quiz1.pdf
Binary file not shown.
Binary file modified paper/figs/individual-knowledge-maps-quiz2.pdf
Binary file not shown.
Binary file modified paper/figs/individual-knowledge-maps-quiz3.pdf
Binary file not shown.
Binary file modified paper/figs/individual-learnings-maps-quiz1-2.pdf
Binary file not shown.
Binary file modified paper/figs/individual-learnings-maps-quiz2-3.pdf
Binary file not shown.
Binary file modified paper/figs/knowledge-smoothness.pdf
Binary file not shown.
Binary file modified paper/figs/knowledge_and_learning_maps.pdf
Binary file not shown.
Binary file modified paper/figs/lecture-question-similarity.pdf
Binary file not shown.
Binary file modified paper/figs/model-comparison.pdf
Binary file not shown.
Binary file modified paper/figs/predict-knowledge-questions.pdf
Binary file not shown.
Binary file modified paper/figs/sliding_windows.pdf
Binary file not shown.
Binary file modified paper/figs/source/bos-qcorrs-peaks.pdf
Binary file not shown.
Binary file modified paper/figs/source/content-mastery.pdf
Binary file not shown.
Binary file modified paper/figs/source/forces-qcorrs-peaks.pdf
Binary file not shown.
Binary file modified paper/figs/source/individual-knowledge-maps-quiz1.pdf
Binary file not shown.
Binary file modified paper/figs/source/individual-knowledge-maps-quiz2.pdf
Binary file not shown.
Binary file modified paper/figs/source/individual-knowledge-maps-quiz3.pdf
Binary file not shown.
Binary file modified paper/figs/source/individual-learnings-maps-quiz1-2.pdf
Binary file not shown.
Binary file modified paper/figs/source/individual-learnings-maps-quiz2-3.pdf
Binary file not shown.
Binary file modified paper/figs/source/lecture-question-similarity.pdf
Binary file not shown.
Binary file modified paper/figs/source/topic-weights-corrmat.pdf
Binary file not shown.
Binary file modified paper/figs/source/topic-weights.pdf
Binary file not shown.
Binary file modified paper/figs/topic-weights.pdf
Binary file not shown.
Binary file modified paper/figs/word-overlap-comparison.pdf
Binary file not shown.
Binary file modified paper/main.pdf
Binary file not shown.
1,010 changes: 740 additions & 270 deletions paper/main.tex

Large diffs are not rendered by default.

Binary file modified paper/supplement.pdf
Binary file not shown.
18 changes: 9 additions & 9 deletions paper/supplement.tex
Original file line number Diff line number Diff line change
Expand Up @@ -32,8 +32,8 @@
\newcommand{\knowledgeMaps}{7}
\newcommand{\topicModelMethods}{\textit{Constructing text embeddings of multiple lectures and questions}}

\title{\textit{Supplementary materials for}: Text embedding models yield
high-resolution insights into conceptual knowledge from short multiple-choice
\title{\textit{Supplementary information for}: Text embedding models yield
detailed conceptual knowledge maps derived from short multiple-choice
quizzes}

\author{Paxton C. Fitzpatrick\textsuperscript{1}, Andrew C.
Expand Down Expand Up @@ -326,7 +326,7 @@ \section*{Supplementary figures}
\caption{\textbf{Individual participants' knowledge maps estimated from
Quiz 1 responses.} Each panel is in the same format as the knowledge map
displayed in the left panel of Figure~\knowledgeMaps A in the main text,
but here the maps are shown separately for each participant.}
but here each participant's map shows only the questions they answered on Quiz 1.}

\label{fig:knowledge-maps-q1}
\end{figure}
Expand All @@ -339,7 +339,7 @@ \section*{Supplementary figures}
\caption{\textbf{Individual participants' knowledge maps estimated from
Quiz 2 responses.} Each panel is in the same format as the knowledge map
displayed in the middle panel of Figure~\knowledgeMaps A in the main text,
but here the maps are shown separately for each participant.}
but here each participant's map shows only the questions they answered on Quiz 2.}

\label{fig:knowledge-maps-q2}
\end{figure}
Expand All @@ -352,7 +352,7 @@ \section*{Supplementary figures}
\caption{\textbf{Individual participants' knowledge maps estimated from
Quiz 3 responses.} Each panel is in the same format as the knowledge map
displayed in the right panel of Figure~\knowledgeMaps A in the main text,
but here the maps are shown separately for each participant.}
but here each participant's map shows only the questions they answered on Quiz 3.}

\label{fig:knowledge-maps-q3}
\end{figure}
Expand All @@ -365,7 +365,7 @@ \section*{Supplementary figures}
\caption{\textbf{Individual participants' learning maps estimated from
Quiz 1 and 2 responses.} Each panel is in the same format as the learning map
displayed in the left panel of Figure~\knowledgeMaps B in the main text,
but here the maps are shown separately for each participant.}
but here each participant's map shows only the questions they answered on Quizzes 1 \& 2.}

\label{fig:learning-maps-q1_2}
\end{figure}
Expand All @@ -378,7 +378,7 @@ \section*{Supplementary figures}
\caption{\textbf{Individual participants' learning maps estimated from
Quiz 2 and 3 responses.} Each panel is in the same format as the learning map
displayed in the right panel of Figure~\knowledgeMaps B in the main text,
but here the maps have been computed separately for each participant.}
but here each participant's map shows only the questions they answered on Quizzes 2 \& 3.}

\label{fig:learning-maps-q2_3}
\end{figure}
Expand All @@ -400,7 +400,7 @@ \section*{Supplementary figures}

\begin{figure}[tp]
\centering
\includegraphics[width=0.57\textwidth]{figs/word-overlap-comparison}
\includegraphics[width=0.52\textwidth]{figs/word-overlap-comparison}

\caption{\textbf{Topic correlations versus word overlap.} \textbf{A.
Single-timepoint comparisons for \textit{Four Fundamental Forces}.} Each
Expand Down Expand Up @@ -428,7 +428,7 @@ \section*{Supplementary figures}

\FloatBarrier

\section*{Supplementary results}
\section*{Supplementary discussion}

\doublespacing

Expand Down