Encoding EEG Signals to Examine Human-Like Next-Word Prediction Behaviour in Language Models

Overview

While LMs are trained to excel at predicting the next word (or token) in a sequence given prior context, this represents only one aspect of intelligence. Although humans also share this predictability, our predictive capabilities are far more diverse. Neuroscience research reveals that neural responses, such as ERPs (Event-related potentials), are critical indicators of brain activity during language processing. Particularly, lexical surprisal correlates robustly with brain responses, influencing ERPs after word onset. In this paper, we generate regressors based on two information measures, including top-1 prediction and surprisal, to predict ERP patterns in humans and LMs. Our results suggest that while the more advanced LMs show a close correspondence to human performance in word prediction accuracy, only surprisal potentially correlates with language-processing ERPs, especially for open-class words with high semantic content. More importantly, our findings also challenge the assumption that increasing model size and training will consistently lead to improved convergence with human-like linguistic processing.

Citation

Link to the paper:

This repository contains the source code for reproducing the paper, along with step-by-step instructions as follows:

• Environment Setup
• Download Dataset
• Run Preprocessing
• Reproduce Results

Environment Setup

After cloning repository github, going to the DERCo folder and do the steps as follows

1. Install Python (Setup instruction). Note: In this project, we used Python 3.10.9
2. Install Conda (Conda Installation) or similar environment systems
3. Create a virtual enviroment

conda create --name [name of env] python==[version]

For example,

conda create --name encoding python==3.10.9

4. Activate enviroment

conda activate [name of env]

Dowload the project (git)

Clone the project

git clone [anonymous github link]

Install Python packages

pip3 install -r requirements.txt 

Project Organization

├── README.md          <- The top-level README for developers using this project.
├── data               <- Contains all the datasets used in the project.
│   ├── EEG_data                    <- Raw and preprocessed EEG (Electroencephalography) data.
│   ├── label_data                  <- Labels or annotations corresponding to the EEG data.
│   ├── fairy_tales_corpus          <- A dataset of fairy tales used for analysis.
│   ├── article                     <- Contains processed article-based data such as transcripts based on each sentences or correct words.
│   │   ├── correct_words_article_0.npy
│   │   ├── correct_words_article_1.npy
│   │   ├── correct_words_article_2.npy
│   │   ├── correct_words_article_3.npy
│   │   ├── correct_words_article_4.npy
│   │   ├── sentences_article_0.npy
│   │   ├── sentences_article_1.npy
│   │   ├── sentences_article_2.npy
│   │   ├── sentences_article_3.npy
│   │   ├── sentences_article_4.npy
│   ├── human_performance         <- Data related to human performance in cloze procedure.
│   │   ├── article_0_human_all_predictions.csv
│   │   ├── article_0_human_performance.csv
│   │   ├── article_0_human_predictions.csv
│   │   ├── article_1_human_all_predictions.csv
│   │   ├── article_1_human_performance.csv
│   │   ├── article_1_human_predictions.csv
│   │   ├── ..._human_all_predictions.csv
│   │   ├── ..._human_performance.csv
│   │   ├── ..._human_predictions.csv
│   │   ├── article_4_human_all_predictions.csv
│   │   ├── article_4_human_performance.csv
│   │   ├── article_4_human_predictions.csv
├── photo   <- Contains visual data, figures, or reference images.
├── utils                   <- Utility scripts for preprocessing EEG data and analysis
│   ├── config.py           <- Configuration file for parameters and settings.
│   ├── eeg_helpers.py      <- Helper functions for EEG data preprocessing and analysis.
│   ├── plot_helpers.py     <- Functions for plotting EEG-related data and visualizations.
│   ├── predict_eeg.py      <- Script for making predictions based on different regressors.
│   ├── get_filenames.py    <- Functions to retrieve and manage file names.
│   ├── get_sentences.py    <- Extracts and processes sentences from datasets.
│   ├── process_words.py    <- Handles word-level processing.
│   ├── process_Ngrams.py   <- Functions to analyze and process N-gram data.
│   └── calculate_metric.py <- Computes various metrics for evaluating EEG-related predictions.
├── visualization                   <- Scripts for visualising EEG data and other results.
│   ├── plot_comparison.py          <- Plots comparison between different models according to different metrics.
│   ├── plot_grand_averaged.py      <- Creates grand-averaged EEG plots.
│   ├── plot_topography.py          <- Generates EEG topographic maps.
│   └── plot_measure_comparison.py  <- Visualizes comparisons between different evaluation measures.
├── requirements.txt        <- The requirements file for reproducing the analysis environment, e.g.
│                         generated with `pip freeze > requirements.txt`.
├── run_regression.py       <- Runs regressors to predict neural response for the next-word prediction.
├── run_correlation.py      <- Computes correlation analyses between predicted neural responses and true responses.
├── run_llms_metric.py          <- Calculates measures using LLMs 
├── run_human_metric.py         <- Calculates measures using cloze probability
├── run_Ngrams_metric.py        <- Calculates measures using Ngrams
├── run_fairy_tales_corpus.py   <- Processes and analyzes the fairy tale corpus.
└── run_preprocessing.py        <- Handles preprocessing of raw EEG and textual data.

---

Download Dataset

EEG recordings were obtained from 22 native English speakers as they read The Grimm Brothers' Fairy Tales. Word-by-word cloze probabilities for each word were also gathered using a cloze procedure conducted on the Mechanical Turk crowdsourcing platform. Further information on EEG data collection and preprocessing can be found in the DERCo dataset.

EEG Data

Approach 1:

To download the DERCo dataset, please

1. Go directly to the OSF link.
2. The follow the path OSF Storage / EEG-based Reading Experiment / EEG_data.
3. Download preprocessed folder.
4. After dowloading, move to path: data/EEG_data/

Approach 2:

1. Install osf to download EEG data

pip install osfclient

2. Fecth the data

osf -p rkqbu clone

3. Only keep preprocessed folder, and move to path: data/EEG_data/

Fairy Tales Corpus

• Paper: Fairy tale corpus organization using latent semantic mapping and an item-to-item top-n recommendation algorithm
• Paper Link: link.
• Download link: link.
• After dowloading, move all folders in corpus-map to path: data/fairy_tales_corpus/

NOTE: Remove .gitkeep files in data/EEG_data and data/fairy_tales_corpus after downloading data.

Preprocessing

1. Get all tales in the fairy tales corpus (excluding those used to collect EEG data).

python run_fairy_tales_corpus.py -statistics [True or False]

• statistics (bool): Optional; Show statistics of the corpus or not. Default is False.

Output is the filenames.pkl in the label_data folder. The filenames.pkl file contain a list or collection of filenames representing the stories in the fairy tales corpus.

2. Extract all sentences in the corpus

python run_preprocessing_corpus.py

Output is the sentences.pkl in the label_data folder. The sentences.pkl file contain a list of all sentences in the fairy tales corpus.

Reproduce Results

For each language model (LM), we will estimate two measures: top1_prediction and surprisal.

N-gram models:

We trained separate models, including bigrams, trigrams, and quadgrams, using no smoothing, Laplace smoothing, and Kneser-Ney smoothing.

python run_Ngrams_metric.py -technique=[name_of_smoothing]

Default: No smoothing

• name_of_smoothing: Laplace, KneserNey

Example,

python run_Ngrams_metric.py -technique=Laplace

Human performance:

python run_human_metric.py

GPT Families:

For GPT Families, the code would be run using GPU.

• Model's names: gpt2, gpt2-medium, gpt2-large, EleutherAI/gpt-neo-125M, EleutherAI/gpt-neo-1.3B, EleutherAI/gpt-neo-2.7B.

Model	Size	Link
GPT2 (or GPT2-Small)	~124M	Here
GPT2-Medium	~355M	Here
GPT2-Large	~774M	Here
GPT-Neo 125M	~125M	Here
GPT-Neo 1.3B	~1.3B	Here
GPT-Neo 2.7B	~2.7B	Here

python run_llms_metric.py -model=[model_name]

Example,

python run_llms_metric.py -model=gpt2

Correlation (Pearson and Spearman)

python run_correlation.py -model1=[model1-name] -model2=[model2-name] -method=[name-technique]

• model-name: human, bigrams, trigrams, quadgrams, gpt2, gpt2-medium, gpt2-large, EleutherAI/gpt-neo-125M, EleutherAI/gpt-neo-1.3B, EleutherAI/gpt-neo-2.7B.
• name-technique: pearson, spearman.

For instance,

python run_correlation.py -model1=human -model2=bigrams -method=pearson

Run Regression

python run_regression.py -model=[model-name] -measure=[measure-name] -bin_size=[num-bins] -subject_id=[subject-name] -group=[group-name]

• model-name: bigrams, trigrams, quadgrams, gpt2, gpt2-medium, gpt2-large, EleutherAI/gpt-neo-125M, EleutherAI/gpt-neo-1.3B, EleutherAI/gpt-neo-2.7B.
• measure-name: top1_prediction, surprisal
• num-bins: 1 to 600 (default: 1)
• subject-name: Each subject (e.g., ACB71) or all.
• group-name: content or function

Note: The output will be in correlations folder.

Visualisation

Preproduce Performance Comparison

This code will generate the performance comparison of various language model families (N-grams, GPT-2, GPT-Neo) and human benchmarks in a next-word prediction task, evaluated by per-model accuracy and joint prediction percentages both correct and both incorrect. Output will be the photo/model_comparison/model_performance_plot.jpg file.

python visualization/plot_model_comparison.py

Output:

Preproduce Encoding Neural Response

This code will generate the correlations between regression model results estimated from human cloze probabilities and neural responses for group1 and group2 across time based on measure (e.g., surprisal and top1-prediction). Output will be the photo/measure_comparison/[mode=name]/[measure]_grand_avg.png file.

python visualization/plot_grand_averaged.py -model=[model-name] -group1=[group-name] -group2=[group-name] -measure=[measure-name] -p_value=[p-values] -permutation=[running permutation or not] -clusterp=[p cluster] -n_iter=[num-of-iteration]

• model-name: bigrams, trigrams, quadgrams, gpt2, gpt2-medium, gpt2-large, EleutherAI/gpt-neo-125M, EleutherAI/gpt-neo-1.3B, EleutherAI/gpt-neo-2.7B.
• group-name: function or content
• measure: surprisal or top1_prediction
• p_value: The threshold of p-values. Default is 0.05.
• permutation: bool True or False. Default is True. Conduct cluster-based permutation test or not.
• clusterp: The threshold of cluster-defining p-values. Default is 0.05.
• n_iter: Default is 1000. The times for iteration.

For example,

python visualization/plot_grand_averaged.py -model=gpt2-large -group1=content -group2=function -measure=surprisal -p_value=0.05 -permutation=True -clusterp=0.05 -n_iter=5000

Output:

Preproduce Encoding Model Comparison

This code will generate the comparison of all prediction models based on the measures: surprisal or top-1 prediction.

python visualization/plot_measure_comparison.py -measure=[measure] -subject=[subject-type] -group=[group-name]

• measure: surprisal or top1_prediction
• subject-type: all, representative, n-grams, gpt2, or gpt-neo
• group-name: content or function

For example,

python visualization/plot_measure_comparison.py -measure=surprisal -subject=representative -group=content

Output:

• Encoding model comparison results across all channel for individual participants.

• Subject-wise radar chart

Preproduce Topography

The code generates the topographies that show the Pearson r of encoding prediction correlations for the selected models.

python visualization/plot_topography.py -measure=[measure] -model=[model-name] -group=[lexical-group]

• measure: surprisal or top1_prediction
• model-name: human, bigrams, trigrams, quadgrams, gpt2, gpt2-medium, gpt2-large, EleutherAI/gpt-neo-125M, EleutherAI/gpt-neo-1.3B, EleutherAI/gpt-neo-2.7B.
• lexical-group: content or function

For example,

python visualization/plot_topography.py -measure=surprisal -model=human -group=content

Output:

Contact

• [Anonymous Authors]