Skip to content
Analysis code for the paper "Does the brain represent words?"
Branch: master
Clone or download
Fetching latest commit…
Cannot retrieve the latest commit at this time.
Permalink
Type Name Latest commit message Commit time
Failed to load latest commit information.
data
encodings
scripts
.gitignore
LICENSE
README.md
analysis.ipynb
analyze_sim.ipynb
dependency_graph.py
heatmap.py
learn_decoder.py

README.md

This repository contains analysis code for the paper:

Does the brain represent words? An evaluation of brain decoding studies of language understanding. {Jon Gauthier and Anna Ivanova}. 2018 Conference on Cognitive Computational Neuroscience.

Our study combines imaging data from Pereira et al. (2018) with various pretrained computational models. See the section "Using the encoding models" for information about using these models to encode novel stimuli.

This repository is open-source under the MIT License. If you would like to reuse our code or otherwise extend our work, please cite our paper:

 @inproceedings{gauthier2018does,
   title={Does the brain represent words? An evaluation of brain decoding studies of language understanding.},
   author={Gauthier, Jon and Ivanova, Anna},
   booktitle={Conference on Cognitive Computational Neuroscience},
   year={2018}
 }

Requirements

  • Python 3
  • numpy
  • scikit-learn
  • Jupyter Lab / Notebook
  • matplotlib
  • pandas
  • tqdm

Reproducing our results

This repository contains pre-computed target representations from the seven models tested in the paper. The remaining necessary analysis steps are as follows:

  1. Fetch the public data of Pereira et al. (2018) using the script scripts/fetch_data.sh (requires 25 GB of disk space).

  2. Use scripts/learn_decoders.sh to learn decoders from subject fMRI data to each of the target representations (stored in the encodings directory).

  3. Open analysis.ipynb with Jupyter and re-run all cells, changing references to the output perf.*.csv files if necessary. The notebook renders per-subject performance graphs for each learned decoder, along with a summary graph (Figure 2 of the paper).

  4. Use heatmap.py to build the model regression heatmap, saving a file heatmap.svg (Figure 3) and numerical data heatmap.csv:

    python heatmap.py encodings/384sentences.{baseline,dissent.books8.epoch9,fairseq.wmt14.en-fr.fconv,imdbsentiment,infersent.allnli,order-embeddings,skipthought}.npy
    

Using the encoding models

We forked seven open-source NLP models (listed below) and modified their processing code to simply output the intermediate representations of their inputs. The target encodings were generated by running each of these forked models on the 384 sentences in data/384sentences.txt (available after running fetch_data.sh).

Task Model GitHub link
Language modeling ELMo https://github.com/allenai/allennlp
Language modeling [GloVe][glove] https://github.com/stanfordnlp/GloVe
Language modeling skipthought https://github.com/ryankiros/skip-thoughts
Natural language inference [InferSent][infersent] https://github.com/facebookresearch/infersent
Discourse understanding DisSent (private)
Machine translation [ConvS2S / fairseq][convs2s] https://github.com/facebookresearch/infersent
Image caption retrieval [Order embeddings][order] https://github.com/ivendrov/order-embedding
Sentiment analysis Vanilla LSTM https://github.com/nervanasystems/neon

We plan to make the modified model scripts available in the near future, so that researchers can compute representations for their own test stimuli under the same framework. (If you're interested in a short-term solution, please ask!)

[order]:

You can’t perform that action at this time.