This is a repo evaluating models for predicting fMRI data associated with the Algonauts Project 2021 Challenge (Cichy et al., 2021; Cichy, Roig & Oliva, 2019). In brief, the challenge is to predict the fMRI responses of 10 participants to over 1000 short natural videos.
In this mini-project - which was conducted as part of Neuromatch Academy 2021 - we were interested in testing a biologically inspired model. We compared PredNet (Lotter, Kreiman & Cox, 2016, 2020), a network inspired by the influential neuroscience theory of predictive coding (Millidge, Seth & Buckley, 2021), to VGG19 (Simonyan & Zisserman, 2015), currently one of the most predictive models of the ventral visual stream (Nonaka et al., 2021, Schrimpf et al., 2020).
We used a PredNet pretrained on a self-driving car dataset (see Lotter, Kreiman & Cox, 2016 for details). Here are some example predictions of this model on the Algonauts videos.
We built two simple encoding models (Naselaris et al., 2011; van Gerven, 2017), extracting the layer activations of each network to 900 training videos, reducing their dimensionality with principal component analysis, and linearly regressing the components onto the fMRI responses of 9 visual regions in all subjects. We evaluated the fitted models on a held-out validation set of 100 videos. Here are the results.
Interestingly, we found that VGG19 - which has about 3 orders of magnitude more parameters than PredNet - needed many more layers to match the performance of PredNet, suggesting that the latter is much more efficient.
The code for the two models, PredNet and VGG19, is stored in different
subdirectories (prednet and vgg19) because they have different dependencies.
Functions used for computation with both models are stored in a custom-made
mini-package called algonauts.