Neuropolis: Building an artificial intelligence system for human brain activity prediction (EEG, fMRI)
Version: 10th March 2024
Project developed by Maël Donoso, Ph.D. in Cognitive and Computational Neuroscience. Affiliations: Ouroboros Neurotechnologies (https://ouroboros-neurotechnologies.com/), Institut Lémanique du Cerveau (https://institut-cerveau.ch/), Policlinique Ostéopathique de Lausanne (https://policlinique-osteopathique-lausanne.ch/).
This project uses data science (NumPy, Pandas, Matplotlib), machine learning (Scikit-Learn), and deep learning (TensorFlow, Keras) tools on brain data obtained with Electroencephalography (EEG) and functional Magnetic Resonance Imaging (fMRI). Our objective is to build an artificial intelligence system for human brain activity prediction, using EEG data to predict fMRI data, whether across experimental conditions or across subjects. For this regression task, we use a variety of machine learning and deep learning models, including linear regression, k-nearest neighbors, decision trees, random forests, support vector machines, fully connected neural networks, convolutional neural networks, recurrent neural networks, and transformers. Our results show that using EEG data to predict fMRI data is possible to a certain extent. Nevertheless, for our voxels of interest, the regression task seems too challenging, at least for relatively simple machine learning and deep learning models, even if the deep learning models perform well on a classification task. The methods developed in this project could provide useful insights for advancing toward a multimodal foundation model for neuroscience, and could help to improve promising brain technologies, such as neurofeedback systems and brain-computer interfaces.
Electroencephalography (EEG) and functional Magnetic Resonance Imaging (fMRI) are two of the major tools used by neuroscientists for human brain activity investigation. While EEG measures the electrical activity of the brain, using electrodes placed on the head, fMRI, on the other hand, measures the activity of the brain indirectly, by detecting the changes in the cerebral blood flow. EEG is typically used to record activity of cortical areas, while fMRI can detect activity in subcortical regions as well. EEG has a good temporal resolution, while fMRI has a good spatial resolution. In EEG-fMRI research works, both techniques are used simultaneously, allowing the researchers to obtain two complementary datasets, acquired at the same time on the same subjects. These studies are often particularly interesting, since they provide two different perspectives on human brain activity for the task of interest.
Most of the time, neuroimaging experiments reported in the scientific literature use only one neuroimaging technique, e.g., EEG or fMRI. Multimodal experiments such as EEG-fMRI research works are rarer, but there is a growing interest for these approaches, and some EEG-fMRI datasets are already accessible in open data repositories. Now, while the human brain can be explored through many different and complementary angles, all neuroimaging techniques, whether they measure the electrical activity on the surface of the head, or the changes in the cerebral blood flow, are ultimately grounded in the same neurophysiological reality. Therefore, a natural research direction would be to explore the degree of redundancy, or shared variance, between two types of neuroimaging data acquired at the same time on the same subjects. Specifically, in this project, we try to train machine learning models, including deep learning models, to make predictions about the human brain activity in a particular EEG-fMRI dataset.
If our general objective is to build an artificial intelligence (AI) system for human brain activity prediction, the most interesting problem, but also the most challenging, is certainly to use EEG data to predict fMRI data. Of course, solving the reverse problem, i.e., to use fMRI data to predict EEG data, could also be attempted. Nevertheless, while fMRI can be used to investigate very specific cognitive processes, this neuroimaging technique is particularly complex and expensive, and requires the subject to remain immobile inside the scanner. By contrast, EEG is a much simpler and more affordable neuroimaging technique, and is better adapted to real-life measurements in a diversity of situations. Hypothetically, if we could achieve near-fMRI precision using only EEG devices, we might be able to obtain the better of both worlds, and this could open entirely new horizons for neuroscience and neurotechnology. Specifically, our focus should be on the fluctuations of the fMRI signal during an experimental condition, since these fluctuations, rather than the absolute value of the signal, can provide important insights into a multiplicity of cognitive processes.
In this project, our specific objective is to predict the fMRI data of the peak voxels described in the following research work, conducted at École Normale Supérieure and other institutions: Donoso, M., Collins, A.G.E., Koechlin, E. (2014). Foundations of human reasoning in the prefrontal cortex. Science, 344, 1481, https://doi.org/10.1126/science.1252254. In this article, which I authored together with Anne G. E. Collins and Étienne Koechlin, we proposed a brain system that could serve as a foundation for human reasoning, and explain our ability to select behavioral strategies in uncertain, changing, and open-ended environments. The components of this brain system were determined with a model-based fMRI approach, and are associated with very specific cognitive processes, such as the evaluation of strategies, and the creation, rejection, and confirmation of strategies. If we were able to predict the brain activity in these peak voxels using only EEG devices, we might be able to considerably expand our understanding of core executive functions of the human brain, such as decision-making, learning, reasoning, planning, and strategic thinking, in a diversity of situations outside the fMRI scanner.
One situation where the human brain might engage in decision-making, learning, reasoning, planning, and strategic thinking, is during a neurofeedback (NF) session. NF consists in providing real-time information to a subject about his/her own brain activity, and asking this subject to try to adapt his/her behavior according to this measure. The objective is to reach a certain brain state, typically by increasing the power of a specific frequency band, in EEG NF, or the activity in a specific brain region, in fMRI NF. Therefore, NF is basically a brain training technology, allowing the subject to learn self-regulation. The brain activity can be measured through a variety of techniques, but EEG and fMRI are the two main choices, and while most NF training protocols rely on a single technique, recent research works have shown the potential of EEG-fMRI NF protocols combining both approaches. Furthermore, it should also be noted that NF is one of the technologies that might benefit the most from an AI system for human brain activity prediction. Indeed, if we could use EEG data to predict fMRI data, we might be able to expand the diversity of feedbacks provided to the subjects, while retaining the simplicity and affordability of EEG devices.
In this project, we use an EEG-fMRI NF dataset downloaded from OpenNeuro (https://openneuro.org/), an open data repository for neuroimaging data, where researchers can publicly store and share brain files obtained from several neuroimaging techniques, including EEG and fMRI. All brain files are stored in the Brain Imaging Data Structure (BIDS) format, a standard format for neuroimaging and behavioral data (https://bids.neuroimaging.io/). Furthermore, all newly published datasets in OpenNeuro, including the one we selected, are released under CC0 license (https://openneuro.org/faq), which means a public domain dedication and no copyright (https://creativecommons.org/publicdomain/zero/1.0/).
Specifically, we use the following dataset: https://openneuro.org/datasets/ds002336/versions/2.0.2. This dataset comes from the following research work, conducted at Inria and other institutions: Lioi, G., Cury, C., Perronnet, L. et al. (2020). Simultaneous EEG-fMRI during a neurofeedback task, a brain imaging dataset for multimodal data integration. Scientific Data, 7, 173, https://doi.org/10.1038/s41597-020-0498-3. This dataset is the first open access bimodal NF dataset integrating EEG and fMRI, and has several interesting characteristics, notably: 1) It contains EEG data, fMRI data, and NF scores for all subjects. 2) The article has detailed methodological sections. 3) EEG data has already been preprocessed by the authors. 4) At the time of writing, it is a relatively recent research work, published in June 2020. 5) The article presenting this research work is publicly available, and can be found here: https://www.nature.com/articles/s41597-020-0498-3. It should be noted that this dataset contains only the data from the first experiment (XP1) reported in the article. The data from the second experiment (XP2) is also accessible on OpenNeuro, but in this project, we focus on XP1.
The scientific detail of this research work is outside the scope of this project, but in summary, the authors conducted a NF experiment where 10 subjects were instructed to use EEG NF scores and fMRI NF scores to perform as well as possible in a motor imagery task, i.e., they needed to execute mentally a movement without any muscle activation. The experiment included six conditions, with an alternance of rest (20 seconds) and task (20 seconds) blocks inside the conditions. In this project, we only use three conditions: the eegfmriNF condition, corresponding to bimodal EEG-fMRI NF, the eegNF condition, corresponding to unimodal EEG NF, and the fmriNF condition, corresponding to unimodal fMRI NF. The three NF conditions were passed in random order by the different subjects. For each condition, the dataset includes the raw fMRI data, the raw EEG data, and the EEG data preprocessed by the authors. It should be noted that EEG acquisition during an fMRI experiment is technically complex, and results in high noise. Therefore, it is extremely useful that the authors included the preprocessed EEG data in the dataset. However, since the fMRI data preprocessed by the authors is not included, we need to perform some fMRI preprocessing steps by ourselves.
Before running the Notebooks, the user should download the dataset by following the instructions on the OpenNeuro website: https://openneuro.org/datasets/ds002336/versions/2.0.2. Then, the fMRI data needs to be preprocessed with fMRIPrep 23.2.0, a robust preprocessing pipeline which automatically performs a series of fMRI preprocessing steps: https://fmriprep.org/en/23.2.0/index.html. Specifically, the fMRI data needs to be normalized using the MNI152Lin output space, while ignoring slice timing correction. We choose the MNI152Lin output space, which is not the standard space of fMRIPrep, in order to be consistent with the Science 2014 article. Indeed, our voxels of interest were defined using the SPM8 software package, where MNI152Lin is the standard space. If the user chooses, as is recommended, the Docker execution with the fmriprep-docker wrapper (https://fmriprep.org/en/23.2.0/installation.html), running fMRIPrep with these parameters can be done with a command such as this one:
fmriprep-docker ds002336-download neuropolis_fmriprep_data participant --participant-label sub-xp101 --fs-license-file freesurfer/license.txt --ignore slicetiming --output-spaces MNI152Lin
This command should be repeated for subjects sub-xp101 to sub-xp110. Depending on the machine, the execution can take several hours per subject to complete. Before running fMRIPrep, the .bidsignore file given in the project directory (neuropolis/.bidsignore) should be added to the dataset directory, in order to ensure BIDS compliance. Also, a FreeSurfer license should be requested here: https://surfer.nmr.mgh.harvard.edu/registration.html. In the command above, this license is located in a freesurfer directory (freesurfer/license.txt). Preprocessing the fMRI data with fMRIPrep is a long step, but one that should absolutely be done before running the Notebooks. If necessary, a complete fMRIPrep tutorial can be found here: https://reproducibility.stanford.edu/fmriprep-tutorial-running-the-docker-image/.
In the first Notebook, we preprocess the fMRI data with the NiBabel and Nilearn libraries. We remove a systematic bias of the BOLD (Blood Oxygen Level Dependent) signal, which tends to increase during an fMRI session. We also normalize the BOLD signal for the target voxels, and replace the outliers (STD > 3) by the value of the previous scan. This normalization is applied independently to all fMRI data, whether this data will eventually be used as a train set, a validation set, or a test set. Indeed, our objective is not to predict the absolute value of the BOLD signal, but rather its fluctuations during an experimental condition, and normalization is a simple way to pursue this objective.
In the second Notebook, we preprocess the EEG data with the MNE and YASA libraries. We compute the Power Spectral Density (PSD) for frequencies between 1 and 40 Hz, using both the Welch method and the multitaper method, and the bandpowers for a series of frequency bands of interest: delta (1-4 Hz), theta (4-8 Hz), alpha (8-12 Hz), sigma (12-16 Hz), beta (16-30 Hz), gamma (30-40 Hz). These frequency bands are widely used in EEG research works, even if their precise definition can vary. We normalize the bandpowers features, and replace the outliers (STD > 4) by the values of the previous scan, while also keeping the bandpowers without normalization as an alternative features set.
In the third Notebook, we perform some preliminary exploration by using linear regression, Ridge regression, and Lasso regression, and trying to minimize the regression Mean Squared Error (MSE). Since fMRI data and EEG data have different dynamics, we explore different sequence lengths, i.e., we use a variable number of EEG time points to predict a single subsequent fMRI time point. Eventually, we select a sequence length of 5 scans, corresponding to 10 seconds, a duration that encompasses the peak of the Hemodynamic Response Function (HRF). In other words, we may expect that any event eliciting a peak in the BOLD signal would have happened in the 10 seconds preceding this peak. Therefore, by choosing a sequence length of 5 scans, we improve our chances of capturing, in the EEG data, a trace of the events that influenced the fMRI data. We also compare the PSD using the Welch method, the PSD using the multitaper method, the bandpowers, and the normalized bandpowers.
In the fourth Notebook, we train a series of machine learning models, including linear regression, k-nearest neighbors, decision trees, random forests, and support vector machines, using the normalized bandpowers and the selected sequence length of 5 scans. Our objective is to minimize the regression Mean Absolute Error (MAE), and when relevant, we fine-tune the models parameters using grid search cross-validation. In the Predictor Brains strategy, we use the eegfmriNF condition as the train set, and the eegNF and fmriNF conditions as the test set, individually for each subject. In the Predictor Groups strategy, we use all the conditions of nine subjects as the train set, and all the conditions of the remaining tenth subject as the test set, and we repeat this process ten times in order to make predictions for all subjects.
In the fifth, sixth, seventh, and eighth Notebooks, we train a series of deep learning models, including fully connected neural networks, convolutional neural networks, recurrent neural networks, and transformers. We use the normalized bandpowers for all models except the convolutional neural networks, for which we rely directly on the EEG data, and we use the selected sequence length of 5 scans for all models except the recurrent neural networks and the transformers, for which we explore different sequence lengths. In the classification task, our objective is to predict the block type, task or rest, using the Stochastic Gradient Descent (SGD) optimizer and the categorical cross-entropy loss. In the regression task, our objective is to minimize the regression MAE, using the Adam optimizer and the MSE loss. In the Predictor Brains strategy, we use the eegfmriNF condition as the train set, the eegNF condition as the validation set, and the fmriNF condition as the test set, individually for each subject. In the Predictor Groups strategy, we use all the conditions of eight subjects as the train and validation set, and all the conditions of the remaining ninth subject as the test set, and we repeat this process nine times in order to make predictions for the nine subjects. One subject has a missing eegNF condition and is not included in these Notebooks.
In the ninth Notebook, we train linear regression and k-nearest neighbors models on the whole brain. We preprocess the fMRI data of each voxel as in the first Notebook. Our objective is to maximize the R^2 metrics. We limit ourselves to the Predictor Brains strategy, i.e., we use the eegfmriNF condition as the train set, and the eegNF and fmriNF conditions as the test set, individually for each subject.
No machine learning model, including linear regression, k-nearest neighbors, decision trees, random forests, and support vector machines, beats the baseline for our target voxels of interest, whether for the Predictor Brains strategy or for the Predictor Groups strategy. Nevertheless, it is important to note that our voxels of interest may not be the easiest targets for a prediction based on EEG data. Indeed, some of these voxels are located in subcortical regions, or in cortical regions where potentials are not easily accessible to EEG electrodes. Furthermore, several of these voxels are primarily associated with high-level, integrated cognitive processes, and not with low-level, sensory mechanisms that might be more easily reflected in EEG activity. When we train machine learning models on the whole brain, these limitations are attenuated, and for some voxels, k-nearest neighbors models beat the baseline. These results show that using EEG data to predict fMRI data is possible to a certain extent, but depends strongly on the target voxels of interest.
No deep learning model, including fully connected neural networks, convolutional neural networks, recurrent neural networks, and transformers, beats the baseline for the regression task on our target voxels of interest, whether for the Predictor Brains strategy or for the Predictor Groups strategy. Nevertheless, all deep learning models beat the baseline for the classification task. Since the models architecture stays essentially the same between the regression task and the classification task, and since these models are trained on the same EEG data, the inability to predict the fMRI data of our target voxels of interest should reflect primarily the complexity of the challenge, and not a general problem with our deep learning models.
Using EEG data to predict fMRI data may be more or less challenging, depending on the context, the objective, and the precise definition of the prediction task. For sleeping subjects, making predictions may be easier, since the different sleep stages are associated both with distinctive EEG patterns and distinctive fMRI patterns. By contrast, for awake subjects, and particularly for subjects performing a cognitive task such as neurofeedback, making predictions may become particularly difficult, since the brain patterns are more complex and unpredictable.
Furthermore, using EEG data to predict fMRI data may be easier when the prediction task is performed within a single subject and a single experimental condition, since the machine learning and deep learning models can be trained, validated, and tested on the same time series. By contrast, in this project, we try to generalize the predictions across several experimental conditions, and even across several subjects. Finally, predicting the absolute value of the BOLD signal may be easier, while our objective is rather to predict the fluctuations of the BOLD signal during an experimental condition.
Overall, in this project, we try to build an AI system for human brain activity prediction in one of the most difficult situations. Moreover, as we mentioned earlier, our voxels of interest may not be the easiest targets for this prediction task, making the problem even more challenging. Still, when we consider other voxels, using EEG data to predict fMRI data becomes possible to a certain extent, suggesting that this endeavor is not entirely impossible. Starting from there, how could we improve Neuropolis, and lay the foundations for a working AI system for human brain activity prediction? We can think of several possibilities for improvement.
First, given enough time, memory, and computing power, we could explore further, and run Neuropolis on new datasets, with new fMRI preprocessings, EEG preprocessings, models architectures and hyperparameters, and prediction strategies. We could also try to include in the training data some characteristics of the subjects, and some elements of the experimental context in which the data was acquired. Of course, we would certainly benefit from larger datasets, and ideally more balanced in terms of sex and age, if they become available in the future.
Second, we might need to develop methods for integrating EEG-fMRI datasets with different characteristics. Indeed, at a certain point, an AI system for human brain activity prediction would certainly require a huge amount of training data, and a single dataset is unlikely to be sufficient. Nevertheless, this integration could pose a series of complex challenges, since neuroimaging datasets can use different fMRI voxel sizes, different EEG montages, different scan durations, and have many other notable differences. It might be necessary to develop new approaches, and maybe to build new multimodal templates, in order to achieve this integration.
Third, instead of relying on simple and general machine learning and deep learning models, we might need to develop more complex and specific models, based on our knowledge of the human brain. For example, it might be necessary to take into account the brain anatomical and functional connectivity, the dynamics of the fMRI BOLD signal, and the sources of the EEG signal. At this point, we might even consider expanding the scope of the project, and trying to integrate other neuroimaging modalities, such as Magnetoencephalography (MEG). This research direction might imply a multidisciplinary scientific collaboration, and the creation of a multimodal foundation model for neuroscience, integrating EEG, fMRI, and other neuroimaging modalities.
Building an AI system for human brain activity prediction should most probably be considered as a long-term endeavor. The methods developed in this project could provide useful insights, but they should only be considered as a first step. Nevertheless, it is difficult to overstate the importance of this research direction, and the impact that a working version of Neuropolis could have for neuroscience and neurotechnology. While fMRI is a complex and expensive neuroimaging technique, EEG is much simpler and more affordable, and better adapted to a diversity of real-life situations. An updated version of Neuropolis could accelerate scientific research in neuroscience, connect the scientific results from several neuroimaging realms, and improve promising brain technologies, such as neurofeedback systems and brain-computer interfaces.
Working toward the creation of a multimodal foundation model for neuroscience, integrating EEG, fMRI, and other neuroimaging modalities, seems to be a natural horizon for scientific research on the human brain. If such a foundation model can be built, we speculate that it could become an important tool for neuroscience laboratories, neurotechnology enterprises, and even AI laboratories willing to replicate more closely the processes of the human brain. From an AI-powered neuroscience platform to a tool for neuroscience-inspired AI, the possibilities of an AI system for human brain activity prediction seem endless, and would naturally require careful reflection and evaluation, considering the sensitive nature of human brain data. Hopefully, the Neuropolis project will contribute to this objective.
This project consists of nine Notebooks, to be run in order:
- fMRI Preprocessing
- EEG Preprocessing
- Linear Regression
- KNN, Decision Trees, Random Forests, SVM
- Neural Networks
- Convolutional Neural Networks
- Recurrent Neural Networks
- Transformers
- Human Brain Predictability Map
The Conda environment for this project can be recreated with the .yml file given in the project directory (neuropolis/neuropolis.yml).