NeVRo – Neuro Virtual Reality
Code base of: Decoding subjective emotional arousal from EEG during an immersive Virtual Reality experience Hofmann, Klotzsche, Mariola, Nikulin, Villringer, & Gaebler. bioRxiv, 2020
We used virtual reality (VR) to investigate emotional arousal under naturalistic conditions. 45 subjects experienced virtual roller coaster rides while their neural (EEG) and peripheral physiological (ECG, GSR) responses were recorded. Afterwards, they rated their subject levels of arousal retrospectively on a continuous scale while viewing a recording of their experience.
We tackled the data with three model approaches. The corresponding code can be found in the respective folders. Methods
SPoC ModelSource Power Comodulation (SPoC) decomposes the EEG signal such that it maximizes the covariance between the power-band of the frequency of interest (here alpha, 8-12Hz) and the target variable (ratings).
CSP ModelCommon Spatial Pattern (CSP) algorithm derives a set of spatial filters to project the EEG data onto compontents whose band-power maximally relates to the prevalence of specified classes (here low and high arousal).
This part of the study was published at IEEE VR 2018 in Reutlingen, Germany:
Klotzsche, Mariola, Hofmann, Nikulin, Villringer, & Gaebler. IEEE VR, 2018. [archived code]
LSTM ModelLong Short-Term Memory (LSTM) recurrent neural networks (RNNs) were trained on alpha-frequency components of the recorded EEG signal to predict subjective reports of arousal (ratings) in a binary (low and high arousal) and a continuous prediction task. The fed EEG components were generated via Spatio Spectral Decomposition (SSD) or SPoC. The SSD emphasizes the frequency of interest (here alpha) while attenuating the adjacent frequency bins. Performances of SPoC-trained models served as benchmark-proxies for models that were trained only on neural alpha information.
Furthermore, we tested whether peripheral physiological responses, here the cardiac information (ECG), increases the performance of the model, and therefore encodes additional information about the subjective experience of arousal.
This part of the study was published at IEEE AIVR 2018 in Taichung, Taiwan:
Hofmann, Klotzsche, Mariola, Nikulin, Villringer, & Gaebler. IEEE AIVR, 2018
2021: additional linear model, new cross-validation regime, and further sub-analyses. Encouraged by valuable feedback via this peer-review.
2018-2020: preprocessing for models (SPoC, CSP, LSTM) was harmonized, and their evaluation and metrics were adapted accordingly. Plus, a more detailed documentation is available. Code for extended & harmonized version of the study (see bioRxiv preprint).
2017-2018: Code of two above mentioned IEEE conference publications.
CollaboratorsSimon M. Hofmann