Flip that Bucket
Moving your body at will seems trivially easy. You probably do it all the time. You can knock on a table, stamp with your feet and flap your arms whenever you want to. But what mechanism enables you to initiate these movements? Moreover, how does this mechanism relate to your conscious experience of wanting to move? In this project, we investigate the relation between intending to move and the neural preparation for a voluntary movement. We aim to predict an intention to move before movement onset based on electroencephalogram (EEG) activity. These predictions will be fed back to a person using Brain-Computer Interfacing (BCI) techniques in order to investigate their awareness of intending to move at that moment in time.
Our results will be used to create a scientific demo, targeted at a general audience and high-school children in particular. In this demo, participants get a chance to experience a BCI system hands-on by playing a computer game. In this game, participants try to beat the computer. However, the computer is sneaky and can read the participant’s brain activity. As soon as it detects an intention to move, it moves before them, trying to outsmart the human opponent. This demo serves as a means to educate the general public about the neural mechanisms that underlie our ability to perform voluntary movements. The connection between a participant’s brain signals and their voluntary movements are made explicit to them by means of this game.
We will communicate our scientific findings to the academic community through the publication of an open access academic paper. Furthermore, we will visit public events to communicate our findings to the general public by means of our demo. Both our scientific experiment and public demo will be implemented as extensions to an existing open-source BCI development toolbox, encouraging further development of our project ideas by both academics and the general developer community. This extended toolbox is made available to anyone with access to an EEG system. It can be used at home, during public events or in scientific experiments.
Moreover, our project results will help other scientific research on neuro-prosthetics. Successful prediction of movement intent will provide us with faster and more convenient ways to control prosthetic devices or wheelchairs. Delays in activating an assistive device could be minimized by detecting movement intent early on, which potentially increases the therapeutic benefit by minimizing the time between motor planning in the cortex and the execution of that plan with the assistive device.
When we move, imagine to move or observe movement in other people, specific parts of our brain activate: the premotor, supplementary and primary motor cortex (summarized as ‘motor cortex’). As shown in Figure 1, these motor cortex areas are located in the top and central parts of your brain. When we perform voluntary movements, we typically see a readiness potential (RP) and event-related desynchronization (ERD) across the motor cortex. Interestingly, we typically see these brain signals before a person reports their feeling of wanting to move. This suggests that the brain starts preparing an act before a person experiences their intention to act (Libet et al., 1983). How exactly our conscious intention to move relates to these neural signals remains unclear.
Figure 1. Schematic overview of the supplementary, premotor and primary motor cortex of the human brain. Original image: images.slideplayer.com/16/5200613/slides/slide_2.jpg
The RP is a slow negative cortical potential that is known to build up over the motor cortex as early as 2 seconds prior to movement onset (see Figure 2). It can be divided into two components: the early RP, characterized by a slow negative slope, starting from 2 seconds before voluntary movement onset in the pre-supplementary motor area (pre-SMA) and the late RP, characterized by a steeper negative slope lateralized over the primary motor area which starts around 400 milliseconds before voluntary movement (Kornhuber & Deecke, 1965; Shibasaki & Hallett, 2006).
The ERD that is observed over the motor cortex indicates that the neurons in this part of the brain no longer fire in synchrony, leading to a decrease in neural activity across a specific frequency range (see Figure 2). The frequency range in which this desynchronization is observed is called the alpha (8-12Hz) and beta (13-30Hz) bands. This alpha/beta ERD occurs around 1.5 - 2 seconds prior to movement onset and starts bilaterally over the motor cortex (Pfurtscheller & Aranibar, 1979). The beta ERD has been found to be contralateral prior to dominant hand movement as opposed to bilateral prior to non-dominant hand movement (Bai, Mari, Vorbach, & Hallett, 2005). These neural signals are typically not seen before movement onset when the movement was passive or involuntary (Müller et al., 2003; Shibasaki & Hallett, 2006).
Both the RP and ERD have been used to predict movement onset on a single trial level (Bai et al., 2011; Blankertz et al., 2006; Fried, Mukamel, & Kreiman, 2011; Lew et al., 2012; Schneider et al., 2013). Bai et al. (2011) were amongst the first to reliably detect the intention to move before movement onset based on EEG signals in real-time. They demonstrated that voluntary movement could be predicted on average 620 ± 250 milliseconds before its onset.
Figure 2. This picture is adapted from Bai et al., 2011. Here we see an example of an RP and alpha/beta ERD across the motor cortex for a single participant. The left bottom plot shows the RP at electrode Cz. The top left plot shows the development and location of the RP before movement onset (time 0 secs). The bottom right plot shows the alpha/beta ERD in blue followed by and ERS in red at electrode C3 after movement onset (time 0 secs). The top right plot shows the development and location of the ERD prior to movement onset.
A demo will be developed to educate people the scientific findings on the neural preparatory activity for movement. This demo will consist of a computer game called "Flip-that-Bucket" (see Figure 3). In this game, participants try to beat the computer. The game consists of several rounds during which both the participant and the computer can gather virtual green slime. At the start of each round, an empty bucket is displayed. This bucket will be filled with green slime over time. The participant and computer can flip over the bucket and empty its contents onto their opponent any time they want by pressing the big red button next to them. Whoever flips the bucket first, will get points for the amount of slime they threw over their opponent. Whoever has gathered the biggest amount of slime after a number of rounds, wins the game.
The computer tries to learn the participant’s behaviour and predict their moves. We envision three versions of this game. In each version, the computer tries to predict the participant’s moves by investigating either their:
- Action times
- Muscle activity
- Brain activity
Using these three versions, we aim to educate people about the temporal order of events involved in performing a voluntary movement. The 1st version, where the computer makes predictions based on the history of action times, is expected to have the lowest performance. However, it is expected to work significantly above chance level since humans are known to be bad random generators and are usually predictable to some extent in this type of setting. The 2nd version, where predictions are made based on the ongoing muscle activity (measured using EMG), is expected to work best. These predictions are expected to occur very close or at the same time as movement onset because muscle activation is involved in the last stages of action preparation. The 3rd version, where predictions are made based on brain activity, is expected to perform better than the 1st version (action history) and a little worse than the 2nd version (muscle activation). These predictions are expected to be the earliest of all since brain activity is involved in the first stages of action preparation.
We believe that this demo will serve as an intuitive and appealing means to explain a neuroscience topic to a general public. Participants get a chance to experience real-time feedback on their brain signals through the predictions of the computer. This demo serves as a thought-provoking means to explain how movement intent travels from the brain to the muscles in order to perform a voluntary movement. Participants might try to outsmart the computer. They might try to perform slow or really fast movements to see if it makes a difference. However, they will learn that movement performance has no effect on the brain signals related to movement intent that is detected in the EEG. Moreover, people may get the feeling that their mind is being read since the computer reacts to their intention to move.
Through this demo we can also inform the general public about the techniques of EEG and BCI. We can present and show the parts and steps you need to create a BCI: EEG hardware device, EEG acquisition software, a stimulus application, gathering labelled training data, training a classifier, applying a classifier, sending feedback to a player, etc. Each of these parts can be explained in more or less detail, depending on the target audience.
Figure 3. Schematic illustration of the BCI demo game. In this game, the human scientist (the player) will try to beat their robot opponent. Both the scientist and robot aim to gather as much of the virtual green slime as possible and throw it over their opponent by pressing the big red button next to them. The bucket can flip only one time each round, so the player needs to be careful and act before the robot does. However, the robot is sneaky and tries to predict when the player will act using their ongoing EEG activity. Whoever threw the biggest amount of slime over their opponent at the end of the game, wins the game.
We will verify our BCI demo game in a scientific experiment. During this experiment, we will investigate
- the relationship between the neural preparation for movement (i.e. RP and alpha/beta ERD) and the experienced intention to act: although the current experimental set-up cannot provide concrete evidence for establishing a causal relationship between the neural preparation for movement and the intention to move, the accuracy of the single-trial predictions can provide evidence for a strong correlation. We consider a real-time prediction of movement intent to be correct if (1) the player confirms that they wanted to move at that point in time, (2) when the recorded muscle activity shows signs of movement preparation/performance.
- whether a person is aware (or at what point they can be made aware) of their intention to move when movement preparation is detected in the brain (i.e. when a prediction is made). When participants want to move at the time of the robot moved, we expect to find a clearly visible RP and alpha/beta ERD prior to movement performance. When participants did not want to move at the time that the robot moved, we expect no or a weakly visible RP and alpha/beta ERD.
In order to investigate the relation between the experience of intending to move and the neural preparation for movement, we will conduct an experiment with 50 healthy participants. The experiment will consist of two stages and will take approximately 30-45mins including the set-up of the EEG equipment. Prior to each stage of the experiment, participants will receive identical instructions. We will inform the participants that we are testing a new BCI that attempts to predict the moment at which they intend to move. During the experiment, participants will play the BCI demo game.
During the first stage of the experiment, the robot tries to predict the human player based on their action history. The EEG activity prior to the human movements will be used to train our BCI classifier. During the second stage of the experiment, the computer will make brain-based predictions of the participant’s movement intentions. The brain-based predictions are generated by our BCI system. Because random-triggered movements are replaced by brain-triggered movements, we expect the frequency at which robot predictions are reported as correct to increase in the second part of the experiment.
This setup provides us with a way to compare the ERD and RP signals prior to predictions that were reported as correct, incorrect and unknown. We expect the ERD and RP signatures to be present for correct and not present for incorrect predictions.
Based on previous research, we expect the onset of the RP and alpha/beta ERD around 1.5 to 2 secs pior to movement performance (Libet et al., 1983; Pfurtscheller & Aranibar, 1979; Bai et al., 2011). Up to 2 secs prior to movement onset, participants report to be aware of their intention to move when probed (Verbaarschot et al., 2016). Up to 150 msecs prior to movement, participants are able to provide a self-initiated report on the onset of their intention to move. Based on these results, we can envision to following possible scenario’s:
Figure 4. As output variables in this experiment we will have brain data, muscle activity, and the subjective reports. This figure shows the possible scenarios we envision. The FES triggers (1-5) show the moment a FES stimulation happens in time (msec.) relative to movement onset. The coloured horizontal bars show our 3 output variables and the ranges we envision them to appear relative to movement onset.
Scenario 1: FES trigger is sent early, before RP and ERD are visible:
- We expect there to be no ERD/RP visible.
- No visible muscle activity.
- Movement is reported as unintended.
- Movement initiation is attributed to FES.
- Can occur with random FES triggers and (too early) brain-based FES triggers.
Scenario 2: FES trigger is sent early, before RP and ERD are visible. The trigger is sent within the range after a point the participant could be aware of intention when probed but no ERD/RP onset is present:
- We expect no ERD/RP signal visible.
- No visible muscle activity.
- A person might be aware in this range. There are 3 options for the awareness report:
- person was already aware when FES trigger happened
- person became aware because of the FES trigger
- person is not aware of their intention to act and FES trigger didn’t do anything
- Movement can be reported either as intended or unintended.
- Movement initiation is unclear, most likely attributed to the FES.
- Can occur with random FES triggers and brain-based FES triggers that are early.
Scenario 3: FES trigger is sent after ERD /RP signal is visible.
- We expect ERD/RP onset to be visible in the brain.
- low muscle activity.
- A person might be aware in this range. There are 3 options for the awareness report:
- Person was already aware when FES trigger happened
- Person became aware because of the FES trigger
- Person is not aware of their intention to act and FES trigger didn’t do anything
- Movement can be reported either as intended or unintended.
- We expect to see more intended than unintended reports.
- Movement initiation is unclear.
- Can occur with random FES triggers and brain-based FES triggers that are on time.
Scenario 4: FES trigger is sent after ERD/RP signal is visible. The participant is generally able to report on their awareness
- We expect there to be an ERD/RP visible.
- There will be muscle activity present.
- The movement will be reported as intended.
- Movement initiation is attributed to participant himself.
- Can occur with random FES triggers, brain-based FES triggers and EMG-based FES triggers.
- Possible report: “I wanted to move, but then it moved for me”
Scenario 5: FES trigger is sent after ERD/RP signal is visible, and happens when muscle contraction starts.
- We expect there to be an ERD/RP visible.
- There will be high muscle activity present.
- The movement will be reported as intended.
- Movement initiation is attributed to participant himself.
- Can occur with random FES triggers, brain-based FES triggers and EMG-based FES triggers.
- Possible report: “I wanted to move, but maybe the FES moved to”
Both our scientific experiment and public demo will be implemented as extensions to an existing open-source development toolbox. The toolbox that we will build upon is called Buffer BCI.
This project will result in 4 deliverables:
- A working Brain-Computer Interface application that is designed to detect movement intent in real-time. The software will be added to the Buffer BCI framework by April 1st, 2018.
- We will submit our demo to at least 3 public events. Examples of relevant public events are: the InScience festival in Nijmegen, Brain Awareness Week, the Music Meeting and Lowlands Science.
- An abstract, describing our validated Brain-Computer Interface, will be submitted to a scientific conference. With this abstract submission, we will apply for an oral or poster presentation at the conference. Examples of relevant, international and well-known conferences are: the Society For Neuroscience conference, the Graz Brain-Computer Interface conference and the International Brain-Computer Interface Society meeting.
- An article describing the results of the validation of our Brain-Computer Interface application will be submitted to a scientific journal. The article will be submitted by January 31st, 2019.
The proposed project would originally take 12 months to complete. We started in October 2017 and should have finished by September 30st 2018. However, due to some unfortunate circumstances, the project is delayed and will be finished by January 31st, 2019. A schematic timeline of our project is provided in Table 1. A step-by-step planning is provided below.
Table 1. Schematic project timeline
In this stage we will be developing the demo game as described in Section 2.3. The first design will be using simple graphics only: the balloon will be represented by a circle and there will be no avatar graphics. This first version will be implemented in Matlab. The computer tries to predict the participant’s movements based on their history of action times. Each time a participant could not move due to the computer, the time of the computer action + 2 secs is added to the current history of human action times. This is done in order to approximate the full distribution of human action times (not only the early part). The computer will draw a user-move time (t) from the current empirical distribution of user moves. Subsequently, some interval (for instance 1.5secs) is subtracted from t, in order to schedule the computer move just before the participant move. A behavioural pilot (10 participants) will be performed to confirm that:
- movements are performed roughly between 5 and 10 secs after trial start
- movements are spontaneous: participants are not performing a stimulus-response task
- the game is fun
- the game is competitive: participants are trying to beat the computer
- the computer predictions are perceived as roughly accurate
[DONE] To deliver: a simple working version of the demo game in which the computer predicts the players moves based on their history of action times.
An EEG pilot (5 participants) using the simple demo game will be performed to confirm that there is:
- A clear RP and alpha/beta ERD in move data: from 2 secs prior to movement until movement onset
- No clear RP or alpha/beta ERD in non move data: baseline data (only the fixation cross is visible), pre-movement data (1 secs after the circle is visible until 4 secs prior to movement onset) and post-movement data (2 secs after movement until the end of the trial)
[DONE] To deliver: an EEG data set of the demo game + ERP and ERD plot of the move and non move data of each participant.
Develop a full functioning BCI system:
- Train separate classifiers offline on move vs. non move data to detect the RP, alpha/beta ERD and EMG onset of individual participants.
- Integrate the classifier output to generate a probabilistic estimate of the participant’s movement intention. Given probabilistic models and assuming (conditional) independence of the information sources this can be done with a simple naive-bayes style output combination.
- Implement an adaptive threshold for the RP and ERD classifiers in order to cope with train-test-transfer issues. Ideally this threshold should be optimised to hit a target False-positive-rate (FPR). A simple way to do this is with an adaptive percential-filter (so the threshold is such that, say, 99% of predictions are below the threshold).
- Intermediate testing of the classifier performance on ourselves (i.e. 3 participants).
[DONE] 1st deliverable: A fully functioning version of the game, including all the bits to make the BCI system work, will be uploaded to the buffer-bci framework by April 1st, 2018. In theory, the BCI is able to predict movement intent prior to movement performance. Whether or not this is indeed the case, will be validated during later stages of this project.
Improve the current implementation of the experiment based on previous experience. Make sure that:
- the real-time classification works for 3 pilot participants
- the expectation effect (CNV) in response to the FES stimuli is not biasing the classifier results
- there is sufficient non move data
Decide where we want to demo our game. The demo's will serve to verify our classifier performance on a large population. Decide to which conference we want to submit an abstract.
[DONE]: working implementation of the scientific experiment
[DONE] To deliver: list of 3 target public events. We have chosen: Lowlands Science August 17-19 (proposal submitted, but rejected), Nijmeegse Zomerfeesten (submitted, accepted and done on July 15th) & InScience November 7-11 (proposal submitted and accepted)
[DONE] To deliver: one target conference. We have chosen two conferences: BCI meeting in Asilomar May 21-25 (accepted for poster presentation, done) + ASSC in Krakow (accepted for poster presentation, done)
Test the full experiment with 50 participants.
[DONE] To deliver: EEG dataset of 50 participants.
Analyse the collected data. Analysis is focused on:
- intended/self-initiated movements are preceded with a clear RP and alpha/beta ERD
- unintended/FES-initiated movements are not preceded with a clear RP and alpha/beta ERD
- unclear/unclear movements are potentially preceded with an early RP and alpha/beta ERD
- the rate of intended movements is higher in the second stage of the experiment (testing) compared to the first (training)
[DONE] To deliver: bullet-point style report of the main results of the scientific experiment.
The demo game will be implemented in Unity in order to polish up its appearance. Special attention is paid to the underlying code. The code should be robust, annotated, efficient and fool proof. Include visualizations of the participant’s action distribution and their ERP and ERD signatures. Allow experimenter tunable settings for the EEG based predictions to try and move the relative prediction time.
[DONE] 2nd and 3rd deliverable: January 31st, 2019, the demo game is submitted (and possibly tested at) 3 public events + the scientific research is submitted to a conference + a polished and fool-proof version of the demo game will be uploaded to the buffer-bci framework.
Write up the results in a scientific paper.
[DONE] 4th deliverable: the paper is submitted to a scientific journal by January 31st, 2019.
Djamari Oetringer (student-assistent)
Bai, O., Mari, Z., Vorbach, S., & Hallett, M. (2005). Asymmetric spatiotemporal patterns of event-related desynchronization preceding voluntary sequential finger movements: a high-resolution EEG study. Clinical Neurophysiology, 116(5), 1213–1221. http://doi.org/10.1016/j.clinph.2005.01.006
Bai, O., Rathi, V., Lin, P., Huang, D., Battapady, H., Fei, D. Y., … Hallett, M. (2011). Prediction of human voluntary movement before it occurs. Clinical Neurophysiology, 122(2), 364–372. http://doi.org/10.1016/j.clinph.2010.07.010
Blankertz, B., Dornhege, G., Krauledat, M., Kunzmann, V., Losch, F., Curio, G., & Müller, K.-R. (2006). The Berlin Brain-Computer Interface: Machine learning based detection of user specific brain states. Computer, 12(6), 581–607. Retrieved from http://eprints.pascal-network.org/archive/00002257/
Kornhuber, H. H., & Deecke, L. (1965). Changes in the brain potential in voluntary movements and passive movements in man: Readiness potential and reafferent potentials. Pflugers Archiv Fur Die Gesamte Physiologie Des Menschen Und Der Tiere, 284, 1–17. http://doi.org/10.1007/s00424-016-1852-3
Lew, E., Chavarriaga, R., Silvoni, S., & Millán, J. del R. (2012). Detection of self-paced reaching movement intention from EEG signals. Frontiers in Neuroengineering, 5(July), 13. http://doi.org/10.3389/fneng.2012.00013
Libet, B., Gleason, C. a., Wright, E. W., & Pearl, D. K. (1983). Time of Conscious Intention To Act in Relation To Onset of Cerebral Activity (Readiness-Potential). Brain, 106(3), 623–642. http://doi.org/10.1093/brain/106.3.623
Müller, G. R., Neuper, C., Rupp, R., Keinrath, C., Gerner, H. J., & Pfurtscheller, G. (2003). Event-related beta EEG changes during wrist movements induced by functional electrical stimulation of forearm muscles in man. Neuroscience Letters, 340(2), 143–147. http://doi.org/10.1016/S0304-3940(03)00019-3
Muralidharan, A., Chae, J., & Taylor, D. M. (2011). Extracting attempted hand movements from eegs in people with complete hand paralysis following stroke. Frontiers in Neuroscience, (MAR). http://doi.org/10.3389/fnins.2011.00039
Pfurtscheller, G., & Aranibar, A. (1979). Evaluation of event-related desynchronization (ERD) preceding and following voluntary self-paced movement. Electroencephalography and Clinical Neurophysiology, 46(2), 138–146. http://doi.org/10.1016/0013-4694(79)90063-4
Popović, D. B. (2014). Advances in functional electrical stimulation (FES). Journal of Electromyography and Kinesiology, 24(6), 795–802. http://doi.org/10.1016/j.jelekin.2014.09.008
Schneider, L., Houdayer, E., Bai, O., & Hallett, M. (2013). What We Think before a Voluntary Movement. Journal of Cognitive Neuroscience, 25(6), 822–829. http://doi.org/10.1162/jocn_a_00360
Shibasaki, H. (2012). Cortical activities associated with voluntary movements and involuntary movements. Clinical Neurophysiology. http://doi.org/10.1016/j.clinph.2011.07.042
Shibasaki, H., & Hallett, M. (2006). What is the Bereitschaftspotential? Clinical Neurophysiology. http://doi.org/10.1016/j.clinph.2006.04.025
Soekadar, S. R., Birbaumer, N., Slutzky, M. W., & Cohen, L. G. (2015). Brain–machine interfaces in neurorehabilitation of stroke. Neurobiology of Disease, 83, 172–179. http://doi.org/10.1016/j.nbd.2014.11.025
van Gerven, M., Farquhar, J., Schaefer, R., Vlek, R., Geuze, J., Nijholt, A., Desain, P. (2009). The brain-computer interface cycle. Journal of Neural Engineering, 6(4), 41001. http://doi.org/10.1088/1741-2560/6/4/041001
Verbaarschot, C., Haselager, P., & Farquhar, J. (2016). Detecting traces of consciousness in the process of intending to act. Experimental brain research, 234(7), 1945-1956.
Wolpaw, J. R., & Wolpaw, E. W. (2012). Brain-computer interfaces : principles and practice. Oxford University Press.