whereswally

Where's Wally

We aim to observe the entire observe-orientate-decide-act decision making loop in a simplified task using EEG with the aim of developing EEG based algorithms able to decode the different stages directly from EEG.

To make the idea's concrete we will use the 'where's wally' style visual search task. #![wheres wally]#(#http://www.dafont.com/forum/attach/orig/2/1/214225.jpg)

For each stage of the OODA loop we will use a different sensory modality and EEG analysis technique. The basic ideas are given here:

1. Observe. During this stage the user is looking round the scene aquiring information on what it contains.

While the user is searching through the scene we need to know where they are looking, and what they are looking at. To get this information we will use a combination of an external eye-tracker and EOG electrodes attached around the eyes.

2. Orientate: During this stage the user is integerating the visual information into their world model.

It is unclear what brain processes occur during this stage so we will directly look at it.

3. Decide: During this stage the user decides what to do based on the processed visual input.

In this task, this corresponds to deciding if the current visual input contains the target (wally). In EEG terms this decision making process corresponds to target nonrecognition with the associated p300 ERP response. Identification of the P300 requires an accurate time-reference for when the visual input arrived. As the scene is fixed we will use the fixation time as this reference. Thus we will do a Fixatation Related Potential (FRP) analysis to identify the p300 and target-processing response.

4. Act : During this stage the user generates the action associated with target detection.

In our where's wally style task, the action taken is to move the index finger from the 'home' location to the location of the detected target. To measure this action we will need;

1. EMG electrodes to measure when the user starts using their muscles
2. 3d-finger tracking system to track the fingers movement in 3-d space as it moves from the home-location to the target location.

As this is a physical movement we will attempted to decode the various movement-related signals. As a minimum this will include;

• Movement-planing brain signatures before movement initation - the Readiness-potential and mu/beta ERD/S signatures.
• Movement-control brain signatures -- in particular as the finger will be moving through space we will look for the movement-related-cortical-potential MRCP and will aim to use this to decode the finger trajectory in real-time using [trajectory-decoding](hand movement kinematics decoding EEG) techniques.

Project Plan

The above experimental plan requires;

1. The development of the experiment stimulus software -- to show the stimuli, respond to users actions and annotate the recorded EEG.
2. The integeration of the various hardware components used in this experiment. These include;

• Screen -- for presentation of the stimuli
• Touch-screen (for home/target presses)
• EEG
• EOG
• EMG
• Eye-tracker (for FRP analysis)
• 3d-finger tracker (for the movement trajectories)

Stage 1: Stimuli

This is the development of a simple where's wally style game integerated into a touch-screen capable device. Ideally this will be made using a standard cross-platform game development environment (e.g. unity, GDX) as this eases development in different platforms and hardware choices for the screen/touch-screen.

The stimuli themselves are pretty simple, consisting of;

• a screen containing a target search task,
• a visual indication of the home-location, with associated logic to ensure the user has their finger placed here while visually searching
• a score indicator. Score will be based on correct targets pressed, incorrect target pressed and time-to complete the search task
• game logic -- to update the score, keep track of when all targets have been found, transition between game stages etc.
• inter-search-screens screens to show between the search games.
• annotations -- to send events to the EEG/EMG/EOG system

Stage 2: Hardware integration

This experiment requires 4 different types of measurement hardware, i.e. touch-screen, eeg, eye-tracker, 3d-tracker. These will need to be integrated and synchronized such that later they can be used as a combined data-set for cross-measurement analysis. For the different measurement modalities there are different issues to address:

0. EEG : This is already setup in the lab for recording EEG, EMG and EOG. The stimulus software must be setup to ensure it sends sufficient events for annotation of this data.
1. Touch screen: This simply requires a translator to convert touch events (screen x/y coordinates) into events to be used to syncronize and annotate the EEG data via. buffer_bci event sending in the stimulus game application.
2. Eye-tracker : This already has a 'buffer' interface for recording eye information. What is required is a 'sync-pulse' sender to syncronize the eye-data with that from the EEG system, and to configure the systems to record correctly. Again such a sync-generator is already available in the buffer-BCI. Another concern with the eye-tracker which remains to be addressed however is it's use in combination with the touch-screen -- we need to setup and test this configuration to ensure the eye-tracker + touch-screen combo can be used together.
3. 3d-finger tracker : This is the most unclear part of the experimental setup. We require a system able to track the figure at relatively high resolution near the touch-screen. Within the BCI lab there are a couple of possible options available, either a leap-motion, or a kinect-3d camera or the opto-track tracking system. Of these we should probably start with the leap-motion as a buffer driver already exists and it is smallest and should work well with a touch screen.

Stage 3 : Pilot experiments

When a complete hardware/software system has been developed and tested an initial EEG data-set will be gathered. This should include a minimum of 5 subjects doing 30min-1hr of where's wally tasks. After this data has been gathered an initial data-analysis will be performed to check for the existence of the expected brain signals, namely.

1. Fixation locked analysis - locking at the 1000ms post fixation and splitting into fixatations on 'targets' vs. those on non-targets, this should show a target-dependent p300 response.
2. Movement locked analysis - locking to the EMG identification of the start of the target-signaling movement, we expect to see a RP in the 1000ms before the movement, with associated ERD in the mu/beta ranges in the same time-range
3. Trajectory analysis - cross-correlating between the low-frequency EEG (<3hz) and the 3d-fingertip trajectories we should be able to re-construct the finger-tip trajectories with reasonable accuracy after learning a convolutional filter.