whereswally

Where's Wally

We aim to observe the entire observe-orientate-decide-act decision making loop in a simplified task using EEG with the aim of developing EEG based algorithms able to decode the different stages directly from EEG.

To make the idea's concrete we will use the 'where's wally' style visual search task.

For each stage of the OODA loop we will use a different sensory modality and EEG analysis technique. The basic ideas are given here:

1. Observe. During this stage the user is looking round the scene aquiring information on what it contains.

While the user is searching through the scene we need to know where they are looking, and what they are looking at. To get this information we will use a combination of an external eye-tracker and EOG electrodes attached around the eyes.

2. Orientate: During this stage the user is integerating the visual information into their world model.

It is unclear what brain processes occur during this stage so we will directly look at it.

3. Decide: During this stage the user decides what to do based on the processed visual input.

In this task, this corresponds to deciding if the current visual input contains the target (wally). In EEG terms this decision making process corresponds to target nonrecognition with the associated p300 ERP response. Identification of the P300 requires an accurate time-reference for when the visual input arrived. As the scene is fixed we will use the fixation time as this reference. Thus we will do a Fixatation Related Potential (FRP) analysis to identify the p300 and target-processing response.

4. Act : During this stage the user generates the action associated with target detection.

In our where's wally style task, the action taken is to move the index finger from the 'home' location to the location of the detected target. To measure this action we will need;

1. EMG electrodes to measure when the user starts using their muscles
2. 3d-finger tracking system to track the fingers movement in 3-d space as it moves from the home-location to the target location.

As this is a physical movement we will attempted to decode the various movement-related signals. As a minimum this will include;

• Movement-planing brain signatures before movement initation - the Readiness-potential and mu/beta ERD/S signatures.
• Movement-control brain signatures -- in particular as the finger will be moving through space we will look for the movement-related-cortical-potential MRCP and will aim to use this to decode the finger trajectory in real-time using [trajectory-decoding](hand movement kinematics decoding EEG) techniques.

Project Plan

The above experimental plan requires;

1. The development of the experiment stimulus software -- to show the stimuli, respond to users actions and annotate the recorded EEG.
2. The integeration of the various hardware components used in this experiment. These include;

• Screen -- for presentation of the stimuli
• Touch-screen (for home/target presses)
• EEG
• EOG
• EMG
• Eye-tracker (for FRP analysis)
• 3d-finger tracker (for the movement trajectories)

Stage 1: Stimuli

This is the development of a simple where's wally style game integerated into a touch-screen capable device. Ideally this will be made using a standard cross-platform game development environment (e.g. unity, GDX) as this eases development in different platforms and hardware choices for the screen/touch-screen.

The stimuli themselves are pretty simple, consisting of;

• a screen containing a target search task,
• a visual indication of the home-location, with associated logic to ensure the user has their finger placed here while visually searching
• a score indicator. Score will be based on correct targets pressed, incorrect target pressed and time-to complete the search task
• game logic -- to update the score, keep track of when all targets have been found, transition between game stages etc.
• inter-search-screens screens to show between the search games.
• annotations -- to send events to the EEG/EMG/EOG system