Statement of Need
When faced with a decision, an organism uses information gathered by its senses in order to determine the best course of action. Vision is one of the primary senses, and tracking eye gaze can offer insight into the cues that affect decision-making behavior. Thus, to study decision-making and other cognitive processes, it is fundamentally necessary to track eye position accurately. However, commercial eye trackers are 1) often very expensive, and 2) incorporate proprietary software to detect the movement of the eye. Closed source solutions limit the researcher’s ability to be fully informed regarding the algorithms used to track the eye and to incorporate modifications tailored to their needs. Here, we present our software solution, openEyeTrack, a low-cost, high-speed, low-latency, open-source video-based eye tracker. Video-based eye trackers can perform nearly as well as classical scleral search coil methods and are suitable for most applications.
Software and Hardware components
openEyeTrack is a video-based eye-tracker that takes advantage of OpenCV, a low-cost, high-speed infrared camera and GigE-V APIs for Linux provided by Teledyne DALSA, the graphical user interface toolkit QT5 and cvui, the OpenCV based GUI. All of the software components are freely available. The only costs are from the hardware components such as the camera (Genie Nano M640 NIR, Teledyne DALSA, ~$450, ~730 frames per second) and infrared light source, an articulated arm to position the camera (Manfrotto: $130), a computer with one or more gigabit network interface cards, and a power over ethernet switch to power and receive data from the camera.
By using the GigE-V Framework to capture the frames from the DALSA camera and the OpenCV simple blob detector, openEyeTrack can accurately estimate the position and area of the pupil. We include pupil size calculations because of its putative link to arousal levels and emotions of the subject.
Tested Hardware Specs
- Computer: Intel(R) Core(TM) i7-5820K CPU @ 3.30GHz, 64GB
- Basic IR light source
- We use this one from Amazon IR illuminator
- Camera: Teledyne DALSA Genie Nano M640 NIR
- Netgear Power-Over-Ethernet Switch
Update 9/1/2020 - The camera firmware that was used can be found here. You will need the windows saperea LT SDK to load it onto the camera and then set the camera to autoboot into this configuration.
Update - We have now tested the openEyeTrack implementation on an INTEL NUC i7 as well as a Zotac EN51050. The eye tracker can perform with these configurations as well. The gev_nettweak tool needs to be run to get consistent performance. The network card is otherwise overloaded and frames are lost. A 725 Hz camera at 672x512 is transmitting ~249 megabytes per second. The net tweak tool changes the size of the buffers allowing jumbo frames to be transmitted over the network.
Update 10/07/2019 : We have tested openEyeTrack on clean installs of 18.04, and 19.04. We do not support Windows at the moment. If possible, we recommend using a clean system for openEyeTrack.
Update 10/07/2019 : The exact Qt version is not an issue. We find the default Qt installed with qt5-default in the install script works perfectly fine with openEyeTrack.
- Install git by typing: $ sudo apt-get install git-core
- Clone repository by typing: $ git clone https://github.com/chand-lab/openEyeTrack.git
- After installing the GigE-V Framework, you can either manually set up Qt and OpenCV or run the installation script denoted as softwareInstall within the repository
- If you choose to run the installation script you may need to first convert it to an executable by typing: $ chmod +x softwareInstall.sh
- Occassionally, you may be prompted to press "y" to continue the installation process
- Please explicitly run the last two commands at the bottom of this file concerning linking of libraries and make sure Ubuntu actually identifies where your libraries are.
Installing GigE-V Framework
- Go to Teledyne dalsa website and download the GigE-V Framework for Linux under support → downloads → SDKs
- Install all prerequisites listed in User Manual (GNU make can be installed through build-essential and libX11-dev should be libx11-dev)
- Unpack the .tar file in downloads and then go inside the newly created directory and copy the .tar file that looks like GigE-V-Framework_x86_2.02.0.0132.tar.gz to $HOME
- Cd into home and unpack tar file (use tar -zxf)
- Cd into DALSA and type $ ./corinstall
- Go to connection settings and edit the ethernet connection that is associated with the camera
- Go to the IPv4 settings and change the method to “Link-Local Only” and save changes
- Make sure the connection is enabled and verify you can connect to camera by going to terminal and typing:$ GigeDeviceStatus
- Try running the genicam_cpp_demo found under DALSA/GigeV/examples
- Running the demo
- run the gev_nettweak tool (found under GigeV/bin) to enhance performance and prevent resetting the image transfer. This tool needs to be re-run if the system restarts
Installing Qt 5.5.1
sudo apt-get install build-essential sudo apt-get install qt5-default
Installing opencv (version 4.1.0)
Navigate to the directory where you want to clone the repository Type:
sudo apt -y update sudo apt -y upgrade sudo apt -y remove x264 libx264-dev
sudo apt -y install build-essential checkinstall cmake pkg-config yasm sudo apt -y install git gfortran sudo apt -y install libjpeg8-dev libjasper-dev libpng12-dev sudo apt -y install libtiff5-dev sudo apt -y install libtiff-dev sudo apt -y install libavcodec-dev libavformat-dev libswscale-dev libdc1394-22-dev sudo apt -y install libxine2-dev libv4l-dev sudo apt -y install libgstreamer0.10-dev libgstreamer-plugins-base0.10-dev sudo apt -y install libgtk2.0-dev libtbb-dev qt5-default sudo apt -y install libatlas-base-dev sudo apt -y install libfaac-dev libmp3lame-dev libtheora-dev sudo apt -y install libvorbis-dev libxvidcore-dev sudo apt -y install libopencore-amrnb-dev libopencore-amrwb-dev sudo apt -y install libavresample-dev sudo apt -y install x264 v4l-utils
sudo apt -y install libprotobuf-dev protobuf-compiler sudo apt -y install libgoogle-glog-dev libgflags-dev sudo apt -y install libgphoto2-dev libeigen3-dev libhdf5-dev doxygen
Clone opencv and opencv_contrib
git clone https://github.com/opencv/opencv.git cd opencv git checkout 4.1.0 cd .. git clone https://github.com/opencv/opencv_contrib.git cd opencv_contrib git checkout 4.1.0 cd ..
(Need to clone the same version of opencv_contrib)
Make a build directory within opencv and start compilation process
cd opencv mkdir build cd build
cmake -D CMAKE_BUILD_TYPE=RELEASE -D CMAKE_INSTALL_PREFIX=/usr/local -D WITH_TBB=ON -D WITH_V4L=ON -D WITH_QT=ON -D WITH_OPENGL=ON -D OPENCV_EXTRA_MODULES_PATH=../../opencv_contrib/modules -D BUILD_EXAMPLES=OFF -D OPENCV_GENERATE_PKGCONFIG=ON ..
(For later versions of OpenCV, need to include OPENCV_GENERATE_PKGCONFIG=ON)
sudo make install -j4
(having the ‘-j4’ says to use use multiple threads which greatly speeds up this step)
sudo sh -c 'echo "/usr/local/lib" >> /etc/ld.so.conf.d/opencv.conf' sudo ldconfig
For detailed usage examples, see USAGE.md found under the docs folder
Usage notes for Genie Nano Cameras
Apparently the default specs are set to be 30 Hz which would be suboptimal. In the firmware folder, we provide a firmware that enables high frame rate eye tracking.