experimentator: Python experiment builder

Documentation contents

Do you write code to run experiments? If so, you've probably had the experience of sitting down to code an experiment but getting side-tracked by all the logistics: crossing your independent variables to form conditions, repeating your conditions, randomization, storing intermediate data, etc. It's frustrating to put all that effort in before even getting to what's really unique about your experiment. Worse, it encourages bad coding practices like copy-pasting boilerplate from someone else's experiment code without understanding it.

The purpose of experimentator is to handle all the boring logistics of running experiments and allow you to get straight to what really interests you, whatever that may be. This package was originally intended for behavioral experiments in which human participants are interacting with a graphical interface, but there is nothing domain-specific about it--it should be useful for anyone running experiments with a computer. You might say that experimentator is a library for 'repeatedly calling a function while systematically varying its inputs and saving the data' (although that doesn't do it full justice).

Not handled here

• graphics
• timing
• hardware interfacing
• statistics
• data processing

The philosophy of experimentator is to do one thing and do it well. It is meant to be used with other libraries that handle the above functionality, and gives you the freedom to choose which you prefer. It is best suited for someone with programming experience and some knowledge of the Python ecosystem, who would rather choose the best tool for each aspect of a project than use an all-in-one package.

Of course, there are alternatives that offer experimental design features along with other capabilities. A selection, as well as recommended complimentary packages, are listed later in the documentation.

An example

To demonstrate, let's create a simple perceptual experiment. For the sake of example, imagine we will present some stimulus to either the left or right side of the screen for a specified amount of time, and ask the participant to identify it. We'll use a factorial 2 (side) x 3 (display time) design, and have a total of 60 trials per participant (10 per condition). Here's how it might look in experimentator:

import random
from time import time
from experimentator import Experiment, order


def present_stimulus_and_get_response(stimulus, side, duration):
    # Use your imagination...
    return random.choice(['yes', 'no'])


def run_trial(experiment, trial):
    stimulus, answer = random.choice(
        list(experiment.experiment_data['stimuli'].items()))
    start_time = time()
    response = present_stimulus_and_get_response(trial.data['side'], trial.data['display_time'])
    result = {
        'reaction_time': time() - start_time,
        'correct': response == answer
    }
    return result


if __name__ == '__main__':
    independent_variables = {
        'side': ['left', 'right'],
        'display_time': [0.1, 0.55, 1],
    }
    stimuli_and_answers = {
        'cat.jpg': 'yes',
        'dog.jpg': 'no',
    }

    experiment = Experiment.within_subjects(independent_variables,
                                            n_participants=20,
                                            ordering=order.Shuffle(10),
                                            filename='exp_1.exp')

    experiment.experiment_data['stimuli'] = stimuli_and_answers
    experiment.add_callback('trial', run_trial)
    experiment.save()

Running this script will create the experiment in the file exp_1.exp. We can now run sessions from the command line:

exp run exp_1.exp participant 1
# or
exp run exp_1.exp --next participant

Eventually, we can export the data to a text file:

exp export exp_1.exp exp_1_data.csv

Or, access the data in a Python session:

from experimentator import Experiment

data = Experiment.load('exp_1.exp').dataframe

In this example the data will be a pandas DataFrame with six columns: two index columns with labels 'participant' and 'trial', two columns from the IVs, with labels 'side' and 'display_time', and two data columns with labels 'reaction_time' and 'correct' (the keys in the dictionary returned by run_Trial).

Installation

Note

If you use experimentator in your work, published or not, please let me know. I'm curious to know what you use it for! If you do publish, citation information can be found here.

Dependencies

Experimentator requires Python 3.3 or later. It also depends on the following Python libraries:

• numpy
• pandas
• docopt
• schema
• PyYAML
• NetworkX

Required for tests:

• pytest

Required for generating docs:

• Sphinx
• numpydoc
• sphinx-rtd-theme

The easiest way to install these libraries, especially on Windows, is with Continuum's free Python distribution Anaconda. For experimentator, Anaconda3 or the lightweight Miniconda3 is recommended, although you can create a Python3 conda environment regardless of which version you initially download.

For example, to install dependencies to a clean environment (with name experiment):

conda update conda
conda create -n experiment python=3 pip
source activate experiment
conda install numpy pandas pyyaml networkx
pip install docopt schema

From PyPi

To install (and upgrade) experimentator:

pip install --upgrade experimentator

Be sure to run pip from a Python 3 environment.

From source (development version)

Experimentator is hosted on GitHub:

git clone git@github.com:hsharrison/experimentator
cd experimentator
pip install -e . --upgrade

Other libraries

Please, feel free to submit a pull request to add your software to one of these lists.

Alternatives

The Python ecosystem offers some wonderful alternatives that provide experiment logistics in addition to other functionality like graphics and input/output:

• expyriment: Graphics, input/output, hardware interfacing, data preprocessing, experimental design. If you are coming from the Matlab world, this is the closest thing to Psychtoolbox.
• OpenSesame: An all-in-one package with a graphical interface to boot. An impressive piece of software.

Complimentary libraries

What about all those important things that experimentator doesn't do? Here's a short selection. If you're already using Python some of these will go without saying, but they're included here for completeness:

• experimental design

  • pyDOE: Construct design matrices in a format that experimentator can use to build your experiment.

• graphics

  • PsychoPy: A stimulus-presentation library with an emphasis on calibration and temporal precision. Unfortunately, at the time of this writing it is not yet Python3-compatible, and so cannot be easily combined with experimentator.
  • Pygame: Very popular.
  • Pyglet: A smaller community than Pygame, but has several advantages, including cross-compatibility and a more pythonic API. Includes OpenGL bindings.
  • PyOpenGL: If all you need is to make OpenGL calls.

• graphical user interfaces

  • urwid: Console user interface library, ncurses-style.
  • wxPython: Python bindings for the wxWidgets C++ library.
  • PyQT: QT bindings.
  • PySide: Another QT option.
  • PyGTK: Python bindings for GTK+.

• statistics and data processing

  • pandas: Convenient data structures. Experimental data in experimentator is stored in a pandas DataFrame.
  • numpy: Matrix operations. The core of the Python scientific computing stack.
  • SciPy: A comprehensive scientific computing library spanning many domains.
  • Statsmodels: Statistical modeling and hypothesis testing.
  • scikit-learn: Machine learning.
  • rpy2: Call R from Python. Because sometimes the model or test you need isn't in statsmodels or scikit-learn.

License

Licensed under the MIT license.