nengo.tex

\documentclass{frontiersSCNS}

%%%%% Some customizations

\usepackage{listings}
\usepackage{inconsolata}
\graphicspath{{figures/}}

%%%%% Frontiers boilerplate

\usepackage{url,lineno}
\linenumbers

\copyrightyear{2013}
\pubyear{2013}

\def\journal{Neuroinformatics}
\def\DOI{}
\def\articleType{Methods Article}
\def\keyFont{\fontsize{8}{11}\helveticabold }
\def\firstAuthorLast{Bekolay {et~al.}}
\def\Authors{Trevor Bekolay\,$^{1,*}$, James Bergstra\,$^{1}$,
  Eric Hunsberger\,$^{1}$, Travis DeWolf\,$^{1}$, Terrence C. Stewart\,$^{1}$,
  Daniel Rasmussen\,$^{1}$, Xuan Choo\,$^{1}$, Aaron Voelker\,$^{1}$,
  and Chris Eliasmith\,$^{1}$}
\def\Address{$^{1}$Center for Theoretical Neuroscience,
  University of Waterloo, Waterloo, ON, Canada}
\def\corrAuthor{Trevor Bekolay}
\def\corrAddress{David R. Cheriton School of Computer Science,
  University of Waterloo, 200 University Avenue West,
  Waterloo, ON, N2L 3G1, Canada}
\def\corrEmail{tbekolay@uwaterloo.ca}

%%%%% Document begins

\begin{document}
\onecolumn
\firstpage{1}

\title[Nengo in Python]{Nengo: A Python tool for
  building large-scale functional brain models}
\author[\firstAuthorLast ]{\Authors}
\address{}
\correspondance{}
\extraAuth{}
\topic{Python in Neuroscience II}

\maketitle

\begin{abstract}
  Neuroscience currently lacks
  a comprehensive theory of
  how cognitive processes can be
  implemented in a biological substrate.
  The Neural Engineering Framework (NEF)
  proposes one such theory,
  but has not yet gathered
  significant empirical support,
  partly due to the technical
  challenge of building and simulating
  large-scale models with the NEF.
  Nengo is a software tool
  that can be used to build and simulate
  large-scale models based on the NEF;
  currently, it is the primary resource
  for both teaching how the NEF
  is used,
  and for doing research
  that generates specific
  NEF models
  to explain experimental data.
  Nengo 1.4, which was implemented in Java,
  was used to create Spaun,
  the world's largest functional
  brain model \citep{eliasmith2012}.
  Simulating Spaun
  highlighted limitations
  in Nengo 1.4's ability to
  support model construction with simple syntax,
  to simulate large models quickly,
  and to collect large amounts of data
  for subsequent analysis.
  This paper describes Nengo 2.0,
  which is implemented in Python
  and overcomes these limitations.
  It uses simple and extendable syntax,
  simulates a benchmark model on the scale of Spaun
  50 times faster than Nengo 1.4,
  and has a flexible mechanism
  for collecting simulation results.

  \tiny \keyFont{\section{Keywords:} Neural Engineering
    Framework, Nengo, Python, neuroscience, theoretical neuroscience,
    control theory, simulation}
\end{abstract}

\section{Introduction}

Modeling the human brain
is one of the greatest
scientific challenges of our time.
Computational neuroscience
has made significant advancements
from simulating low-level biological parts in great detail,
to solving high-level problems that humans find difficult;
however, we still lack a mathematical account of
how biological components
implement cognitive functions such as sensory processing,
memory formation, reasoning, and motor control.
Much work has been put into neural simulators
that attempt to recreate neuroscientific
data in precise detail with the thought that
cognition will emerge by connecting
detailed neuron models according
to the statistics of biological synapses \citep{markram2006}.
However, cognition has not yet emerged
from data-driven large scale models,
and there are good reasons to think
that cognition may never emerge \citep{eliasmith2013a}.
At the other end of the spectrum,
cognitive architectures \citep{anderson2004,aisa2008}
and machine learning approaches \citep{hinton2006}
have solved high-dimensional statistical problems,
but do so without respecting biological constraints.

Nengo\footnote{Nengo is available for download through
  the Python Package Index at
  \url{https://pypi.python.org/pypi/nengo}.
  The full source is available at
  \url{https://github.com/ctn-waterloo/nengo}.}
is a neural simulator
based on a theoretical framework proposed
by \citet{eliasmith2003}
called the Neural Engineering Framework
(NEF).
The NEF is a large-scale modeling approach
that can leverage single neuron models
to build neural networks with
demonstrable cognitive abilities.
Nengo and the NEF has been used to build
increasingly sophisticated neural subsystems
for the last decade
(e.g., path integration [\citealp{conklin2005}],
working memory [\citealp{singh2006}],
list memory [\citealp{choo2010}],
inductive reasoning [\citealp{rasmussen2014}],
motor control [\citealp{dewolf2011}],
decision making [\citealp{stewart2012}])
culminating recently with Spaun,
currently the world's
largest functional brain model \citep{eliasmith2012}.
Spaun is a network of 2.5 million spiking neurons
that can perform eight cognitive tasks
including memorizing lists and inductive reasoning.
It can perform any of these eight tasks
at any time by being presented
the appropriate series of images
representing the task to be performed;
for example, sequentially presenting images
containing the characters $A3[1234]$ instructs Spaun
to memorize the list $1234$.
If asked to recall the memorized list,
Spaun generates motor commands for a simulated arm,
writing out the digits $1234$.
While the tasks that Spaun performs are diverse,
all of the tasks use a common set of
functional cortical and subcortical components.
Each functional component corresponds
to a brain area that has been hypothesized
to perform those functions
in the neuroscientific literature.

The NEF provides principles
to guide the construction of
a neural model
that incorporates anatomical constraints,
functional objectives, and
dynamical systems or control theory.
Constructing models from this starting point,
rather than from single cell electrophysiology
and connectivity statistics alone,
produces simulated data that explains
and predicts a wide variety of experimental results.
Single cell activity \citep{stewart2012},
response timing \citep{stewart2009a},
behavioral errors \citep{choo2010},
and age-related cognitive decline \citep{rasmussen2014},
of NEF-designed models match
physiological and psychological findings
without being built specifically into the design.
These results are a consequence of
the need to satisfy functional objectives
within anatomical and neurobiological constraints.

The transformation principle of the NEF
proposes that the connection weight matrix
between two neural populations
can compute a nonlinear function,
and can be factored into
two significantly smaller matrices.
By using these factors
instead of full connection weight matrices,
NEF-designed models are
more computationally efficient,
which allows Nengo to run
large-scale neural models
on low-cost commodity hardware.

In order to make Nengo more simple, extensible, and fast,
we have rewritten Nengo 2.0 from scratch in Python,
leveraging NumPy \citep{oliphant2007}
for manipulating large amounts of data.
While NumPy is its only dependency,
Nengo contains optional extensions
for plotting if Matplotlib is available
\citep{hunter2007}
and for interactive exploration
if IPython is available \citep{perez2007}.
Since Nengo only depends on one third-party library,
it is easy to integrate Nengo models
in arbitrary CPython programs,
opening up possibilities
for using neurally implemented algorithms
in web services, games, and other applications.

Nengo 2.0 has a simple object model,
which makes it easy to
document, test, and modify.
Model creation and simulation are decoupled,
allowing for models to be run with
other simulators as drop-in replacements for Nengo 2.0's
platform-independent reference simulator.
To date, we have implemented one other simulator
that uses PyOpenCL to take advantage
of a GPU or multicore CPU.
The OpenCL simulator can simulate
large models on the scale of Spaun
at least 50 times faster than Nengo 1.4
using inexpensive commodity hardware.

In all, Nengo 2.0 provides a platform for
simulating larger and more complex models than Spaun,
and can therefore further test the NEF
as a theory of neural computation.

\section{Neural Engineering Framework (NEF)} \label{sec:nef}

The Neural Engineering Framework (NEF; \citealp{eliasmith2003})
proposes three quantitatively specified principles
that enable the construction
of large-scale neural models.
Briefly, this mathematical theory defines:
\begin{enumerate}
  \item \textbf{Representation:} A population of neurons
    collectively represents a time-varying vector of real numbers
    through nonlinear encoding and linear decoding.
  \item \textbf{Transformation:} Linear and nonlinear
    functions on those vectors
    are computed by linear decodings
    that are used to analytically compute
    the connections between populations of neurons.
  \item \textbf{Dynamics:} The vectors represented
    by neural populations can be considered state variables
    in a (linear or nonlinear) dynamical system,
    and recurrent connections can be computed
    using principle 2.
\end{enumerate}
Figure~\ref{fig:nef} provides a graphical
summary of these three principles.

\paragraph{Representation}
Information is encoded by populations of neurons.
The NEF represents information
with time-varying vectors of real numbers,
allowing theorists to propose possible
neural computations by
manipulating that information
using conventional mathematics.
The NEF suggests that we can characterize
the \textit{encoding} of
those vectors by injecting
specific amounts of current into
single neuron models based on
the vector being encoded.
This drives the neuron,
causing it to spike.
With enough neurons,
the originally encoded vector
can be estimated
through a \textit{decoding} process.
This idea is a kind of population coding
\citep{georgopoulos1986, salinas1994}, but generalized
to vectors of arbitrary dimensionality.

In the encoding process, the input signal drives
each neuron based on its \textit{tuning curve},
which describes how likely
that neuron is to respond to a given input signal.
The tuning curve is a function of the gain
of a neuron (how quickly the activity rises),
the bias (the activity of a neuron given no signal),
and the \textit{encoding weight}
(the direction in the input vector space
that causes the neuron to be the most active).
Importantly, tuning curves can be determined
for any type of neuron,
and therefore the encoding process
(and the NEF as a whole)
is not dependent on any particular neuron model.

In the decoding process,
the spike trains are first filtered
with an exponentially decaying filter
accounting for the process
of a spike generating a postsynaptic current.
Those filtered spike trains are summed together
with weights that are determined
by solving a least-squares minimization problem.
Note that these decoding weights
do not necessarily depend on the input signal;
instead, we typically perform
this minimization on points
sampled from
the vector space
that the population represents.

In Nengo, the representation principle
can be seen in the \texttt{Ensemble}
object (see Section~\ref{sec:ensemble}).

\paragraph{Transformation}
Neurons communicate through
unidirectional connections called synapses.
When a neuron spikes,
it releases neurotransmitter across the synapse,
which typically causes some amount of current
to be imparted in the postsynaptic (downstream) neuron.
Many factors affect the
amplitude of the imparted current;
we summarize those factors
in a scalar connection weight
representing the strength
of the connection between two neurons.
In order to compute any function,
we set the connection weights between
two populations to be the product of
the decoding weights for that function
in the first population,
the encoding weights
for the downstream population,
and any linear transform.

This implies that the NEF makes
a hypothesis about synaptic weight matrices;
specifically, that they
have low rank, and can be factored
into encoders, decoders, and a linear transform.
Note that, in practice, we rarely use
the full connection weight matrix,
and instead store
the encoders, decoders, and linear transform separately
(i.e., the three factors of the connection weight matrix).
This provides significant
space and time savings during simulation.

In Nengo, the transformation principle
can bee seen in the \texttt{Connection}
object (see Section~\ref{sec:connection}).

\paragraph{Dynamics}
While feedforward vector transformations
suffice to describe
some neural systems,
many require persistent activity through recurrent connections.
When recurrent connections are introduced,
the vectors represented by neural populations
can be thought of as state variables
in a dynamical system.
The equations governing dynamics
in such a system
can be designed and analyzed
using the methods of control theory,
and translated into neural circuitry
using the principles
of representation and transformation.

In Nengo, dynamics can be seen when
an \texttt{Ensemble} is connected to itself.
Several of the \texttt{Network}s
implemented in Nengo also exhibit dynamics.

\paragraph{NEF and Nengo}
Large models can be built
by using the principles of the NEF
as connectable components
that describe neural systems,
just as a circuit diagram
describes an electronic circuit.
The goal of Nengo is to enable
modellers to create and connect those components.
Ensembles describe
what information is being represented,
and connections describe
how that information is transformed.
Nengo implements those descriptions
with its object model,
and translates those objects
to a network of interconnected neurons,
situating it as a ``neural compiler''
that translates
a high-level functional model
to a low-level neural model.

\section{Nengo object model}

To describe an NEF model,
Nengo defines six core objects.
\begin{enumerate}
  \item The \texttt{Ensemble} contains a group of neurons
    that encodes a time-varying vector of real numbers.
  \item The \texttt{Node} represents non-neural information,
    such as sensory inputs and motor outputs.
  \item The \texttt{Connection} describes how
    nodes and ensembles are connected.
  \item The \texttt{Probe} gathers data during a simulation
    for later analysis.
  \item The \texttt{Network} encapsulates a functionally related
    group of interconnected nodes and ensembles.
  \item The \texttt{Model} encapsulates a Nengo model.
\end{enumerate}
These six objects contain symbolic information
about a Nengo model,
enabling a strict division between
model construction and simulation.
This allows a Nengo model
to be run on multiple simulators.

\subsection{Ensemble} \label{sec:ensemble}

An \texttt{Ensemble} is
a population of neurons
that represents information
in the form of a real-valued vector.
When creating an ensemble,
the user must provide a name,
an object that describes
a population of neurons,
and the dimensionality
(i.e., the length of the vector it represents).
For example,
\begin{quote}
  \texttt{nengo.Ensemble(nengo.LIF(50, tau\_ref=0.002), 1)}
\end{quote}
describes an ensemble
that uses 50 leaky integrate-and-fire neurons \citep{lapicque1907}
with a 2 millisecond refractory period
to represent a one-dimensional vector.
The \texttt{nengo.LIF} class defines
the parameters of the LIF neurons symbolically
so that each simulator can compute
the LIF nonlinearity efficiently.
The neuron model used by the ensemble
is changed by passing in a different symbolic neuron object;
however, the simulator used must be aware
of that type of neuron.

Other attributes of the \texttt{Ensemble},
such as its encoding weights,
can be specified
either as keyword arguments
to the \texttt{Ensemble} constructor,
or by setting an attribute on the instantiated object.
While an ensemble makes a hypothesis
about the information being represented by neurons,
these additional attributes
allow modellers to set
neural parameters according to \textit{in vivo}
electrophysiology data.
If these attributes are not set,
Nengo attempts to maintain
neurobiological constraints
by using default parameters
consistent with
neocortical pyramidal cells.

\subsection{Node}

A \texttt{Node} contains a user-defined
Python function that directly calculates
the node's outputs from its inputs at each timestep.
Available inputs include
the simulator timestep,
the decoded output of an ensemble,
or the output of another node.
However, unlike ensembles,
there are no constraints on the type
of function that the node computes.
A node can track any number of variables internally,
and use the state of those variables
when computing its function.
For example, it can interact directly with hardware,
and interface with other programs
using shared memory or sockets.

Generally, a node represents information
that cannot be decoded from an ensemble.
As a simple example,
a node can be used to model sensory stimuli
that are predefined functions of time.
As a more sophisticated example,
a node can be used to model
a complex experimental environment that
both provides input to the neural model
and responds to the neural model's output.
Nodes allow Nengo to represent
neural components,
the body that those components drive,
and the environment that body interacts with
in a single unified model.
This makes Nengo models more explicit,
and enables simulators
to control and optimize node execution.

\subsection{Connection} \label{sec:connection}

Ensembles and nodes can be connected together
in several ways.
A \texttt{Connection} contains symbolic information
about how two objects are connected.
That information either includes
a factored or full connection weight matrix,
or includes enough information
to generate weights during simulation.

When an ensemble is connected
to another object,
the connection implements
the NEF's transformation principle.
In other words, the \texttt{Connection}
allows ensembles to project
encoded information---or
a transformation of that information---to
another ensemble or node.
This functionality is what enables Nengo models
to appear conceptual,
even though the underlying implementation
can translate that connection
to synaptic weights.

However, neurons in an ensemble can be directly connected
to neurons in another ensemble
with synaptic connection weights
by connecting an ensemble's
neurons directly to another ensemble's neurons
(e.g., \texttt{nengo.Connection(ens1.neurons, ens2.neurons, ...)}).
All connections can be temporally filtered,
and the weights involved in the connection
can be modified over time with learning rules.

\subsection{Probe}

A \texttt{Probe} monitors
a particular part of another object
in order to record its value throughout a simulation.
Nengo models contain many variables
that change over time,
including membrane potentials,
spike events, and encoded vectors.
It is resource intensive to store the values of
large numbers of variables
at each timestep, and it is also not necessary,
as typically only a small fraction
of these variables are analyzed after a simulation.
The modeller chooses which variables to
record by connecting a probe to an object.

Like nodes, a probe could be implemented
outside of the neural model.
However, doing so requires detailed knowledge
of the simulator,
and can incur significant overhead
if not implemented carefully.
For these reasons, we have made probes
a core component of a Nengo model,
and are therefore explicit
and optimizable.
Further, since probes are described
at a symbolic level,
the underlying implementation
can output probed data in many different formats.
Currently, simulators store probed data
directly in memory,
but the ability to store data
in files or to stream data
directly to sockets is forthcoming.

\subsection{Network}

A network is a collection of interconnected ensembles and nodes.
Networks provide a way of grouping together
a set of connected objects
that collectively perform a complex function.
Encapsulating them in a network
makes its purpose explicit
and hides the complexity of the function
(see Section~\ref{sec:cconv} for an example).
This grouping can be hierarchical,
as networks can contain other network.
\texttt{Network} is a base class designed to be
subclassed by modellers.
The code that creates and connects
several objects in a model can be
grouped into a \texttt{Network} subclass
with only minor changes.
Nengo comes with several networks already implemented
which can be used directly,
or can be used as a template
for modelers wanting to implement their own networks.

As a simple example, the \texttt{Integrator} network
is composed of only one recurrently connected ensemble.
By encapsulating that logic in a network,
the purpose of that ensemble is made explicit.
As a complex example,
the \texttt{BasalGanglia} network
is composed of five groups of ensembles
connected with several specific functions
that together implement a ``winner-take-all'' circuit.
Encapsulating the code to create those ensembles
and connections in a network
makes a complicated section of code
easy to include in many different models.

\subsection{Model}

The \texttt{Model} object is a container
for Nengo objects.
Conceptually, they are similar to networks,
but are privileged in that
simulators must have a model
passed into their constructor,
making the \texttt{Model} akin
to a network that contains
all of the objects defined for a Nengo model.
A simulator's job is to simulate a model.

\section{Nengo simulators} \label{sec:simulators}

Decoupling model creation and simulation
has been done previously
by PyNN \citep{davison2009}.
In PyNN, the same Python script
can be used to run a model
on four different simulators.
Nengo follows this programming model by
decoupling neural model creation and simulation,
which enables Nengo simulators
to allocate memory and schedule computations
in the most efficient ways possible.
Simulators are given a \texttt{Model}
as an argument;
this \texttt{Model} is a static symbolic description.
The simulator can take the model description
and build whatever data structures
are best suited to implement the simulation.

We have implemented a platform-independent
reference simulator as an example
for simulator designers.
This simulator is not a specification;
any object that accepts a Nengo \texttt{Model}
as an argument is considered a Nengo simulator.
To show that model creation and simulation
are fully decoupled,
we have also implemented
an OpenCL simulator
that uses PyOpenCL to parallelize
computations on GPUs and multicore CPUs.
However, in the remainder of this section,
we will describe
the reference simulator implementation;
the OpenCL simulator shares many
of the reference simulator's architectural choices,
but the details of its implementation
include OpenCL-specific optimizations
that are beyond the scope of this paper.

\subsection{Nengo reference simulator}

The Nengo reference simulator
makes a copy of the objects in the model
and fills in many of the details
not specified at the symbolic level.
For example, \textit{encoders}
are often not specified when the model is created,
so the reference simulator randomly chooses
them as unit vectors in the space that
the ensemble is representing.
After filling in these details,
the reference simulator
builds a reduced set of objects
that describe the computations
occurring in the model.
Specifically, the simulator
uses signals, which represent values,
and operators, which represent computations
performed on signals.
Figure~\ref{fig:sim} shows the signals
and operators used in a simple model.

\paragraph{Signals}

A \texttt{Signal} represents any number that
will be used by the simulator.
Several signals are created
for each high-level Nengo object;
for example, for each ensemble,
the simulator creates signals
that represent the high-level input
signal that will be encoded
to input currents,
and the encoding weights.
The ensemble also contains a neural population,
for which the simulator creates signals that represent
input currents, bias currents,
membrane voltages, and refractory times for each cell.

As can be seen in Figure~\ref{fig:sim},
the signals used in a Nengo simulation
can be conceptually grouped into
those that track low-level neural signals,
and those that track high-level signals
defined by the NEF.
Other neural simulators only track
low-level signals.
Operators commonly map
between related low- and high-level signals.

\paragraph{Operators}

Operators represent computations
to be performed on signals on each timestep.
Once the model has been built,
only a small set of mathematical
operations are necessary for simulation.

Many of the computations
done in a simulation
are linear transformations (e.g.,
the decoding and encoding steps
in Figure~\ref{fig:sim}),
and therefore can share a common operator;
this is helpful for parallelizing computations.
Nonlinear functions, however,
require specific operators.
Each supported neuron model and learning rule
has an associated operator.
The simulator explicitly maps
from symbolic neuron objects in ensembles
and from symbolic learning rule objects
in connections to their associated operators.

\paragraph{Reference simulator}

Before the first timestep, the reference simulator
\begin{enumerate}
  \item fills in unspecified details of high-level objects,
  \item translates high-level objects to
    a set of signals and operators,
  \item allocates NumPy arrays for each signal, and
  \item sorts operators based on a dependency graph.
\end{enumerate}
On each timestep, the reference simulator
\begin{enumerate}
  \item computes each operator in order, and
  \item copies probed signals to memory.
\end{enumerate}
Figure~\ref{fig:sim} depicts
the state of the reference simulator
after two timesteps of a simple model;
all subsequent timesteps perform the same
operations as the first two.

\section{Example scripts} \label{sec:examples}

The scripting interface provides
a simple way to add Nengo objects to a model,
simulate that model,
and extract data collected during the simulation.
Rather than list the functions in the scripting interface,
we instead provide three concrete example scripts
that highlight the types of models that can be
built with Nengo.
We have also implemented two of these three examples
in PyNN to provide a comparison
for the length and clarity of the code
describing the models.

\subsection{Communication channel} \label{sec:comm-channel}

% A communication channel represents some signal
% and transmits it unchanged to another population
% (i.e. it implements an identify function $f(x) = x$).
% One of the most important operations
% in large neural models is routing with this sort of channel.
% When connected recurrently,
% a communication channel can also be seen as
% a simple memory, mapping a real-valued number
% $x$ to itself at a future time [awkward wording].
% In this simple case,
% we can see how the principles of the NEF
% help to translate a mathematical description
% of our model to neural circuitry.
% An Ensemble embodies the idea
% that a population of neurons
% should represent a real-valued quantity
% $x \in \mathbb{R}^1$.
% Technically, Nengo does this
% by creating a random projection matrix
% $E$ and driving the first population
% with current $Ex + b$,
% where $b$ is a vector of bias values.
% A vector $s$ of spikes emitted
% by this population are weighted and combined
% by a {\em decoder} matrix D
% so that $Ds \approx f(x)$.
% Similarly the second population
% has encoders $E_2$ and $D_2$.
% A full weight matrix $W$
% of synapse strengths
% between populations 1 and 2
% can be obtained
% by multiplying matrices $E_2$ and $D_1$ together:
% $W=E_2 D_1$.
% We ran simulations in PyNN
% by computing this product explicitly
% and using $W$ and the connection matrix.
% Unfortunately it much slower to multiply
% $s$ by $W$ than it is to compute $E_2(D_1s)$,
% which is why Nengo
% is a faster simulator for NEF-style models.

As detailed in Section~\ref{sec:nef},
NEF models are based on the principles
of representation, transformation, and dynamics.
One of the most important operations
in a large neural model is to route
represented information from
one ensemble to another without any change.
For example, in the visual system of Spaun,
a high-dimensional representation of the visual field
is compressed to a low-dimensional representation,
and then sent unchanged to several areas,
including the working memory
and action selection networks.
This routing is implemented
with a transformation
called a communication channel.
This transform simply implements
the identity function, $f(x) = x$.

Figure~\ref{fig:comm-channel}
depicts a scalar communication channel
in which band-limited Gaussian white noise
is represented in one ensemble
and projected to another ensemble.

The communication channel
is a simple enough model
that it can be readily implemented in PyNN.
Figure~\ref{fig:pynn} compares the code
for implementing a communication channel in Nengo and PyNN.
This figure highlights many of the differences
between Nengo models and conventional neural models;
we also use these script for benchmarking
(see Section~\ref{sec:benchmark}).

\subsection{Lorenz attractor network} \label{sec:lorenz}

While the communication channel exemplifies
the representation and transformation
principles of the NEF,
the Lorenz attractor exemplifies the dynamics principle.
Many models in theoretical neuroscience
are based on attractor networks \citep{amit1992, deco2003}.
The NEF has been used in the past
to implement many different types of
attractor networks by recurrently connecting
ensembles with functions that implement
dynamical systems \citep{eliasmith2005}.
Figure~\ref{fig:lorenz} depicts
a Nengo implementation of the Lorenz chaotic attractor
with a single ensemble
composed of 2000 leaky integrate-and-fire neurons.
We have implemented the Lorenz attractor
in PyNN for benchmarking purposes
(code not shown; the PyNN script is $\sim$100 lines long,
while the Nengo script in Figure~\ref{fig:lorenz}
is 20 lines long).

\subsection{Circular convolution} \label{sec:cconv}

Communication channels and attractor networks
show up in many Nengo models,
but are still relatively simple
to implement without Nengo,
as can be seen with the PyNN implementation
in Figure~\ref{fig:pynn}.
As the NEF has been used to construct
larger models that have the capabilities
of non-neural cognitive models,
a theory called the
Semantic Pointer Architecture \citep{eliasmith2013}
has emerged.
This theory uses high-dimensional vectors
as symbol-like structures
that be combined together to form
novel concepts.

One of the functions that
is performed on these vectors
is to compress two $n$-dimensional vectors
into a single $n$-dimensional vector,
which can be decompressed
into noisy versions of
the two originally compressed vectors.
We implement this compression
using the circular convolution function.
Circular convolution
is best implemented in a two-layer network,
rather than in a single connection,
which we have simplified through
the \texttt{CircularConvolution} network.
The complexity encapsulated in that network
can be seen in Figure~\ref{fig:cconv}.

Unlike the previous two examples,
we do not implement
circular convolution in PyNN.
The resulting script would be
too long to be instructive.

\section{Benchmarks} \label{sec:benchmark}

While benchmark models are not indicative
of performance on all models,
increasing simulation speed
was a primary goal of Nengo 2.0.
To validate that performance has improved,
we ran the models described in Section~\ref{sec:examples}
for various numbers of neurons and dimensions
for each ensemble.

The communication channel and Lorenz attractor
are small models that demonstrate
the principles of the NEF.
Their small size enables us to write
PyNN scripts that implement roughly
the same functionality
with Brian \citep{goodman2008}, NEURON \citep{hines2009},
and NEST \citep{eppler2009}.\footnote{We
  were unable to compile PCSim \citep{pecevski2009} on the
  machine on which we ran benchmarks.}
We ran each parameter set five times
on the same machine,
and plot the mean time elapsed
in Figure~\ref{fig:benchmarks}.
In most cases, the coefficient of variation
for the five sample times
is well below 0.1, except for two
outliers with coefficients of 0.18 and 0.22,
overall indicating that the reported means are robust.
The results, shown in Figure~\ref{fig:benchmarks}A and B,
suggest that all versions of Nengo are significantly
faster than the simulators accessible
through PyNN, especially
as the size of models increases.
This is likely due to Nengo's
use of factorized weight matrices,
rather than storing and computing with
the entire weight matrix
on each timestep.
While NEST and NEURON were not
run on multiple cores using message passing,
the reference simulator of Nengo
also only uses one CPU core.
The results further suggest that Nengo 2.0's
simulators are faster than Nengo 1.4's simulator.

As a larger-scale example,
we have also benchmarked
the circular convolution model.
Circular convolution is an important test case,
as a significant portion of Spaun's
2.5 million neurons are used to
implement circular convolution.
In this case, only versions of Nengo were tested.
Instead of running each simulation multiple times,
we instead ran the simulator for 10 timesteps
in order to fill various levels of CPU or GPU cache,
and then ran the simulator for 1000 more timesteps;
there is very little variance using this method.
As can be seen in Figure~\ref{fig:benchmarks}C,
for large models, the OpenCL simulator
performs much faster than Nengo 1.4;
in particular, a Radeon 7970 GPU performs
500-dimensional circular convolution
with about half a million neurons
faster than real time,
and 50 times faster than Nengo 1.4.
In the 50-dimensional case,
the Radeon 7970 GPU is 200 times faster
than Nengo 1.4.
Additionally, although both Nengo 1.4
and the OpenCL simulator on CPUs
use all available CPU cores,
Nengo's OpenCL simulator is many times faster.

\section{Discussion}

\subsection{Comparison to similar projects}

There are many other neural simulators
dedicated to building large-scale neural models
\citep{eppler2009,goodman2008,hines2009},
and many tools for simulating cognitive phenomena
with various levels of biologically plausibility
\citep{cooper1998,sun2001,anderson2004,franklin2007,
  aisa2008,dekamps2008,laird2012}.
However, Nengo is unique in being built
on a theoretical framework
that has enabled a cognitive architecture
(the Semantic Pointer Architecture)
that maintains a high level
of biological plausibility,
and has been validated
through the Spaun model and other past work.

The most closely related projects
in terms of software design
are PyNN \citep{davison2009}
and Topographica \citep{bednar2009},
both of which provide a high-level scripting
interface to low-level neural simulators.
PyNN in particular has a similar to the
high-level object model in Nengo,
and provides a convenient interface
to the three most widely used
neural simulators, according
to a survey by \citet{hanke2011}.

The APIs of Nengo and PyNN are similar,
but differ significantly
in how groups of neurons are connected together.
In Nengo, connections commonly describe
the mathematical operation that is performed
through the connection between
two ensembles;
e.g., \texttt{nengo.Connection(A, B,
function=square)} connects ensemble A
to ensemble B, transmitting the square of
the value represented by A to B.
In PyNN, connections commonly describe
features of the connection weight matrix
between two populations;
e.g., \texttt{FixedProbabilityConnector(0.5)}
connects two ensembles together,
with a probability of 0.5
that there will be a connection
between a pair of neurons in the two populations.
This difference reflects the
fundamental difference that Nengo
is built on a theoretical framework
that enables modellers to think
about information processing in the brain
at a conceptual level.

On the neural simulator side,
we have shown that both Nengo's
reference simulator and OpenCL simulator
are able to simulate
two benchmark models much faster
than Brian, NEST and NEURON
(see Figure~\ref{fig:benchmarks}).
This is, in part,
because Nengo stores the factors
of the connection weight matrix,
rather than storing the entire matrix.
However, these simulators are able to simulate
many detailed neuron models
and learning rules,
and have access to
a wealth of existing neuron models
and learning rules.
Because Nengo 2.0 is in an earlier development stage,
many of these detailed neuron models
and learning rules remain to be added.
Neural simulators like Brian, NEST,
and NEURON are therefore currently better suited for
simulating a wider range of single cell models,
while Nengo is designed for large networks
of simple neural models
that are connected together according to
the principles of the Neural Engineering Framework,
and can simulate these types of models efficiently.

One key difference between Nengo's simulators
and traditional neural simulators
is the target platform.
While NEST and NEURON
can be run on commodity hardware,
networks of modest size are
typically simulated in high-performance computing
environments by using
the Message-Passing Interface (MPI).
Nengo enables large-scale neural simulation
on commodity hardware,
allowing researchers and hobbyists
without access to large computing clusters
the ability to experiment with
theoretical neuroscience
as is currently practiced
in cutting edge research.
In particular, GPUs are a
powerful, low-cost computing resource
that are available in most
modern workstations.
The OpenCL simulator makes full use
of GPUs, while the previously discussed simulators
currently do not.

\subsection{Future work}

Our short-term goal
is to implement the Nengo 1.4 use cases
not currently covered by Nengo 2.0.
While we have ensured that all of the models
currently used by Nengo 1.4 tutorials
can be run in Nengo 2.0,
several large models, like Spaun,
include custom extensions written in Java.
We will incorporate useful extensions
in Nengo's API directly,
and reimplement more specific extensions
to use Nengo's API.

We are also developing two simulators
that will take the same NEF model description
as the existing simulators,
but will target two pieces
of neuromorphic hardware
to achieve greater speed and power efficiency
than the OpenCL simulator.

Our long-term goal is to
create a graphical interface
to build models and
interactively inspect simulations.
Nengo 1.4 includes a graphical interface
that includes an interactive simulation inspector.
We will use our experience building that interface
to construct an improved interface
for Nengo 2.0.

Additionally, we hope that in the future
some work in Nengo will be
done through outside contributions.
Nengo 2.0 is a complete rewrite that
has started with a deliberately minimal base
and a well-defined API
in order to make development easier than in
previous versions.

\subsection{Conclusion}

Nengo 2.0 is the next generation of Nengo.
It has been rewritten from scratch,
but can already simulate most models
that have been built using Nengo 1.4.
It does this with 11\% as many lines of code
as its predecessor,
and interacts seamlessly with
other scientific Python tools.
While the reference simulator
is simple and easy to understand,
the OpenCL simulator is extremely fast;
it can simulate circular convolution models
50-200 times faster than Nengo 1.4,
which itself is faster than alternative simulators
on simpler models.
This makes the creation and simulation
of models that are many times larger than Spaun
tractable with current hardware.
These models will further test
the NEF as a theory of neural computation;
Nengo makes those models
accessible to anyone with
a modern computer.

\appendix
\section*{Appendix: Neural Engineering Framework details}

This description of the Neural Engineering Framework
is adapted from the supplementary material
of \citep{eliasmith2012} with permission.

The Neural Engineering Framework (NEF; \citealp{eliasmith2003})
provides a set of methods
for building biologically plausible models
based on a functional specification of a neural system.
The central idea behind the NEF
is that a group of spiking neurons
can represent a vector space over time,
and that connections between
groups of neurons
can compute functions on those vectors.
The NEF provides a set of methods
for determining what the connections
need to be to compute a given function
on the vector space
represented by a group of neurons.

Suppose we wish to compute the function
$\mathbf{y}=f(\mathbf{x})$,
where vector space $\mathbf{x}$
is represented in population A,
and vector space $\mathbf{y}$
is represented in population B.
To do so, the NEF assumes that
each neuron in A and B has a
``preferred direction'' or ``encoding'' vector.
The preferred direction vector
is the vector (i.e. direction in the vector space)
for which that neuron will fire most strongly.
The spiking activity of every neuron
in population A can be written as
\begin{equation} \label{eq:enc}
  a_{i}(\mathbf{x})=G_{i}[\alpha_{i}\mathbf{e}_{i}\mathbf{x}+J_{i}^{bias}],
\end{equation}
where $a_{i}$ is the spike train
of the $i$th neuron in the population,
$G$ is the spiking neural nonlinearity,
$\alpha$ is the gain of the neuron,
$\mathbf{e}$ is the preferred direction
(or ``encoding'') vector,
and $J^{bias}$ is a bias current
to account for background activity of the neuron.
Notably, the elements in the square brackets
determine the current flowing into the cell,
which then drives the spiking
of the chosen single cell model $G$.
Equation \eqref{eq:enc} describes
how a vector space
is encoded into neural spikes.
This equation is depicted
for a 1-dimensional vector space
in Figure~\ref{fig:nef}A.

The NEF proposes that linear decoders
can be found to provide an appropriate
estimate of any vector $\mathbf{x}$
given the neural activities from
the encoding equation.
We can write this as a decoding equation:
\begin{equation} \label{eq:dec}
  \hat{\mathbf{x}}=\sum_{i}^{N}a_{i}\mathbf{(x)}\mathbf{d}_{i},
\end{equation}
where $N$ is the number of neurons in the group,
$\mathbf{d}_{i}$ are the linear decoders,
and $\hat{\mathbf{x}}$ is the estimate of
the input driving the neurons.

The NEF determines this complementary decoding
for any given encoding.
Specifically, this decoding is found
using a least-squares optimization:
\begin{equation} \label{eq:error}
  E=\frac{1}{2}\int[\mathbf{x}-\sum_{i}a_{i}(\mathbf{x)}\mathbf{d}_{i}]^{2}d\mathbf{x},
\end{equation}
where $\mathbf{d}_{i}$ are the decoding vectors
over which this error is minimized.

In effect, this optimization process
replaces learning in most other
approaches to constructing neural networks.
This optimization is not
biologically plausible on its own,
although networks generated in
this manner can also be learned
with a spike-based rule \cite{macneil2011}.

The decoding process is depicted
in Figure~\ref{fig:nef}B,
where the optimal linear decoders
have been found and used for eight neurons.
Notably, this kind of temporal decoding
requires an assumption about
the nature of the temporal filter being used.
Here we assume that post-synaptic currents
are such filters, and set the time constants
to reflect the kind of neurotransmitter receptors
in the connection
(e.g., AMPA receptors have short time constants, $\sim10$ ms,
and NMDA receptors have longer time constants, $\sim50$ ms).

Such temporal filters map
to biophysical processes
once we connect groups of neurons together.
Defining the encoding and decoding for
groups A and B using equations
\eqref{eq:enc} and \eqref{eq:dec}
provides a means of connecting groups.
For example, we can substitute
the decoding of A into the encoding of B,
thereby deriving connection weights
\begin{equation} \label{eq:weights}
  \omega_{ij}=\mathbf{d}_{i}\alpha_{j}\mathbf{e}_{j},
\end{equation}
where $i$ indexes the neurons in group A
and $j$ indexes the neurons in B.
These weights will compute the function
$\mathbf{y}=\mathbf{x}$
(where $\mathbf{y}$ is the vector space
represented in B and $\mathbf{x}$
is the vector space represented in A).
For the more general case,
it is possible to solve for decoders
$\mathbf{d}_{i}^{f}$
for any function by substituting
$f(\mathbf{x})$ for $\mathbf{x}$
in equation \eqref{eq:error}, i.e., solving
\begin{equation} \label{eq:error-fcns}
E=\frac{1}{2}\int[f(\mathbf{x})-\sum_{i}a_{i}(\mathbf{x)}\mathbf{d}_{i}^{f}]^{2}d\mathbf{x}.
\end{equation}

In addition, if the function to be computed is linear,
the relevant linear operator can be introduced
into Equation \eqref{eq:weights}.
The resulting general weight equation
for computing any combination
of linear and nonlinear functions becomes:
\begin{equation} \label{eq:weights-general}
  \omega_{ij}=\alpha_{j}\mathbf{d}_{i}^{f}\mathbf{L}\mathbf{e}_{j}
\end{equation}
for any nonlinear function $f$ and
$N_{B}\times N_{A}$ linear operator $\mathbf{L}$.
Computing the linear function
$\mathbf{y}=\mathbf{-x}$ and
computing the nonlinear function
which is the element-wise
square of the vector $\mathbf{x}$
(i.e., $\mathbf{y}=[x_{1}^{2},x_{2}^{2},...,x_{n}^{2}]$
is shown in Figure~\ref{fig:nef}C.

This brief discussion is insufficient
to fully introduce the generality of the NEF.
However, showing how to compute linear
and nonlinear functions of vector spaces
is sufficient for many neural computations.
As these same methods can be used
to compute connection weights for recurrent connections,
the NEF also allows for the neural implementation
of a wide variety of linear and nonlinear
dynamical systems in recurrent networks.
A simple harmonic oscillator is shown in Figure~\ref{fig:nef}D.

\section*{Disclosure/Conflict-of-Interest Statement}

The authors declare that the research was conducted in the absence of
any commercial or financial relationships that could be construed as a
potential conflict of interest.

\section*{Author contributions}

TB led development of
Nengo's object model and scripting interface,
wrote the text of the paper,
and prepared all of the figures.
JB led development of
the Nengo reference simulator and OpenCL simulator,
edited text, created an early version
of Figure~\ref{fig:sim},
and ran the benchmarks shown in
Figure~\ref{fig:benchmarks}C.
EH contributed significantly
to Nengo and both of its simulators, and edited text.
TD led development
of a Theano-backed
version of Nengo that
identified issues with Theano,
and provided the base for
the version of Nengo described in this paper.
TCS contributed
to Nengo, helped implement
the PyNN scripts used in
Sections~\ref{sec:examples} and \ref{sec:benchmark},
and implemented the Nengo 1.4 scripting interface
on which the Nengo object model is based.
DR contributed
to Nengo and the reference simulator,
and edited text.
XC contributed to Nengo
and the reference simulator.
AV ran the benchmarks
shown in Figure~\ref{fig:benchmarks}A and B.
CE oversaw all development,
contributed to Nengo,
wrote the NEF appendix,
and co-created the NEF with Charles Anderson.

\section*{Acknowledgments}

We thank Bryan Tripp
for editing this paper,
contributing to Nengo
and the Nengo OpenCL simulator
and for creating Nengo 1.4,
which has been indispensable
for the past six years,
and will continue
to be used for many years to come.
We thank Peter Blouw and Brent Komer,
who have contributed
to Nengo by providing examples,
unit tests, and bugfixes.
Finally, we thank the two reviewers
who have provided constructive
feedback on this paper.

\paragraph{Funding\textcolon}
NSERC Discovery, NSERC Graduate Fellowships,
NSERC Banting Fellowship, ONR (N000141310419)
and AFOSR (FA8655-13-1-3084).

\bibliographystyle{frontiersinSCNS&ENG}
\bibliography{nengo}

\clearpage

\begin{figure}
 \textbf{\refstepcounter{figure}\label{fig:nef} Figure \arabic{figure}.}{
   Summary of the three principles of the Neural Engineering Framework
   (NEF). \textbf{(A)} By the representation principle, signals are encoded
   in neural populations based on the \textit{tuning curve}
   of each neuron (top). The tuning curve describes
   how active a neuron is given some input signal.
   If we drive the eight neurons in the top panel
   with the input signal in the middle panel,
   we see the spike trains in the bottom panel.
   \textbf{(B)} By the representation principle,
   the spiking activity of a neural population
   can be decoded to recover the original input signal,
   or some transformation of that input signal.
   First, the firing pattern shown in the top panel
   is filtered with a decaying exponential filter (middle panel).
   The filtered activity is then summed together
   with a set of weights that approximates
   the input signal (bottom panel, green)
   and the cosine of the input signal (bottom panel, purple).
   \textbf{(C)} A sine wave is encoded by population A (top panel);
   the negative of that signal is projected
   to population B (middle panel)
   and the square of that signal is projected
   to population C (bottom panel).
   By the transformation principle,
   populations of neurons can send signals
   to another population by decoding
   the desired function from the first population
   and then encoding the decoded estimate
   into the second population.
   These two steps can be combined into a single step
   by calculating a set of weights
   that describe the strength of the connections
   between each neuron in the first population
   and each neuron in the second population.
   \textbf{(D)} A neurally implemented dynamical system
   has negative feedback across its two dimensions,
   resulting in a harmonic oscillator.
   The oscillator is plotted across time (top)
   and in state space (bottom).
   By the dynamcs principle,
   signals being represented by population of neurons
   can be thought of as state variables in a dynamical system.}
\end{figure}

\begin{figure}
 \textbf{\refstepcounter{figure}\label{fig:sim} Figure \arabic{figure}.}{
   Detailed breakdown of the Nengo reference simulator
   running a simple model for two timesteps.
   \textbf{(A)} Code describing the model being simulated.
   It consists of ensemble A projecting the sine of its
   encoded signal to ensemble B,
   which is recurrently connected.
   \textbf{(B)} Diagram depicting the model being simulated.
   \textbf{(C)} A detailed diagram of how the reference simulator
   organizes this model. Signals (blue) represent the values
   tracked in the simulation.
   Operators (red) represent the computations done on signals.
   Signals can be grouped as low-level neural signals
   that are used to compute the nonlinear functions
   underlying neuron models,
   and high-level NEF signals that are used to
   drive neurons and track the signals
   that the neurons are representing.
   The operators that implement the decoding
   and encoding steps map between
   the low-level neural signals
   and the high-level NEF signals.
   \textbf{(D)} The signals tracked at the low level
   can be interpreted as a model
   commonly seen in computational neuroscience literature;
   a population of leaky integrate-and-fire neurons
   is driven by some time-varying input current, $J(t)$.
   These neurons project to a population
   of recurrently connected neurons.
   The connection weights between the two populations,
   and from the second population to itself,
   can be computed by the NEF's transformation
   principle, bypassing the need for
   the high-level NEF signals
   used by the reference simulator
   for speed and data collection purposes.
   \textbf{(E)} The signals tracked at the high level
   can be interpreted as a dynamical system.
   State variable A simply represents its input,
   and passes its state to a sine function
   which becomes the input to B.
   State variable B is a simple linear system
   that can be described with the typical
   $\dot{x}(t) = A x(t) + B u(t)$ equation.
   These dynamical systems can be simulated
   directly, without the use of spiking neurons,
   in order to quickly analyze system behavior,
   if desired.}
\end{figure}

\begin{figure}
 \textbf{\refstepcounter{figure}\label{fig:comm-channel}
   Figure \arabic{figure}.}{
   A communication channel implemented with Nengo.
   \textbf{(A)} Diagram depicting the model. Ensemble A
   projects its encoded signal to ensemble B unchanged.
   \textbf{(B)} Nengo code to build and simulate the model
   for 1 second.
   \textbf{(C)} The results of the simulation.
   The input signal (top panel) is white noise limited to 0 to 5 Hz.
   The signal is well represented by both ensemble A (middle panel)
   and ensemble B (bottom panel) despite the neural firing patterns
   (underlaid in middle and bottom panels) being different.}
\end{figure}


\begin{figure}
 \textbf{\refstepcounter{figure}\label{fig:pynn} Figure
   \arabic{figure}.}{
   Implementation of the communication channel (left) in Nengo
   and (right) in PyNN. Solving for decoding weights
   takes approximately 40 lines of code,
   which are not included in this figure.}
\end{figure}

\begin{figure}
 \textbf{\refstepcounter{figure}\label{fig:lorenz} Figure \arabic{figure}.}{
   A Lorenz attractor implemented with Nengo.
   \textbf{(A)} Nengo code to build and simulate the model
   for 6 seconds.
   \textbf{(B)} Diagram depicting the model. The state ensemble
   is recurrently connected with a complex function
   implementing the dynamics of the Lorenz attractor.
   Note that this population does not receive
   any input that might drive its initial value;
   instead, the initial value is determined by
   the baseline firing of the 2000 leaky integrate-and-fire
   neurons that make up the state ensemble.
   \textbf{(C)} The trajectory that the state ensemble takes
   in its three-dimensional state space.
   For the parameters chosen, the trajectory takes
   the well-known butterfly shape.
   \textbf{(D)} The state vector plotted over time.
   \textbf{(E)} The spikes emitted by a random sample of 25
   neurons from the state ensemble.
   Some neurons fire uniformly across the 6 second simulation,
   but most change depending on the state being tracked
   due to the recurrent connection.}
\end{figure}

\begin{figure}
 \textbf{\refstepcounter{figure}\label{fig:cconv} Figure
   \arabic{figure}.}{
   Circular convolution implemented with Nengo.
   \textbf{(A)} Nengo code to build and simulate the model
   for 0.2 seconds.
   \textbf{(B)} Diagram depicting the model.
   The input vectors, A and B, represent four-dimensional vectors
   which are mapped onto six ensembles within the
   Circular Convolution network through
   complicated transformation matrices
   that implement a discrete Fourier transform.
   Each ensemble within the network represents a
   two-dimensional vector. The product of the two dimensions
   is projected through another complicated transformation matrix
   that implements the inverse discrete Fourier transform,
   computing the final four-dimensional result.
   Note that the complicated parts of the model
   are contained within the network;
   the number of ensembles and the transform matrices shown
   are automatically generated by the network depending on
   the dimensionality of the input vectors.
   \textbf{(C)} The result of the simulation.
   Straight horizontal lines represent
   the target values that each ensemble
   should represent.
   Wavy lines represent the decoded values
   for each dimension represented by the
   A, B, and Result ensembles
   (top, middle and bottom panels, respectively).
   The ensembles represent the correct values,
   after a startup transient of less than 0.1 seconds.}
\end{figure}

\begin{figure}
 \textbf{
   \refstepcounter{figure}\label{fig:benchmarks}
   Figure \arabic{figure}.}{
   Benchmark results for several simulators
   on the example models described
   in Section~\ref{sec:examples}.
   In (A) and (B), all of the simulators
   except the Nengo OpenCL simulator
   were run on an Intel Core i7-965.
   Nengo 1.4 used all 4 cores of this processor;
   all other simulator used only 1 core.
   The Nengo OpenCL simulator
   was run on an NVidia GTX280 GPU.
   \textbf{(A)} Benchmark results from simulating
   the communication channel for 10 simulated seconds
   at a 1 millisecond timestep.
   For all model sizes, Nengo simulators
   are faster than Nengo 1.4,
   which is significantly faster than NEURON and NEST,
   which are significantly faster than Brian.
   The full Brian results are not shown;
   for the largest model,
   the Brian simulation takes $\sim$768 seconds.
   \textbf{(B)} Benchmark results from simulating
   the Lorenz attractor for 10 simulated seconds
   at a 1 millisecond timestep.
   For most model sizes,
   the results are the same as (A),
   except that NEURON is notably faster.
   The full results for Brian and NEST are not shown;
   for the largest model, simulations in Brian and NEST
   take $\sim$1467 and $\sim$601 seconds, respectively.
   \textbf{(C)} Benchmark results from simulating
   circular convolution for 1 simulated second
   at a 1 millisecond timestep.
   For the blue lines, the simulator used
   was the Nengo OpenCL simulator.
   The CPU used for Nengo 1.4 and
   the Nengo reference simulator
   was an Intel Core i7-3770;
   all 4 cores were used by Nengo 1.4,
   while Nengo's reference simulator
   only used one core.
   For all model sizes,
   the OpenCL simulator is faster
   than the Nengo 1.4 simulator,
   which is faster than the Nengo
   reference simulator.
   The reference simulator was only run
   up to 50 dimensions.
   The full results for Nengo 1.4
   are not shown; for the largest model,
   simulation with Nengo 1.4
   takes $\sim$45 seconds.}
\end{figure}

\end{document}