Fast numerical array expression evaluator for Python, NumPy, PyTables, pandas, bcolz and more
Python C++ CMake C PowerShell Batchfile
Clone or download
Fetching latest commit…
Cannot retrieve the latest commit at this time.
Permalink
Failed to load latest commit information.
LICENSES When on AMD/Intel platforms, copies for unaligned arrays are disabled. Jun 23, 2009
appveyor Change Appveyor to use official NumPy wheels for Windows Apr 4, 2017
bench Corrected an important performance regression when using VML Jul 15, 2016
doc Update docs to discuss NUMEXPR_MAX_THREADS Apr 30, 2018
numexpr Post 2.6.8 release actions Aug 19, 2018
.gitignore Make test module imports lazy in base module Jul 19, 2018
.hgeol Added .hgeol file Jan 23, 2012
.mailmap more mappings for Mark Nov 30, 2013
.travis.yml Fail build on test fail/error Sep 30, 2017
ANNOUNCE.rst Post 2.6.8 release actions Aug 19, 2018
AUTHORS.txt Getting ready for release 2.6.3 Sep 14, 2017
CMakeLists.txt Split apart interpreter.cpp into more manageable chunks Mar 1, 2012
FindNumPy.cmake Sometimes the length of NUMPY_VALUES is longer than 2. Closes #104 Aug 29, 2013
FindPythonLibsNew.cmake Updates to the custom cmake Find files Jun 5, 2013
INSTALL.rst Moved the resttext files to .rst extension Jan 25, 2014
LICENSE.txt Added licence boilerplates with proper copyright information. Solves #62 Nov 27, 2011
MANIFEST.in Added numexpr/win32 directory explicitly in MANIFEST.in Apr 7, 2014
README.rst Post 2.6.7 release actions done Aug 12, 2018
RELEASE_NOTES.rst Post 2.6.8 release actions Aug 19, 2018
RELEASING.txt Fix numpy version check by using `distutils.version.StrictVersion` Aug 11, 2018
appveyor.yml Fail build on test fail/error Sep 30, 2017
requirements.txt Update NumPy requirement to 1.7 Apr 4, 2017
setup.py Remove debugging output for MLK test Jul 16, 2018
site.cfg.example Getting ready for release v2.6.6 Jul 16, 2018

README.rst

NumExpr: Fast numerical expression evaluator for NumPy

Author: David M. Cooke, Francesc Alted and others
Contact: faltet@gmail.com
URL:https://github.com/pydata/numexpr
Documentation:http://numexpr.readthedocs.io/en/latest/
Travis CI:travis
Appveyor:appveyor
PyPi:version
readthedocs:docs

What is NumExpr?

NumExpr is a fast numerical expression evaluator for NumPy. With it, expressions that operate on arrays (like '3*a+4*b') are accelerated and use less memory than doing the same calculation in Python.

In addition, its multi-threaded capabilities can make use of all your cores -- which generally results in substantial performance scaling compared to NumPy.

Last but not least, numexpr can make use of Intel's VML (Vector Math Library, normally integrated in its Math Kernel Library, or MKL). This allows further acceleration of transcendent expressions.

How NumExpr achieves high performance

The main reason why NumExpr achieves better performance than NumPy is that it avoids allocating memory for intermediate results. This results in better cache utilization and reduces memory access in general. Due to this, NumExpr works best with large arrays.

NumExpr parses expressions into its own op-codes that are then used by an integrated computing virtual machine. The array operands are split into small chunks that easily fit in the cache of the CPU and passed to the virtual machine. The virtual machine then applies the operations on each chunk. It's worth noting that all temporaries and constants in the expression are also chunked. Chunks are distributed among the available cores of the CPU, resulting in highly parallelized code execution.

The result is that NumExpr can get the most of your machine computing capabilities for array-wise computations. Common speed-ups with regard to NumPy are usually between 0.95x (for very simple expressions like 'a + 1') and 4x (for relatively complex ones like 'a*b-4.1*a > 2.5*b'), although much higher speed-ups can be achieved for some functions and complex math operations (up to 15x in some cases).

NumExpr performs best on matrices that are too large to fit in L1 CPU cache. In order to get a better idea on the different speed-ups that can be achieved on your platform, run the provided benchmarks.

Usage

>>> import numpy as np
>>> import numexpr as ne

>>> a = np.arange(1e6)   # Choose large arrays for better speedups
>>> b = np.arange(1e6)

>>> ne.evaluate("a + 1")   # a simple expression
array([  1.00000000e+00,   2.00000000e+00,   3.00000000e+00, ...,
         9.99998000e+05,   9.99999000e+05,   1.00000000e+06])

>>> ne.evaluate('a*b-4.1*a > 2.5*b')   # a more complex one
array([False, False, False, ...,  True,  True,  True], dtype=bool)

>>> ne.evaluate("sin(a) + arcsinh(a/b)")   # you can also use functions
array([        NaN,  1.72284457,  1.79067101, ...,  1.09567006,
        0.17523598, -0.09597844])

>>> s = np.array(['abba', 'abbb', 'abbcdef'])
>>> ne.evaluate("'abba' == s")   # string arrays are supported too
array([ True, False, False], dtype=bool)

Documentation

Please see the official documentation at numexpr.readthedocs.io. Included is a user guide, benchmark results, and the reference API.

Authors

Please see AUTHORS.txt.

License

NumExpr is distributed under the MIT license.