The DyND Library
DyND is an array library for managing structured and semi-structured data, written with C++ as a first-class target and extended to Python with a lightweight binding. It aims to be a cross-language platform for data analysis, by bringing the popularity and flexibility of the Python data science stack to other languages. It is inspired by NumPy, the Python array programming library at the core of the scientific Python stack, but tries to address a number of obstacles encountered by some of NumPy’s users. Examples of these are support for variable-sized strings, missing values, variable-sized (ragged) array dimensions, and versatile tools for creating and manipulating functions that apply generic patterns across containers.
At a high level, the cornerstones of DyND are its type system, array container, and callable (function) objects. These represent the description, storage, and manipulation of dynamic, reinterpretable bytes across languages. At a low level, DyND defines an primitive execution kernel that brings together computation and data in a compact form that can be executed rapidly across array elements. Where DyND begins to shine is in its support for functional composition. For example, NumPy-like broadcasting is not built into every callable, rather it is made available as a functional transformation applied to a scalar callable.
DyND was created by Irwin Zaid and Mark Wiebe. The core team consists of Irwin Zaid, Mark Wiebe, and Ian Henriksen. Others who made important contributions include Phillip Cloud, Michael Droettboom, Stefan Krah, Travis Oliphant, and Andy Terrel. Much of the funding that made this project possible came through Continuum Analytics and DARPA-BAA-12-38, part of XDATA.
We pronounce DyND as "dined", though others refer to it as "dy-n-d". It's not something we're picky about.
This library is actively developed together with its Python bindings. The Python bindings provide a good way to become familiar with the library from a high level perspective. See the github site for the Python bindings.
C++ is a first-class target of the library, the intent is that all features should be easily usable in that language. This approach makes it so that DyND can expose a more uniform interface to C++, Python, and hopefully other languages that eventually get bindings to the core DyND library.
DyND is still experimental, so many of the interfaces provided here will continue to change. That said, feedback and bug reports are greatly appreciated.
History Of DyND
DyND was started as a personal project of Mark Wiebe in September 2011 to begin dabbling in ideas for how a dynamic, multidimensional array library could be structured in C++. See here for the very first commit. Mark was at the University of British Columbia, then joined Continuum Analytics part-time when it was founded in January 2012, and later became full-time in the spring of 2012. He parted ways with Continuum at the end of 2014, joining Thinkbox Software.
During the formation of Continuum, DyND was open-sourced and brought into the company as a part of the Blaze project. Continuum secured funding for the Blaze project through DARPA's XDATA program, giving the project space and time to develop as it needed and providing real data sets and challenge problems to tackle and measure against. Travis Oliphant has told the story of the early days at Continuum, and why it supports DyND, on the NumPy mailing list - see here.
Irwin Zaid joined the project at the beginning of 2014, as its first major outside contributor, while he was a research fellow at Christ Church, University of Oxford. He initially added CUDA support to the library, then played a gradually increasing role in the design and goals of the project. He took over development of DyND in the spring of 2015, and was funded by Continuum from June 2015 until April 2016.
DyND is still experimental. Some of its components are mature while others are severely lacking. New contributors are welcome, so if you have the patience to collaborate on a maturing code base, and enjoy C++, array-oriented, and numeric programming, DyND might be the the open source project you're looking for.
The build system of this library is based on CMake. See the build instructions for details about how to build the library.
DyND requires a C++14 compiler, the minimum versions supported are gcc 4.9, MSVC 2015, and Clang 3.6. The last release to support C++98 was DyND 0.6.6. C++14 brings several things to the library, including a roughly factor of two compile time improvement and generality with variadic templates.
Running The Tests
The tests are built using Google Test. To execute the test suite,