This section has been moved to ipython notebook tutorials.
Tutorial on carray objects
This section has been moved to ipython notebook tutorial_carray.
Tutorial on ctable objects
This section has been moved to ipython notebook tutorial_ctable.
Writing bcolz extensions
Did you like bcolz but you couldn't find exactly the functionality you were looking for? You can write an extension and implement complex operations on top of bcolz containers.
Before you start writing your own extension, let's see some examples of real projects made on top of bcolz:
- Bquery: a query and aggregation framework, among other things it
- provides group-by functionality for bcolz containers. See https://github.com/visualfabriq/bquery
- Bdot: provides big dot products (by making your RAM bigger on the
- inside). Supports
matrix . vectorand
matrix . matrixfor most common numpy numeric data types. See https://github.com/tailwind/bdot
Though not a extension itself, it is worth mentioning Dask. Dask plays nicely with bcolz and provides multi-core execution on larger-than-memory datasets using blocked algorithms and task scheduling. See https://github.com/dask/dask.
In addition, bcolz also interacts well with itertools, Pytoolz or Cytoolz too and they might offer you already the amount of performance and functionality you are after.
In the next section we will go through all the steps needed to write your own extension on top of bcolz.
How to use bcolz as part of the infrastructure
Go to the root directory of bcolz, inside
docs/my_package/ you will
find a small extension example.
Before you can run this example you will need to install the following
pip install cython,
pip install numpy and
install bcolz to install these packages. In case you prefer Conda
package management system execute
conda install cython numpy bcolz
and you should be ready to go. See
.. literalinclude:: my_package/requirements.txt :language: python
Once you have those packages installed, change your working directory
docs/my_package/, please see pkg. example and run
python setup.py build_ext --inplace from the terminal, if
everything ran smoothly you should be able to see a binary file
my_extension/example_ext.so next to the
If you have any problems compiling these extensions, please make sure
you have a recent version of bcolz as old versions (pre 0.8) don't
contain the necessary
.pxd file which provides a Cython interface
to the carray Cython module.
setup.py file is where you will need to tell the compiler, the
name of you package, the location of external libraries (in case you
want to use them), compiler directives and so on. See bcolz setup.py as a possible
reference for a more complete example. Along your project grows in
complexity you might be interested in including other options to your
Extension object, e.g. include_dirs to include a list of
directories to search for C/C++ header files your code might be
.. literalinclude:: my_package/setup.py :language: python
.pyx files is going to be the place where Cython code
implementing the extension will be, in the example below the function
will return a sum of all integers inside the carray.
Keep in mind that carrays are great for sequential access, but random access will highly likely trigger decompression of a different chunk for each randomly accessed value.
For more information about Cython visit http://docs.cython.org/index.html
.. literalinclude:: my_package/my_extension/example_ext.pyx :language: python
Let's test our extension:
>>> import bcolz >>> import my_extension.example_ext as my_mod >>> c = bcolz.carray([i for i in range(1000)], dtype='i8') >>> my_mod.my_function(c) 499500