Resolved an issue with zarr.creation.array
where dtype was given as None (#80).
Resolved an issue when no compression is used and chunks are stored in memory (#79).
Various minor improvements, including: Group
objects support member access via dot notation (__getattr__
); fixed metadata caching for Array.shape
property and derivatives; added Array.ndim
property; fixed Array.__array__
method arguments; fixed bug in pickling Array
state; fixed bug in pickling ThreadSynchronizer
.
- Group objects now support member deletion via
del
statement (#65). - Added
zarr.storage.TempStore
class for convenience to provide storage via a temporary directory (#59). - Fixed performance issues with
zarr.storage.ZipStore
class (#66). - The Blosc extension has been modified to return bytes instead of array objects from compress and decompress function calls. This should improve compatibility and also provides a small performance increase for compressing high compression ratio data (#55).
- Added
overwrite
keyword argument to array and group creation methods on thezarr.hierarchy.Group
class (#71). - Added
cache_metadata
keyword argument to array creation methods. - The functions
zarr.creation.open_array
andzarr.hierarchy.open_group
now accept any store as first argument (#56).
The bundled Blosc library has been upgraded to version 1.11.1.
Support has been added for organizing arrays into hierarchies via groups. See the tutorial section on tutorial_groups
and the zarr.hierarchy
API docs for more information.
Support has been added for configuring filters to preprocess chunk data prior to compression. See the tutorial section on tutorial_filters
and the zarr.codecs
API docs for more information.
To accommodate support for hierarchies and filters, the Zarr metadata format has been modified. See the spec_v2
for more information. To migrate an array stored using Zarr version 1.x, use the zarr.storage.migrate_1to2
function.
The bundled Blosc library has been upgraded to version 1.11.0.
Thanks to Matthew Rocklin (mrocklin), Stephan Hoyer (shoyer) and Francesc Alted (FrancescAlted) for contributions and comments.
- The bundled Blosc library has been upgraded to version 1.10.0. The 'zstd' internal compression library is now available within Blosc. See the tutorial section on
tutorial_compress
for an example. - When using the Blosc compressor, the default internal compression library is now 'lz4'.
- The default number of internal threads for the Blosc compressor has been increased to a maximum of 8 (previously 4).
- Added convenience functions
zarr.blosc.list_compressors
andzarr.blosc.get_nthreads
.
This release includes a complete re-organization of the code base. The major version number has been bumped to indicate that there have been backwards-incompatible changes to the API and the on-disk storage format. However, Zarr is still in an early stage of development, so please do not take the version number as an indicator of maturity.
The main motivation for re-organizing the code was to create an abstraction layer between the core array logic and data storage (#21). In this release, any object that implements the MutableMapping
interface can be used as an array store. See the tutorial sections on tutorial_persist
and tutorial_tips_storage
, the spec_v1
, and the zarr.storage
module documentation for more information.
Please note also that the file organization and file name conventions used when storing a Zarr array in a directory on the file system have changed. Persistent Zarr arrays created using previous versions of the software will not be compatible with this version. See the zarr.storage
API docs and the spec_v1
for more information.
An abstraction layer has also been created between the core array logic and the code for compressing and decompressing array chunks. This release still bundles the c-blosc library and uses Blosc as the default compressor, however other compressors including zlib, BZ2 and LZMA are also now supported via the Python standard library. New compressors can also be dynamically registered for use with Zarr. See the tutorial sections on tutorial_compress
and tutorial_tips_blosc
, the spec_v1
, and the zarr.compressors
module documentation for more information.
The synchronization code has also been refactored to create a layer of abstraction, enabling Zarr arrays to be used in parallel computations with a number of alternative synchronization methods. For more information see the tutorial section on tutorial_sync
and the zarr.sync
module documentation.
NumPy is no longer a build dependency for the zarr.blosc
Cython extension, so setup.py will run even if NumPy is not already installed, and should automatically install NumPy as a runtime dependency. Manual installation of NumPy prior to installing Zarr is still recommended, however, as the automatic installation of NumPy may fail or be sub-optimal on some platforms.
Some optimizations have been made within the zarr.blosc
extension to avoid unnecessary memory copies, giving a ~10-20% performance improvement for multi-threaded compression operations.
The zarr.blosc
extension now automatically detects whether it is running within a single-threaded or multi-threaded program and adapts its internal behaviour accordingly (#27). There is no need for the user to make any API calls to switch Blosc between contextual and non-contextual (global lock) mode. See also the tutorial section on tutorial_tips_blosc
.
The internal code for managing chunks has been rewritten to be more efficient. Now no state is maintained for chunks outside of the array store, meaning that chunks do not carry any extra memory overhead not accounted for by the store. This negates the need for the "lazy" option present in the previous release, and this has been removed.
The memory layout within chunks can now be set as either "C" (row-major) or "F" (column-major), which can help to provide better compression for some data (#7). See the tutorial section on tutorial_tips_order
for more information.
A bug has been fixed within the __getitem__
and __setitem__
machinery for slicing arrays, to properly handle getting and setting partial slices.
Thanks to Matthew Rocklin (mrocklin), Stephan Hoyer (shoyer), Francesc Alted (FrancescAlted), Anthony Scopatz (scopatz) and Martin Durant (martindurant) for contributions and comments.