Use numcodecs #139
Use numcodecs #139
Conversation
zarr/core.py
Outdated
| @@ -506,8 +506,8 @@ def __setitem__(self, item, value): | |||
| >>> z = zarr.zeros(100000000, chunks=1000000, dtype='i4') | |||
| >>> z | |||
| Array((100000000,), int32, chunks=(1000000,), order=C) | |||
| nbytes: 381.5M; nbytes_stored: 301; ratio: 1328903.7; initialized: 0/100 | |||
| compressor: Blosc(cname='lz4', clevel=5, shuffle=1) | |||
| nbytes: 381.5M; nbytes_stored: 325; ratio: 1230769.2; initialized: 0/100 | |||
zarr/core.py
Outdated
| @@ -528,8 +528,8 @@ def __setitem__(self, item, value): | |||
| >>> z = zarr.zeros((10000, 10000), chunks=(1000, 1000), dtype='i4') | |||
| >>> z | |||
| Array((10000, 10000), int32, chunks=(1000, 1000), order=C) | |||
| nbytes: 381.5M; nbytes_stored: 323; ratio: 1238390.1; initialized: 0/100 | |||
| compressor: Blosc(cname='lz4', clevel=5, shuffle=1) | |||
| nbytes: 381.5M; nbytes_stored: 347; ratio: 1152737.8; initialized: 0/100 | |||
zarr/core.py
Outdated
| store: dict | ||
| >>> z.append(np.vstack([a, a]), axis=1) | ||
| (20000, 2000) | ||
| >>> z | ||
| Array((20000, 2000), int32, chunks=(1000, 100), order=C) | ||
| nbytes: 152.6M; nbytes_stored: 7.5M; ratio: 20.3; initialized: 400/400 | ||
| compressor: Blosc(cname='lz4', clevel=5, shuffle=1) | ||
| compressor: Blosc(cname='lz4', clevel=5, shuffle=SHUFFLE, blocksize=0) | ||
| store: dict |
There was a problem hiding this comment.
Simplify these examples output in line with tutorial.
|
cc @jakirkham |
|
Do we already have all the codecs from Zarr in Numcodecs or do a few still need to transition? |
|
All codecs are ported except Quantize which I forgot (https://github.com/alimanfoo/numcodecs/issues/28). |
|
I've upgraded the numcodecs dependency to 0.2.0, which now includes Quantize and aliases the Zlib codec to 'gzip' for compatibility with h5py. So now back to full feature parity with Zarr when codecs were bundled. I still want to create a data fixture before merging this, as a way to verify that this PR retains compatibility with data saved with the previous version. I'll do that then rebase this PR. |
alimanfoo
force-pushed
the
use-numcodecs
branch
from
aa95ca7
to
deff4c1
Compare
|
I've rebased this on master with the new data fixture, all tests pass which gives some confidence that switching to use numcodecs will not break ability to read any previously written data. |
alimanfoo
added
enhancement
This PR addresses #74 by removing the
zarr.codecsmodule and usingnumcodecsinstead, which is now a dependency.A couple of issues came up from initial work that need to be resolved upstream: https://github.com/alimanfoo/numcodecs/issues/28, https://github.com/alimanfoo/numcodecs/issues/29.
Also I really should create a data fixture (#138) prior to these changes, then rebase and run tests, to confirm that data created prior to this PR can still be read afterwards.
TODO: