Commit
This commit does not belong to any branch on this repository, and may belong to a fork outside of the repository.
archivers/py-bitshuffle: Add py-bitshuffle 0.5.1
Bitshuffle is an algorithm that rearranges typed, binary data for improving compression, as well as a python/C package that implements this algorithm within the Numpy framework. The library can be used along side HDF5 to compress and decompress datasets and is integrated through the dynamically loaded filters framework. Bitshuffle is HDF5 filter number 32008. Algorithmically, Bitshuffle is closely related to HDF5's Shuffle filter except it operates at the bit level instead of the byte level. Arranging a typed data array in to a matrix with the elements as the rows and the bits within the elements as the columns, Bitshuffle "transposes" the matrix, such that all the least-significant-bits are in a row, etc. This does not in itself compress data, only rearranges it for more efficient compression. To perform the actual compression you will need a compression library. Bitshuffle has been designed to be well matched to Marc Lehmann's LZF as well as LZ4 and ZSTD. Note that because Bitshuffle modifies the data at the bit level, sophisticated entropy reducing compression libraries such as GZIP and BZIP are unlikely to achieve significantly better compression than simpler and faster duplicate-string-elimination algorithms such as LZF, LZ4 and ZSTD. Bitshuffle thus includes routines (and HDF5 filter options) to apply LZ4 and ZSTD compression to each block after shuffling.
- Loading branch information
Showing
7 changed files
with
91 additions
and
0 deletions.
There are no files selected for viewing
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,28 @@ | ||
PORTNAME= bitshuffle | ||
PORTVERSION= 0.5.1 | ||
CATEGORIES= archivers python | ||
MASTER_SITES= PYPI | ||
PKGNAMEPREFIX= ${PYTHON_PKGNAMEPREFIX} | ||
|
||
MAINTAINER= sunpoet@FreeBSD.org | ||
COMMENT= Bitshuffle filter for improving typed data compression | ||
WWW= https://github.com/kiyo-masui/bitshuffle | ||
|
||
LICENSE= MIT | ||
LICENSE_FILE= ${WRKSRC}/LICENSE | ||
|
||
BUILD_DEPENDS= ${PYTHON_PKGNAMEPREFIX}h5py>=2.4.0:science/py-h5py@${PY_FLAVOR} \ | ||
${PYTHON_PKGNAMEPREFIX}numpy>=0,1:math/py-numpy@${PY_FLAVOR} \ | ||
${PYTHON_PKGNAMEPREFIX}setuptools>=0.7:devel/py-setuptools@${PY_FLAVOR} \ | ||
${PYTHON_PKGNAMEPREFIX}wheel>=0:devel/py-wheel@${PY_FLAVOR} | ||
RUN_DEPENDS= ${PYTHON_PKGNAMEPREFIX}h5py>=2.4.0:science/py-h5py@${PY_FLAVOR} \ | ||
${PYTHON_PKGNAMEPREFIX}numpy>=1.6.1,1:math/py-numpy@${PY_FLAVOR} \ | ||
${PYTHON_PKGNAMEPREFIX}setuptools>=0.7:devel/py-setuptools@${PY_FLAVOR} | ||
|
||
USES= pkgconfig python | ||
USE_PYTHON= autoplist concurrent cython pep517 | ||
|
||
post-install: | ||
${FIND} ${STAGEDIR}${PYTHON_SITELIBDIR} -name '*.so' -exec ${STRIP_CMD} {} + | ||
|
||
.include <bsd.port.mk> |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,3 @@ | ||
TIMESTAMP = 1710712300 | ||
SHA256 (bitshuffle-0.5.1.tar.gz) = 988f224739aa6858475a4c59172968c7b51cc657d2249580c8f96848708fbae3 | ||
SIZE (bitshuffle-0.5.1.tar.gz) = 229441 |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,14 @@ | ||
--- lzf/lzf_filter.c.orig 2022-11-26 00:17:16 UTC | ||
+++ lzf/lzf_filter.c | ||
@@ -51,11 +51,7 @@ | ||
macro H5_USE_16_API is set | ||
*/ | ||
|
||
-#if H5_VERS_MAJOR == 1 && H5_VERS_MINOR == 8 && (H5_VERS_RELEASE < 3 || !H5_USE_16_API) | ||
#define H5PY_H5Z_NEWCLS 1 | ||
-#else | ||
-#define H5PY_H5Z_NEWCLS 0 | ||
-#endif | ||
|
||
size_t lzf_filter(unsigned flags, size_t cd_nelmts, | ||
const unsigned cd_values[], size_t nbytes, |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,11 @@ | ||
--- pyproject.toml.orig 2022-11-26 00:17:16 UTC | ||
+++ pyproject.toml | ||
@@ -3,7 +3,7 @@ requires = [ | ||
requires = [ | ||
"setuptools>=0.7", | ||
"Cython>=0.19", | ||
- "oldest-supported-numpy", | ||
+ "numpy", | ||
"h5py>=2.4.0", | ||
] | ||
|
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,11 @@ | ||
--- setup.py.orig 2022-11-26 00:17:16 UTC | ||
+++ setup.py | ||
@@ -371,7 +371,7 @@ class build_ext(build_ext_): | ||
else: | ||
openmpflag = "-fopenmp" | ||
archi = platform.machine() | ||
- if archi in ("i386", "x86_64"): | ||
+ if archi in ("amd64", "i386", "x86_64"): | ||
compileflags = COMPILE_FLAGS + ["-march=%s" % self.march] | ||
else: | ||
compileflags = COMPILE_FLAGS + ["-mcpu=%s" % self.march] |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,23 @@ | ||
Bitshuffle is an algorithm that rearranges typed, binary data for improving | ||
compression, as well as a python/C package that implements this algorithm within | ||
the Numpy framework. | ||
|
||
The library can be used along side HDF5 to compress and decompress datasets and | ||
is integrated through the dynamically loaded filters framework. Bitshuffle is | ||
HDF5 filter number 32008. | ||
|
||
Algorithmically, Bitshuffle is closely related to HDF5's Shuffle filter except | ||
it operates at the bit level instead of the byte level. Arranging a typed data | ||
array in to a matrix with the elements as the rows and the bits within the | ||
elements as the columns, Bitshuffle "transposes" the matrix, such that all the | ||
least-significant-bits are in a row, etc. | ||
|
||
This does not in itself compress data, only rearranges it for more efficient | ||
compression. To perform the actual compression you will need a compression | ||
library. Bitshuffle has been designed to be well matched to Marc Lehmann's LZF | ||
as well as LZ4 and ZSTD. Note that because Bitshuffle modifies the data at the | ||
bit level, sophisticated entropy reducing compression libraries such as GZIP and | ||
BZIP are unlikely to achieve significantly better compression than simpler and | ||
faster duplicate-string-elimination algorithms such as LZF, LZ4 and ZSTD. | ||
Bitshuffle thus includes routines (and HDF5 filter options) to apply LZ4 and | ||
ZSTD compression to each block after shuffling. |