Unexpected compression rate on LZ4 filter

[Willian-Zhang]

Using 32004 (LZ4) filter on obviously-compressible (OC) data not properly compressing data

Expected Behavior

h5ls -vr shows very large utilization rate on OC data (20227.16% on Reproduction Section result)

Current Behavior

h5ls -vr shows 100.00% utilization on OC data

Possible Solution

#120

Steps to Reproduce

import h5py
import numpy as np
import hdf5plugin

nda = np.full(2**20, 0.233)
f = h5py.File("data/2.hdf")
f.create_dataset(name="nda", data=nda, **hdf5plugin.LZ4())
f.close()
!h5ls -vr data/2.hdf
Opened "data/1.hdf" with sec2 driver.
/                        Group
    Location:  1:96
    Links:     1
/nda                     Dataset {1048576/1048576}
    Location:  1:800
    Links:     1
    Chunks:    {4096} 32768 bytes
    Storage:   8388608 logical bytes, 8388608 allocated bytes, 100.00% utilization
    Filter-0:  HDF5 lz4 filter; see http://www.hdfgroup.org/services/contributions.html-32004 OPT {0}
    Type:      native double

Context (Environment)

pip list | grep -e h5py -e numpy -e hdf
h5py                2.10.0
hdf5plugin          3.0.0
numpy               1.19.2

Detailed Description

problem call path

LZ4_compress_default(rpos, roBuf+4, blockSize, nBlocks*4);
-> LZ4_compress_fast(rpos, roBuf+4, blockSize, nBlocks*4, 1);
-> ...

fixed path

LZ4_compress(rpos, roBuf+4, blockSize);
-> LZ4_compress_default((rpos, roBuf+4, blockSize, LZ4_compressBound(inputSize));
-> LZ4_compress_default((rpos, roBuf+4, blockSize, 
          ((unsigned)(blockSize) > (unsigned)0x7E000000   /* 2 113 929 216 bytes */ ? 
                  0 : 
                  (blockSize) + ((blockSize)/255) + 16)
-> ...
);

Possible Implementation

#120

[t20100]

Thanks for the report!

I reported this issue upstream: nexusformat/HDF5-External-Filter-Plugins#16 since hdf5plugin is only packaging it, the reference implementation being there (see https://confluence.hdfgroup.org/display/support/Filters#Filters-32004).