Skip to content

Add support for lz4hdf5 codec#576

Merged
MarkRivers merged 9 commits intomasterfrom
add_lz4hdf5
Apr 11, 2026
Merged

Add support for lz4hdf5 codec#576
MarkRivers merged 9 commits intomasterfrom
add_lz4hdf5

Conversation

@MarkRivers
Copy link
Copy Markdown
Member

This adds support for the lz4hdf5 codec to NDPluginCodec and NDFileHDF5. The codec itself is in ADSupport. There are also changes to ADEiger to use this new codec for Stream2 LZ4 compression, and to ADViewers to display arrays with this codec in ImageJ.

@MarkRivers MarkRivers requested review from jwlodek and tguruswamy April 3, 2026 01:37
Copy link
Copy Markdown
Member

@ericonr ericonr left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Isn't this missing the changes to

else if (pArray->codec.name == codecName[NDCODEC_LZ4]) {
to properly pass the data through to the HDF5 file without inserting any metadata?

Comment thread ADApp/pluginSrc/NDPluginCodec.cpp Outdated

NDArrayInfo_t info;
input->getInfo(&info);
int outputSize = LZ4_compressBound((int)info.totalBytes);
Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Should we add our own compress_bound_lz4hdf5 function in ADSupport? The checking performed by compress_lz4hdf5 is reasonably complex and I don't think it should be repeated across the code base.

As is, I believe this size would cause compress_lz4hdf5 below to error out.

Comment thread ADApp/Db/NDCodec.template
Comment on lines -45 to +48
field(FRST, "BSLZ4")
field(FRST, "LZ4HDF5")
field(FRVL, "4")
field(FVST, "BSLZ4")
field(FVVL, "5")
Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think this change in order would be nice to at least document in release notes, since I believe it could break someone's autosave.

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I have added this to RELEASE.md

@MarkRivers
Copy link
Copy Markdown
Member Author

Isn't this missing the changes to
ADCore/ADApp/pluginSrc/NDFileHDF5Dataset.cpp to properly pass the data through to the HDF5 file without inserting any metadata?

No. The codecName for lz4hdf5 is NDCODEC_LZ4HDF5, not NDCODEC_LZ4, and there is no special treatment required for that codec in NDFileHDF5Dataset.cpp. I have tested this. Here is the output of h5dump --header -p on a file saving NDArrays compressed with lz4hdf5 with a block size of 65000. This was generated with ADSimDetector and NDPluginCodec.

            DATATYPE  H5T_STD_U8LE
            DATASPACE  SIMPLE { ( 100, 1024, 1024 ) / ( 100, 1024, 1024 ) }
            STORAGE_LAYOUT {
               CHUNKED ( 1, 1024, 1024 )
               SIZE 1545960 (67.827:1 COMPRESSION)
            }
            FILTERS {
               USER_DEFINED_FILTER {
                  FILTER_ID 32004
                  COMMENT HDF5 lz4 filter; see http://www.hdfgroup.org/services/contributions.html
                  PARAMS { 0 0 }
               }
            }

@MarkRivers MarkRivers merged commit f7cf89c into master Apr 11, 2026
16 checks passed
@ericonr ericonr deleted the add_lz4hdf5 branch April 13, 2026 13:19
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants