Adding experimental "chunked compression"#172
Merged
ptahmose merged 91 commits intoJul 10, 2026
Conversation
Refactor bitmap creation logic to modularize decoding from compressed data, introducing new internal helpers and a public API (CreateBitmapFromSubBlockData) for direct bitmap creation from raw data and metadata. Update unit test CMake integration to use gtest_discover_tests when not cross-compiling. Switch CLI11 include to single-header. Improves modularity, testability, and API flexibility.
Introduce AttachmentStatistics struct and GetAttachmentStatistics() method to IAttachmentRepository and its implementations. Provide attachment count retrieval in CCZIReader, CCziReaderWriter, and CCziAttachmentsDirectory. Update headers and source files to support querying attachment statistics.
Changed Codecov flags from ${{matrix.OS}} to ${{matrix.os}} to resolve a case-sensitivity issue with the matrix variable in the workflow configuration.
Bump project version to 0.68.0 in CMakeLists.txt. Add version_history.md entry for v0.68.0, highlighting improved CMake packaging, better downstream build integration, and minor API additions (see PR ZEISS#170).
Refactored switch statement formatting for clarity. Added null checks for options pointer in JpgXr and Zstd0 cases, defaulting to true for mismatch handling if options is null.
Add preprocessor checks to select appropriate bitmap copy or conversion routines depending on host endianness and pixel type. Ensures correct pixel data handling on both big-endian and little-endian systems.
Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com>
Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com>
The cmake.yml workflow was updated to set -DLIBCZI_BUILD_UNITTESTS=OFF, preventing unit tests from being built or run during the build process. This change may improve build speed or reflect that tests are not required in this environment.
Introduced chunked compression and decompression with support for Zstd and LZ4 codecs. Added new source files for chunked compression logic, header creation/parsing, and a dedicated decoder. Implemented shared utilities for argument validation. The chunked header format is extensible and supports optional preprocessing (hi-lo byte packing). Updated CMakeLists.txt to include new files. All new code is LGPL-3.0-or-later licensed.
Comprehensive tests for chunked compression were added: - New `test_chunkedcompression.cpp` covers encoding/decoding for Gray8, Gray16, Bgr24, Bgr48, roundtrip tests, and hi-lo-byte-packing for both Zstd and LZ4 codecs. - Header parsing, creation, and error handling tests added to `test_ZstdCompress.cpp`. - Compression option parsing tests added to `test_Utilities.cpp`. - `test_reader.cpp` extended with helpers and tests for handling chunked-compressed subblocks with size mismatches and resolution protocol. - CMake updated to include new test file. These changes improve test coverage for chunked compression, edge cases, and error handling.
Codecov Report❌ Patch coverage is Additional details and impacted files@@ Coverage Diff @@
## main #172 +/- ##
==========================================
+ Coverage 66.63% 67.91% +1.27%
==========================================
Files 96 99 +3
Lines 12402 13467 +1065
==========================================
+ Hits 8264 9146 +882
- Misses 4138 4321 +183
Flags with carried forward coverage won't be shown. Click here to find out more. ☔ View full report in Codecov by Harness. 🚀 New features to boost your workflow:
|
Replaced uint32_t with int32_t for CHUNKEDCOMPRESSION_CODEC CompressParameter to ensure type consistency. Added missing "inc_libCZI_Config.h" include and cleaned up include formatting in decoder_chunkedcompression.cpp. No functional changes.
Contributor
There was a problem hiding this comment.
Pull request overview
This PR introduces an experimental “chunked compression” format (encoding + decoding) and wires it through libCZI’s codec selection and bitmap-creation pipeline, alongside a couple of API additions and CMake/test infrastructure updates.
Changes:
- Add chunked-compression encoder/decoder (zstd-based, optional LZ4 codec) and expose it via
CompressionMode::ChunkedExtensibleandImageDecoderType::ChunkedCompression. - Add new public APIs:
CreateBitmapFromSubBlockData(...)andIAttachmentRepository::GetAttachmentStatistics(). - Improve CTest/GTest integration (
gtest_discover_tests), update CLI11, and extend CI/CMake configuration for the new functionality.
Reviewed changes
Copilot reviewed 40 out of 40 changed files in this pull request and generated 11 comments.
Show a summary per file
| File | Description |
|---|---|
| Src/libCZIAPI_UnitTests/CMakeLists.txt | Switch unit test registration to gtest_discover_tests (with cross-compile fallback). |
| Src/libCZI/zstdCompress.cpp | Refactor shared compression argument validation into CompressionUtilities. |
| Src/libCZI/utilities.h | Add parsing helpers and Utilities::ContainsToken declaration. |
| Src/libCZI/utilities.cpp | Implement new parsing helpers and Utilities::ContainsToken. |
| Src/libCZI/libCZI.h | Add new bitmap creation API and new attachment statistics API surface; extend bitmap options for chunked mismatch handling. |
| Src/libCZI/libCZI_Utilities.h | Add string keys for chunked compression options. |
| Src/libCZI/libCZI_Utilities.cpp | Extend compression-option parsing to support chunked compression and parameters. |
| Src/libCZI/libCZI_Site.h | Add ImageDecoderType::ChunkedCompression. |
| Src/libCZI/libCZI_Site.cpp | Lazily instantiate and return the chunked-compression decoder. |
| Src/libCZI/libCZI_Pixels.h | Add CompressionMode::ChunkedExtensible. |
| Src/libCZI/libCZI_Config.h.in | Add build-time macro for experimental chunked-compression availability. |
| Src/libCZI/libCZI_compress.h | Add chunked compression APIs and ChunkedCompressionHeaderHelper. |
| Src/libCZI/DimCoordinate.cpp | Replace local int parsing with shared Utilities::TryParseInt32. |
| Src/libCZI/decoder_zstd.cpp | Use shared Utilities::ContainsToken for decoder options. |
| Src/libCZI/decoder_chunkedcompression.h | New: chunked-compression decoder interface. |
| Src/libCZI/decoder_chunkedcompression.cpp | New: chunked-compression decoding + size mismatch resolution. |
| Src/libCZI/CziUtils.cpp | Map CompressionMode::ChunkedExtensible to/from integer values. |
| Src/libCZI/CziReaderWriter.h | Add GetAttachmentStatistics() implementation hook. |
| Src/libCZI/CziReaderWriter.cpp | Implement GetAttachmentStatistics() for reader/writer. |
| Src/libCZI/CZIReader.h | Add GetAttachmentStatistics() to reader. |
| Src/libCZI/CZIReader.cpp | Implement GetAttachmentStatistics() for reader. |
| Src/libCZI/CziAttachmentsDirectory.h | Add GetAttachmentCount() helper. |
| Src/libCZI/CziAttachmentsDirectory.cpp | Implement GetAttachmentCount(). |
| Src/libCZI/CreateBitmap.cpp | Add chunked decode path; add new CreateBitmapFromSubBlockData(...) API. |
| Src/libCZI/compresscommon.h | New: common compression argument validation utilities. |
| Src/libCZI/compresscommon.cpp | New: implementation of compression argument validation utilities. |
| Src/libCZI/CMakeLists.txt | Add new sources; add optional LZ4 FetchContent/external linkage; define availability macro. |
| Src/libCZI/chunkedCompress.cpp | New: chunked compression encoder + header creation/parsing utilities. |
| Src/libCZI_UnitTests/test_ZstdCompress.cpp | Add header-walk/parse/create tests for chunked header helper. |
| Src/libCZI_UnitTests/test_Utilities.cpp | Add tests for parsing chunked compression options. |
| Src/libCZI_UnitTests/test_readerwriter.cpp | Add tests for attachment statistics API and not-operational behavior. |
| Src/libCZI_UnitTests/test_reader.cpp | Add tests for CreateBitmapFromSubBlockData and chunked mismatch handling. |
| Src/libCZI_UnitTests/test_chunkedcompression.cpp | New: end-to-end tests for chunked compress/decode (zstd + LZ4). |
| Src/libCZI_UnitTests/CMakeLists.txt | Add new test file; switch to gtest_discover_tests (with cross-compile fallback). |
| Src/CZICmd/cmdlineoptions.cpp | Update CLI11 include usage. |
| Src/CZICmd/CMakeLists.txt | Bump CLI11 dependency version. |
| docs/source/pages/version_history.md | Add version-history entry describing this PR. |
| CMakeLists.txt | Add experimental build options; bump project version. |
| cmake/patch_cmake_minimum_version.cmake | New: helper to patch FetchContent deps’ cmake_minimum_required() for CMake 4.x warnings. |
| .github/workflows/cmake.yml | Adjust workflow flags casing and disable unit tests for arm64-windows-static build. |
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
Introduced LZ4 compression as an optional build feature via CMake. Updated CI pipeline to install LZ4 dependencies on all platforms. Replaced the experimental chunked compression macro with a new LZ4 availability macro. Updated source and config files to use the new macro and enable LZ4-based compression and decompression when available.
Changed liblz4-dev installation from vcpkg to apt-get in cmake.yml. Now, only azure-storage-blobs-cpp and azure-identity-cpp are installed via vcpkg, improving package management consistency.
Replaced apt-get installation of liblz4-dev with vcpkg installation of lz4, grouping it with other Azure SDK dependencies for consistency.
Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com>
Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com>
Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com>
Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com>
Unknown chunk IDs in the compression header now trigger an invalid_argument exception instead of being ignored. This change aligns with documentation and includes comments and a TODO regarding future forward compatibility.
….com/ptahmose/libczi-zeiss into jbl/chunked_compression_experimental
Added detailed documentation for the experimental chunked compression mode, including a new page with format and usage details. Updated build and API docs to describe enabling and using the feature. Adjusted Doxyfile to define the feature macro when enabled. Updated the documentation index to reference the new content.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Description
This PR adds experimental support for a new chunked CZI subblock compression format.
The new format stores compressed subblock pixel data as a compact chunked header followed by independently compressed data chunks. It is exposed as
CompressionMode::ChunkedExtensibleand currently supports Zstd and LZ4 as per-chunk codecs. The format also records per-chunk compressed/decompressed sizes and optional Hi/Lo-byte packing preprocessing information.The implementation includes:
CreateBitmapFromSubBlockand reader/accessor workflowschunked:option syntaxThis feature is intentionally experimental. It is disabled by default and only available when
LIBCZI_BUILD_EXPERIMENTAL_CHUNKED_COMPRESSIONis enabled, or through the global experimental policyLIBCZI_BUILD_ENABLE_EXPERIMENTAL_FUNCTIONALITY=ON.Type of change
How Has This Been Tested?
Test coverage was added for the new experimental chunked-compression path, including:
CompressionMode::ChunkedExtensibleValidated through the libCZI unit-test suite.
Checklist: