feat: Adds Level 7 (Ultra) compression to wrappers#317
Merged
Conversation
Introduces a new compression level, `LevelUltra`, which represents the highest density setting. This level offers maximum compression by using Huffman-coded literals and sequence tokens with a deep parse, resulting in the slowest compression but the best ratio. The addition of this level required updates to constants, documentation, and test cases across various wrappers (Go, Node.js, Python, Rust, WASM) to reflect the expanded range of compression levels from 1-6 to 1-7. The `max_level` function and associated documentation have been updated accordingly.
Introduces a new maximum compression level, `ZXC_LEVEL_ULTRA` (7), by updating documentation and internal logic to support this expanded range. The changes include: - Updating `zxc_min_level` and `zxc_max_level` documentation to reflect the new maximum level. - Modifying comments in `zxc_compress_block` and `zxc_block_workspace_size` to indicate the new level range (1-7). - Adjusting the initialization logic in `zxc_dispatch.c` to correctly handle the allocation of optimal parser scratch for the new highest levels. - Updating the `zxc_compress_opts_t` struct documentation in `zxc_opts.h` to show the new level range. - Ensuring the `zxc_cstream_create` function correctly initializes and handles options, including disabling dictionary options for the stream-based API.
This commit introduces several improvements to the Go wrappers for zxc: - **New Error Types:** Defines and maps new error codes related to dictionary usage (`ErrDictRequired`, `ErrDictMismatch`, `ErrDictTooLarge`), Huffman table validation (`ErrBadHufTable`), and unsupported dictionary operations for the push streaming API (`ErrDictUnsupported`). - **Option Inheritance:** Implements a mechanism for `Cctx` (compression context) to store creation-time options (`level`, `checksum`) and have `CompressBlock` calls inherit them if not explicitly overridden per call. This is managed by new `levelSet` and `checksumSet` flags in the `options` struct. - **Footer Plausibility Check:** Adds a check in `Decompress` to validate the decompressed size reported in the archive footer against reasonable limits, preventing panics on malformed data. - **Windows File Handle Duplication:** Refactors `dupFileRead` and `dupFileWrite` on Windows to use `syscall.DuplicateHandle` correctly, ensuring that the Go `*os.File`'s underlying HANDLE is not closed when the duplicate C `FILE*` is closed. This prevents handle leaks. - **Stream API Dictionary Rejection:** Explicitly rejects dictionary options for `NewCStream` and `NewDStream` by returning `ErrDictUnsupported`, as the push stream format does not support dictionary IDs. - **Error Handling Improvements:** Adds checks for `fclose` return values in `CompressFile` and `DecompressFile` to catch potential I/O errors during final output flushing. Also, enhances error reporting in `Writer.Write` for cases where the encoder makes no progress. - **Regression Tests:** Introduces a new test file (`zxc_regression_test.go`) to cover the fixes and improvements made, including option inheritance, crafted footer handling, dictionary rejection in streams, error code mapping, and Huffman table validation.
This commit introduces new error constants for dictionary-related issues (`ERROR_DICT_REQUIRED`, `ERROR_DICT_MISMATCH`, `ERROR_DICT_TOO_LARGE`) and enhances the `ZxcReader` and `ZxcSeekable` implementations. Key changes include: - Exposing new dictionary-related error codes in the Python wrapper. - Modifying `ZxcReader.readinto` to return `None` when no data is available, aligning with `io.RawIOBase` semantics for non-blocking reads. - Improving exception handling in `ZxcSeekable` by stashing and restoring exceptions raised by Python reader callbacks, ensuring that the original traceback is preserved when errors occur during decompression. - Refactoring `seekable_read_at_trampoline` to correctly manage the GIL when calling Python callbacks from potentially different threads. - Adding a regression test for multithreaded decoding with a Python reader. - Updating type hints and documentation for `readinto`.
This commit introduces new error constants for dictionary-related issues (`ERROR_DICT_REQUIRED`, `ERROR_DICT_MISMATCH`, `ERROR_DICT_TOO_LARGE`) to the Node.js wrapper. Additionally, it refines the `decompress` function to correctly slice the output buffer to its actual written size. Previously, it would return a buffer that might contain uninitialized memory past the decompressed data if the provided `size` hint was larger than the actual decompressed size. This change ensures that only the valid decompressed data is returned, preventing potential exposure of stale heap contents. A test case has been added to verify this behavior. The `SeekableWrap::decompressRange` function has also been updated to slice the output buffer, ensuring that only the actual decompressed bytes are returned, similar to the `decompress` function. This prevents returning uninitialized memory.
This commit refactors the `build.rs` script to introduce a `compile_variant` helper function, improving modularity and reducing code duplication when compiling different ZXC variants. It also enhances the handling of the `CARGO_FEATURE_SYSTEM` by allowing a custom library directory to be specified via `ZXC_LIB_DIR`, making it more flexible for users who manage system installations of ZXC. The changes also align the Rust build script's architecture-specific compilation flags with those used in other build systems (CMake, Meson) for consistency.
This commit introduces support for compression level 7 and updates the Emscripten build configuration to include `getTempRet0` in exported runtime methods. It also adjusts the test suite to cover the new compression level and expands the memory allocation for compression options to accommodate larger structures.
…treams This commit introduces several improvements across different wrappers: - **Go Wrappers:** Refactored dictionary training functions (`TrainDict`, `TrainDictHuf`, `TrainDictionary`) to use a shared `pinSamples` helper. This helper leverages `runtime.Pinner` (Go >= 1.21) to pin sample data directly, avoiding unnecessary C-level memory allocations and copies, thus improving performance and reducing memory overhead. The `zxc_block.go` also had a documentation update regarding dictionary usage. - **Emscripten Build:** Removed the unused `_zxc_compress_block_bound` export and simplified the `EXPORTED_RUNTIME_METHODS` in `CMakeLists.txt`. - **Node.js Wrapper:** Replaced dynamic buffer reallocations within `CStreamWrap` and `DStreamWrap` with a reusable staging buffer (`std::vector<uint8_t> stage_`). This reduces allocation overhead for streaming operations, especially when processing many small chunks. The `compress` function also now uses a `std::unique_ptr` for its destination buffer, avoiding the creation of an unnecessary JS Buffer for compression scratch space. - **Python Wrapper:** Updated `Seekable` documentation to reflect zero-copy behavior when opening from bytes. Removed unused `_inbuf` from `ZxcReader` and updated `ZxcWriter` to accept memoryviews directly, optimizing buffer handling. - **Wasm Wrapper:** Optimized `_concatChunks` to return the single chunk directly when only one is present, avoiding an unnecessary copy. Also, zero-filled the `COMPRESS_OPTS_SIZE` and `DECOMPRESS_OPTS_SIZE` structures more efficiently.
Codecov Report✅ All modified and coverable lines are covered by tests. 📢 Thoughts on this report? Let us know! |
Remove an outdated comment in the Emscripten `CMakeLists.txt` regarding ES6 output. Correct the `self` parameter signature in the Python wrapper's `pyzxc_get_decompressed_size` function for consistency with Python C API conventions.
|
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Introduces a new
Level 7(Ultra) compression preset, offering maximum density for the slowest compression, and updates all language wrappers to support it.Enhances various aspects of the wrappers:
Cctxfor consistent level/checksum application, fixes file handle duplication on Windows, adds plausibility checks for decompressed sizes, and enables zero-copy for dictionary training and seekable archive opening (Go 1.21+).readAtcallbacks to prevent reentrant calls, and correctly slices output buffers to prevent exposing uninitialized memory. Updates the Emscripten build to use ES6 modules and correctly export 64-bit return values.Seekablereader callbacks in multi-threaded contexts, correctly propagates callback exceptions, and enables zero-copy forSeekablearchive opening by pinning input buffers. Adds validation for compression levels andblock_sizein streaming APIs.StreamCompressOptionsto allowseekablefiles, addscompress_file_with_optionsanddecompress_file_with_optionsfor more granular control, introduces new dictionary-related error types, and rejects dictionary options in push-streaming APIs.free()methods for contexts and streams, and adds specific validations for dictionary tables andSeekableoffsets.New dictionary error constants are introduced across all wrappers for more precise error reporting. Several internal memory management and allocation strategies are optimized for improved performance and reduced overhead.