uf-lzma2 1.6.4

A compression level above the maximum (UF2_maxCLevel() = 11) now clamps to the maximum instead of silently falling back to the default level. The one-shot and streaming entry points passed the out-of-range level to UF2_CCtx_setParameter without checking its error, so e.g. level 12 left the context at the default. In-range levels (1-11) are unchanged; make check passes (83883 mutation cases, 0 failures). The same fix is in fast-lzma2 1.1.2.

Full changelog: https://github.com/YadeWira/ultra-fast-lzma2/wiki/Changelog

fast-lzma2 1.1.2

For fast-lzma2 users. A compression level above the maximum (FL2_maxCLevel() = 10) now clamps to the maximum instead of silently falling back to the default level.

The one-shot (FL2_compress*) and streaming (FL2_initCStream) entry points passed an out-of-range level straight to FL2_CCtx_setParameter and did not check its error, so e.g. level 11 left the context at the default level 6 and compressed worse than level 10. Now level 11/12/… clamp to 10. In-range levels (1-10) are unchanged (byte-identical output). Found via zpaq-std's -ma:flzma2:11.

fast-lzma2 1.1.1

For fast-lzma2 users — supersedes 1.1.0, which was missing its decoder fixes.

The decoder fixes 1.1.0 claimed were not actually in its lzma2_dec.c: the change was applied to the working tree but lost before the commit, and the valid-data tests did not catch it (the fixes only affect corrupt input). The zpaq-std project found it with an AddressSanitizer corpus — a heap read overflow in LZMA_decodeReal_3 on damaged blocks. Thanks to them for the catch.

1.1.1 restores the three decoder fixes into lzma2_dec.c:

• No read past the end of a chunk's input (the heap read overflow), via a zero-padded tail copy.
• A chunk with no properties after a dictionary reset is rejected rather than decoded with an uninitialised pb (a fast-lzma2 typo set lc twice).
• A chunk that ends before consuming its declared packed size is rejected rather than desynchronising the stream.

Verified against the reporter's repro corpus under AddressSanitizer: segv_min, segv_real, cut_0 (heap-buffer-overflow READ of size 1) and trunc_a / trunc_min272 (silent mis-decode) are now clean corruption errors. Compressed output is unchanged; valid-data round-trip and cross-version decode are unaffected.

Use this instead of 1.1.0.

fast-lzma2 1.1.0

⚠️ Superseded by fast-lzma2 1.1.1 — do not use

This 1.1.0 tag shipped without the three decoder fixes it lists below: the change to lzma2_dec.c was applied to the working tree but lost before the commit, so damaged input can cause a heap read overflow in LZMA_decodeReal_3 (found by zpaq-std under AddressSanitizer). Use 1.1.1, which restores them. Everything else below still describes this release.

---

fast-lzma2 1.1.0 backports the correctness and robustness fixes that uf-lzma2 found by fuzzing, and adds two compatible build features (ARM64 assembler decoder, macOS Apple Silicon build). The maintained fast-lzma2 lives in the fast-lzma2/ directory of this repository.

uf-lzma2 1.6.3

Build and platform release: uf-lzma2 now builds on macOS (Apple Silicon). The compressed bytes are unchanged from 1.6.2; valid input compresses and decompresses exactly as before.

• On Darwin the shared library links as a .dylib with -install_name (detected from uname -s); ldconfig, absent on macOS, is guarded in install/uninstall. The ARM64 assembler decoder already carried the Mach-O symbol name and runs unchanged.
• CI runs make check on an Apple Silicon runner — the first end-to-end test of the ARM64 assembler decoder under Mach-O. All seven configurations (x86_64, ASan+UBSan, x86 32-bit, single-threaded, ARM64, macOS, Windows x64/x86) pass.
• Intel Macs are not yet supported: the x86_64 assembler decoder uses a MASM-style syntax the Mach-O assembler rejects.
• The repository .gitignore was slimmed from the 317-line Visual Studio/.NET template to a focused list.

Full changelog: https://github.com/YadeWira/ultra-fast-lzma2/wiki/Changelog

uf-lzma2 1.6.2

Decoder-only release from the stability campaign's differential decoder fuzzer. The compressed bytes do not change; valid input compresses and decompresses exactly as in v1.6.1.

Decoding corrupt data

• A corrupt native stream could hold an LZMA chunk whose declared packed size is larger than its real LZMA data, so the range decoder reaches a clean end (code zero at the output limit) with packed bytes still unconsumed. The one-shot and single-threaded streaming decoders then read the next control byte from inside the chunk's leftover bytes, hit a zero there, and stopped reporting success; the multi-threaded streaming decoder, which locates chunks by their packed size, read on into the next chunk. So the four decode paths disagreed on the same corrupt input, each returning success with different output. A completed LZMA chunk must now consume its whole packed size, as the LZMA SDK requires, so it is rejected on every path. Valid multi-chunk streams are unaffected. The same defect is in fast-lzma2.

Full changelog: https://github.com/YadeWira/ultra-fast-lzma2/wiki/Changelog

uf-lzma2 1.6.1

Stability release from a fuzzing and sanitizer campaign — libFuzzer with ASan and UBSan, ThreadSanitizer, injected allocation failures, and a real-file corpus. The compressed bytes do not change; valid input compresses and decompresses exactly as in v1.6.0. Eleven defects were fixed, most inherited from fast-lzma2.

Decoding corrupt data

• The LZMA decoder could read up to 20 bytes past the end of a chunk's input. The limit now leaves that much room and the last bytes are decoded from a zero-padded copy, as the LZMA SDK does; using more than the input holds is a corruption error.
• An LZMA chunk with no properties after a dictionary reset was decoded with properties no chunk had set. A fast-lzma2 typo (p->prop.lc = 2 where pb was meant) left pb out of range, which made the C decoder shift by 190 bits and the assembler decoder index past its tables and crash. The chunk is now rejected and pb starts at 2.

Threading

• UF2POOL_waitAll() could return while a job was still running; it now reads the pool's state under the lock.
• A streaming compression with a timeout could publish and read its results before they were finished.
• The match finder's shared list index is now read atomically, so ThreadSanitizer is clean.

Robustness

• Compression returns an error, instead of crashing, when the match finder's per-thread builders cannot be allocated.
• Every .xz allocation goes through UF2_setAllocator().
• Raw LZMA2 of an empty input no longer writes a stray property byte or one byte past the buffer.
• UF2_compressBuffer() no longer advances a pointer past the end of a small input.
• The match finder's 4-byte repeat probes are read with MEM_read32() instead of an unaligned cast.

Full changelog: https://github.com/YadeWira/ultra-fast-lzma2/wiki/Changelog

uf-lzma2 1.6.0

Streamed .xz that decompresses on several threads

New parameter UF2_p_xzSizedHeaders (off by default).

Until now, a streamed .xz file decompressed on one thread only. Each Block Header went out before the block's data, when neither size was known, so it omitted both, as xz does when it compresses on one thread.

With the parameter set to 1, the encoder holds each block's compressed data in memory until the block ends. It then writes the header, now with both sizes, into room kept before the data. The file decompresses on several threads like one-shot output, here and in xz 5.4 and later.

• APIs: works with every streaming API: UF2_compressStream(), the zero-copy dictionary functions, timeouts and dual buffering.
• Memory: up to one block's compressed size. A block is dictionary size × UF2_p_resetInterval of input; a reset interval of 1 brings it down to one dictionary.
• Flushing: in this mode UF2_flushStream() writes nothing of the open block, since its sizes are not known yet.
• Default output: with the parameter off, streaming output is unchanged, byte for byte.

The check runs while decoding, in every one-shot path

A single .xz block with dictionary resets inside is decoded by the multi-threaded LZMA2 decoder, which splits it at the resets. That was the last one-shot path that checked a block only after decoding all of it.

Each thread now runs the check over its part as it decodes, 256 KiB at a time, while the data is still in cache. The parts' CRCs are then combined in order, with the same formula as zlib's crc32_combine.

Smaller changes

• Two compiler warnings inherited from fast-lzma2 are gone.
• bench/lzbench/add_uflzma2.py now renames the bundle's internal symbols instead of making them local. On Windows, the PE linker merged a COMDAT section that both libraries had, and a program linking fast-lzma2 and uf-lzma2 together crashed from level 2 on. zpaq-std found it.

Tested

• make check covers both features:
  • all four streaming APIs with sized headers, checking that every header states both sizes (4 to 11 blocks), plus xz -t and xz -lvv;
  • an 8 MiB block with a reset after every dictionary, decoded on several threads;
  • 2,000 random CRC combines.
• The tests can fail where they should: with the CRC combine made to drop the earlier parts, the multi-threaded case fails on CRC32 and CRC64.
• CI passes on x86_64, ASan+UBSan, 32-bit x86, single-threaded without xxhash, ARM64 (qemu), and Windows x64 and x86.

Compatibility

One parameter added; nothing removed or changed in meaning. The soname stays libuf-lzma2.so.1.

uf-lzma2 1.5.1

Fixed: crash on 32-bit x86 Windows

In 1.5.0, the new hardware CRC of .xz crashed with an access violation (0xC0000005) on 32-bit x86 Windows. It happened when the CRC ran in a thread that Windows started: multi-threaded or asynchronous .xz decompression, or any caller's own thread.

Windows starts threads with a stack aligned to 4 bytes on x86. A GCC that assumes 16 bytes, as Debian's mingw-w64 does, had turned a load into an SSE instruction that needs 16-byte alignment. The fold functions now realign the stack on entry.

Affected: 32-bit x86 Windows builds made by such a GCC. If you use 1.5.0 there, update.

Not affected: 64-bit builds, MSVC builds (the MSVC x86 DLL of 1.5.0 passes on Windows 7), Linux, and MSYS2's i686 GCC as configured by default.

Verified:

• Reproduced on Windows 7 and Windows 10 (x86).
• With the fix, the full test suite passes on Windows 7 SP1 (x86 and x64) and Windows 10 (x86).
• CI now runs make check on Windows x86 too. It is set up so that it crashes without the fix: this was checked on a branch without it.

Nothing else changed. Output, API and soname (libuf-lzma2.so.1) are the same as 1.5.0.

uf-lzma2 1.5.0

Hardware CRCs for .xz

CRC32 and CRC64 now fold the data 64 bytes at a time with a carry-less multiply where the processor has one: PCLMULQDQ on x86 (32 and 64 bit; GCC, Clang and MSVC), PMULL on ARM64 Linux and Apple. The instruction set is detected at run time. Elsewhere, and for input under 64 bytes, the CRCs use the tables as before.

One-shot decompression also runs the check 256 KiB at a time as each block is decoded, while the data is still in cache. Before, it checked a block only after decoding all of it, so the CRC read tens of MiB back from memory.

Measured on a Xeon E5-2697A v4, one thread:

• CRC alone: both CRCs run at 7.2 GB/s from memory and 16 GB/s from cache, against 3.1 (CRC32) and 2.5 GB/s (CRC64) with the tables.
• The check's cost to decompression of Silesia at level 6, against .xz with no check:

	1.4.0	1.5.0
one-shot, CRC64	-6.4%	about -0.8%
one-shot, CRC32	-4.1%	about -0.8%
streamed, CRC64	-4.9%	-0.4%
streamed, CRC32	-3.2%	-0.5%

The one-shot figure is an average of three runs that overlap, so read it as "under 1%", not as a precise number. ARM64 correctness is tested under qemu; its speed has not been measured on hardware.

Tests and CI

• make check runs the .xz and streaming tests: files written by xz 5.8.3 and 7-Zip 26.03, the library's own output one-shot and through every streaming API, a mutation stage, and a test of the hardware CRCs against the tables at every length to 1100 bytes and every alignment.
• The tests were checked for their power to fail: a wrong fold constant, a disabled block check, the check updated on the first step only, and the 1.4.0 crash put back each make them fail.
• CI runs make check on every push for x86_64, x86_64 under ASan and UBSan, 32-bit x86, a single-threaded build without xxhash, ARM64 under qemu, and Windows x64 (MSYS2).
• By hand, the tests also pass on Windows 10 built with mingw for x64 and x86, and against the DLLs MSVC builds for x64 and x86.

Fixes

• Streamed .xz output passed a NULL buffer to memcpy when no framing was queued. UBSan found it. Zero bytes were copied, so no output changed.
• test/Makefile kept its include path only when CFLAGS came from the file, and now links the static library by name.
• The musl job of the binaries workflow did not compile uf2_xz.c.
• The .xz test fixtures are kept byte for byte on Windows checkouts.

Compatibility

Output is unchanged: CRCs are the same values computed faster. No function or parameter was removed or changed in meaning. The soname stays libuf-lzma2.so.1.