zpaq std plugin

zpaq-std plugins

Two codecs, each its own file (zpaq-std loads one codec per file, codecs/<name>, its name the same as the file's and -ma:<name>'s):

codec	file	format	library starts with	levels	default
sylz	sylz.so / sylz.dll (entry 0, and zpaqstd_codec_v1)	v2 (frozen at v0.2.2)	"sylz format v2 "	0 .. 10	1
sylzec	the same file, entry 1 (sylzec.so / .dll of its own before v0.4.2)	v3 (frozen at v0.2.14): v2 plus Huffman-coded literals	"sylz format v3 "	0 .. 10	1

sylzec writes a v3 block when that is smaller, else a plain v2 block, and reads both; its levels are syc's -ec levels (0 .. 8 parse as v2's 2 .. 10, 9 and 10 price literals by their Huffman codes). zpaq-std tags its blocks sylzec3 and uses the plugin to decode them only when its library starts with "sylz format v3 "; otherwise its own ZPAQL decoder (ZPAQSYLZ3) runs. Its format: sylz format; a Python reference decoder (tools/sylz_ref.py) and test vectors (sylz-vectors.zip) for writing that decoder. Since v0.4.2 both codecs are in one file, sylz.so / sylz.dll, built by plugin/build.sh from plugin/sylz_zpaqstd.pas and plugin/zs_codec.inc: zpaqstd_codec_v1 returns sylz (what zpaq-std 1.7 and earlier load) and zpaqstd_codecs_v1(i) returns 0 sylz, 1 sylzec, then NULL. plugin/test_plugin.c [--codec N] tests either.

sylz

sylz is available to zpaq-std as -ma:sylz (experimental there), through its plugin ABI 1 (compressors/plug/zs_codec.h): a shared library in zpaq-std's codecs/ folder exporting one C function, zpaqstd_codec_v1, which returns a static zs_codec. zpaq-std ships its own ZPAQL decoder for the format, so archives open in any zpaq, with or without the plugin.

• Shipped in zpaq-std: v1.1.0 with syc v0.2.2's plugin; v1.2.0 with v0.2.7's (levels 0–10, and blocks flagged as format v3 refused on both sides); v1.6.0 with v0.2.8's; v1.7.0 with v0.2.10's.
• Format: sylz v2, frozen at tag v0.2.2. A later sylz format would be a new codec name in zpaq-std. From the commit after v0.2.6 the plugin refuses blocks flagged as format v3 (bit 31 of the first u32): before, it passed them to SylzDecompress, which reads v3 too.
• Source: plugin/sylz_zpaqstd.pas; binaries for Linux x86-64/arm64 and Windows x86-64/32-bit in every release since v0.2.2, and for Linux x86 and macOS arm64 since v0.5.4 (sylz-zpaqstd-*, rename to sylz.so / sylz.dylib / sylz.dll). On macOS the exports carry Darwin's leading underscore (_zpaqstd_codec_v1), which FPC 3.2.2 leaves out unless told.

Format identity (agreed with zpaq-std)

The library field of zs_codec starts with "sylz format v2 " for as long as the plugin writes and reads format v2 (today: "sylz format v2 (syc 0.2.5)"). zpaq-std uses the plugin to decode blocks tagged sylz2 only when it does. A plugin for a later format says "sylz format v3 ..." (and so on), never reusing an earlier prefix, and zpaq-std keeps decoding old v2 blocks with its own ZPAQL decoder (ZPAQSYLZ).

zpaq-std's segments carry the original size (u32le) in front, and its ZPAQL decoder never writes more than that, so a damaged block can't expand without bound.

Levels in zpaq-std

From syc v0.2.5 sylz has eleven levels, 0–10 (default 1), and the plugin reports min_level = 0, max_level = 10. zpaq-std uses the same numbers: -ma:sylz:0 .. -ma:sylz:10, 0 is sylz's fastest level, the default is 1 (zpaq-std no longer turns 0 into the default for sylz). sylz.pas uses the unit sylzhuf.pas (format v3's Huffman coder), so a vendored copy needs both files.

Up to v0.2.4 the levels were −2..12. The level is only a compressor setting: every block decodes the same whatever level made it.

Contract

levels	0–10
bound(n, level)	SylzBound(n); 0 for blocks over 2 GiB − 64 KiB (32-bit match positions)
compress	never writes past cap; −1 if cap is too small, −2 (ZS_ERR_MEMORY) if an allocation fails
decompress	needs the exact original size (cap); param unused; −1 on a damaged stream, never reads or writes outside the buffers
query, data	NULL
threads	safe to call from several threads at once (thread-safe RTL heap, no mutable global state)

Every target writes the same bytes. Until syc 0.5.1, 32-bit targets hashed the fast levels on 4 bytes instead of 6, so their output differed from the 64-bit builds' (both decode everywhere).

Building

plugin/build.sh linux-x86_64
plugin/build.sh win64
FPC=$SRC/compiler/ppc386 plugin/build.sh win32 -Fu$SRC/rtl/units/i386-win32
FPC=$SRC/compiler/ppca64 plugin/build.sh linux-arm64 -Fu$SRC/rtl/units/aarch64-linux \
  -Fl/usr/lib/gcc-cross/aarch64-linux-gnu/14 -Fl/usr/aarch64-linux-gnu/lib
FPC=/opt/fpc-i386/fpc plugin/build.sh linux-i386     # scripts/fpc-i386.sh
plugin/build.sh linux-arm64-native                    # on an arm64 host
plugin/build.sh macos-arm64                           # on the Mac: sylz.dylib

Cross compilers: Building. The arm64 library links against libc (pthreads, for cthreads), hence the gcc cross directories.

Test

plugin/test_plugin.c loads a plugin like zpaq-std (dlopen / LoadLibrary) and checks: the entry and ABI fields; roundtrips of every file at every level; guard bytes past cap for compress (also with cap too small) and decompress; 20 corrupted copies and a truncated one per file and level; 8 threads compressing and decompressing at once; and that loading the library leaves the floating-point state (MXCSR, and the x87 control word on 32-bit x86) unchanged.

cc -O2 -pthread plugin/test_plugin.c -o out/test_plugin -ldl
out/test_plugin out/plugin/linux-x86_64/sylz.so FILES...

Passes on Linux x86-64, Linux arm64 (qemu-user), Windows 10 x64 and Windows 7 SP1 (32-bit plugin under WoW64). Since v0.5.4 CI builds the plugin and runs this test on both codecs on every platform (Linux x86-64/arm64/x86, macOS arm64, Windows x64/x86). 32-bit Linux: gcc -m32 -msse; macOS: cc without -ldl.