Repository navigation
The zpaq std plugin
PJA also ships as a codec plugin for
zpaq-std, so JPEG and JPEG-LS
and JPEG 2000 files inside a zpaq archive get PJA's compression. zpaq-std
ships it and uses it with -sa (since 2.6.0 no longer experimental), on
its plugin ABI 1; -sat/-sae name a codec by hand.
Each release has the plugin built for the same six platforms as the
command: pja-VERSION-zpaqstd-plugin-PLATFORM (pja.so on Linux and macOS,
pja.dll on Windows; up to 0.6.3 the macOS file was named pja.dylib,
rename it to pja.so). Put that file in zpaq-std's codecs/ folder.
pja-VERSION-zpaqstd.tar.gz has the plugin's sources as zpaq-std vendors
them, with build instructions in its VENDOR.md.
Two things make up a codec, and each is frozen under the codec's name:
-
the stream: the format of the compressed data PJA writes for a file
(stream version 5, 6 or 7). The
pjacommand and the plugin read every version. - the ZPAQL decoder: the small program, in zpaq's own language, that every block carries so that any zpaq extracts it without the plugin. Once released, a codec's decoder never changes, byte for byte. A smaller or faster decoder, even for the very same stream, comes as a new codec with a new name.
A codec named with a z (pja7z, pjalsz, ...) is its codec with an
LZ77 in front of the decoder: zpaq-std uses it for a whole -sa group,
when the files' streams share data (the same image saved several ways).
Same files, same streams.
| files | what -sa uses (zpaq-std 2.5.0 and later) |
older codecs, still read |
|---|---|---|
| JPEG (Huffman or arithmetic) |
pja78 (stream v7, ZPAQL opt=3), since PJA 0.8.0 |
pja (stream v5), pja7 (v7, opt=2) |
| JPEG with JPEGs inside (MPO, Ultra HDR), 12 bits, DNL |
pjax78 (stream v6 with v7 inside, opt=3) |
pjax (v6 with v5 inside), pjax7 (opt=2) |
| lossless JPEG (SOF3) |
pjall8 (stream v5, opt=3) |
pjall (not optimized), pjallo (opt=2) |
| JPEG-LS |
pjals8 (stream v5, opt=3) |
pjals, pjalso
|
| JPEG-LS with subsampled components |
pjalsx8 (stream v5, opt=3) |
pjalsx, pjalsxo
|
| JPEG 2000 (Part 1) |
pja2k (stream v8, modes 0 and 1), since PJA 0.9.6 and zpaq-std 2.6.0 |
Each has its z: pjaz, pjaxz (read only); pja7z, pjax7z,
pjallz, pjalsz, pjalsxz (what -sa uses); pjallzo, pjalszo,
pjalsxzo (0.7.0); pja7z8, pjax7z8, pjalsz8, pjallz8, pjalsxz8
(0.8.0). There is no version 7 of the lossless JPEG and JPEG-LS codecs:
their stream is the samples model's, which version 7 did not change.
pja2k has no z: an LZ77 across a block's JPEG 2000 files gains nothing
measurable.
Codec 30, pja2k, takes Part 1 JPEG 2000 files (JP2, JPX, JPM, Motion
JPEG 2000, raw codestreams): stream version 8 as the command writes it,
each codestream's decisions through the model (mode 1) or as bytes. Its
ZPAQL (51,521 bytes, comp 0 0 $3 $4 0: $3 H's words and $4 M's bytes,
both from the query) lets any zpaq extract it without the plugin; it is
4.7 to 7.9 times slower than the plugin. About 7.7% saved on 113 real
codestreams (the command, with its pixel modes, 8.4%). HTJ2K codestreams
and the command's pixel modes have no ZPAQL: they stay as bytes, and a
file of HTJ2K codestreams only is not taken. zpaq-std 2.6.0 tries it in
-sa, last of the family (before, -sae:pja:pja2k). Frozen from 0.9.6:
its name, stream, what it takes and its ZPAQL.
The plugin holds thirty-one codecs: the five
below, their z twins, the four of version 7, the o ones, the 8 ones and
pja2k.
These five were the first; -sa and -sa:pja try their newest
versions in this order, pjax78, pja78, pjall8, pjals8, pjalsx8,
then pja2k (zpaq-std 2.6.0). What each takes:
| codec | takes |
|---|---|
pjax |
Huffman JPEGs that need stream version 6: 12 bits, a height in a DNL, a DRI with no restarts, or 256 KiB or more of JPEGs after the end (an MPO's or Ultra HDR's further images) |
pja |
every other JPEG, Huffman or arithmetic, EXIF thumbnails included |
pjall |
lossless JPEG (SOF3), 2 to 16 bits |
pjals |
JPEG-LS |
pjalsx |
JPEG-LS with subsampled components |
Each codec recognises its files by their bytes; zpaq-std decodes every block before writing it.
Each of the five has a z twin: pjaz, pjalsz, pjaxz, pjallz,
pjalsxz. A z codec takes the same files and writes the same streams;
what changes is the block. zpaq-std passes the streams of a block's files
through an LZ77 (zpaq's -m1) whose matches reach back into the files
before, so what files share is stored once: the same image saved
progressive and baseline, or with and without restarts, gives the same
coefficients. zpaq-std uses a z codec for a block only when it comes out
smaller. In zpaq-std: 244 arithmetic-coded JPEGs 71.3% -> 18.8%, 640
generated variants 72.3% -> 17.9%; photos, corpus and MPO a little
smaller, none larger.
The pja command writes stream version 7 since 0.1.0: about 3% smaller
and 1.6 times as fast as version 5, the one pja and pjax's ZPAQL read.
0.6.0 brings it to the plugin as four new codecs: pja7 and pjax7 take
what pja and pjax take, and pja7z and pjax7z are their z twins.
The plugin links version 7's C model for them. zpaq-std 2.1.0 uses them
for -sa and -sa:pja; the names only show in listings and in -sat.
Measured by zpaq-std through its whole path (-sa -t8), against its
2.0.3:
| set | 2.0.3 | with version 7 | |
|---|---|---|---|
| photos | 59,009,713 | 57,151,702 | -3.1% |
| corpus | 23,290,117 | 22,739,830 | -2.4% |
| MPO | 30,570,193 | 30,254,230 | -1.0% |
| arithmetic-coded | 1,251,109 | 1,212,377 | -3.1% |
| generated | 3,183,477 | 3,099,865 | -2.6% |
Writing 1.3 to 1.6 times as fast, extracting 1.2 to 1.6 times, and plain zpaq 7.15 extracts 1.5 to 1.6 times as fast with the new ZPAQL (photos 191 -> 121 s).
Every block carries the codec's decoder written in ZPAQL, zpaq's own decoding language, so a plain zpaq (7.15, zpaqfranz) extracts the files without the plugin. Several files may share a block.
An archive written today has to open forever, so each codec's stream and
ZPAQL never change under its name: pja and pjals are as 0.0.1 left
them, pjax as 0.0.3, pjall as 0.0.5, pjalsx as 0.0.6, the five z
codecs as 0.4.0, pja7, pjax7, pja7z and pjax7z as 0.6.0, the six o codecs as 0.7.0, the ten 8 codecs as 0.8.0.
New formats and new models come as new codecs.
Because of that, the plugin takes a little less than the pja command:
Photoshop's progressive JPEGs (handled by the command since 0.0.8) and
12-bit arithmetic-coded progressive JPEGs go through none of its codecs,
and zpaq-std compresses them its usual way.
They are compiled from pas/plugin/zpaql/ with tools/zpaql/ (Python,
build time only):
| codec | ZPAQL | covers |
|---|---|---|
pja |
62,294 bytes | every JPEG PJA handles as of 0.0.1, Huffman or arithmetic, sequential or progressive |
pjals |
55,989 bytes | every JPEG-LS file, with either of its two models |
pjax |
57,709 bytes | the Huffman JPEGs of stream version 6 |
pjall |
30,508 bytes | lossless JPEG, 2 to 16 bits, through JPEG-LS's samples model |
pjalsx |
60,776 bytes | JPEG-LS whose components are of other sizes than the frame |
pjaz, pjalsz, pjaxz, pjallz, pjalsxz
|
39,458, 32,984, 37,476, 18,003, 36,110 bytes | their codec's, after expanding the block's LZ77 |
pja7, pjax7, pja7z, pjax7z
|
35,104, 35,211, 35,829, 35,987 bytes | stream version 7, and their z |
pjalso, pjallo, pjalsxo (0.7.0) |
24,076, 15,058, 25,623 bytes | pjals', pjall's and pjalsx's, compiled with opt=2: 2.2 to 2.3 times as fast in zpaq 7.15 |
pjalszo, pjallzo, pjalsxzo
|
24,724, 15,693, 26,278 bytes | their z |
pja78, pjax78, pja7z8, pjax7z8 (0.8.0) |
32,954, 33,197, 33,603, 33,949 bytes | pja7's, pjax7's and their z, compiled with opt=3 (inlining, operands and return values through B): 2 to 6% faster in zpaq 7.15 |
pjals8, pjall8, pjalsx8, pjalsz8, pjallz8, pjalsxz8
|
22,430, 13,901, 24,611, 23,046, 14,517, 25,244 bytes | the o codecs' programs with opt=3
|
pja2k (0.9.6) |
51,521 bytes | JPEG 2000 Part 1, stream version 8 modes 0 and 1, opt=3
|
The z decoders are smaller than their codecs' although they do more: the compiler learnt, for programs not frozen yet, to build constants in fewer instructions and to put the most used variables where one instruction reaches them (from zpaq-std's issue #4). Version 7's decoder came out at 71,052 bytes that way, over ZPAQL's 65,535; for it the compiler also lets functions that never call one another share their variables' places, and drops a few redundant instructions: 35,104 bytes.
Their memory is sized for each block: the plugin tells zpaq-std how much the decoder will need for that file (2 MB for a small one, 16 MB for a photo of 200 KB, at most 256 MB).
Checked with zpaq-std's test build on Linux and on Windows 10 (64 and 32 bits): every block decoded by the ZPAQL as by the plugin, and every file back without the plugin, plain zpaq 7.15 included (doc/CODEC.md has the sets).
zpaq-std spreads its threads across files, so each plugin call used one
thread, and a few large files (a camera's MPO of 10 MB) left the rest
idle. A zpaq-std that calls the optional export zpaqstd_threads_v1(n)
with its number of threads lets each call take n / (the calls running)
for its file's bands: four such MPO compress about twice as fast and
extract 2.6 times as fast, with the same streams.
pas/plugin/build.sh DIR builds pja.so, pja.dll and, with PPC386
and RTL386, the 32-bit pja.dll.