Repository navigation
v2.1.0
Minor Changes
-
a44e897: Brotli dictionary compression streams now hold at most the first 4 MiB
less 16 bytes (4,194,288 bytes) of input and then stream, instead of
holding the whole input until it ends:createBrotliCompressDictStream()
andcreateBrotliCompressDictTransform()emit the output of those bytes as
soon as the input passes them, then compress each chunk as it arrives. A
stream that ends within them, as far as brotli can refer back to the
dictionary, gives the output ofbrotliCompressWithDict(). A longer stream
is compressed without the dictionary, which only helps the start of a
stream, because brotli's encoder can panic with a custom dictionary on some
data past the first 8 MiB. Its output decodes the same with or without the
dictionary, and is about as small: on JSON lines with a 2 KiB dictionary,
from 5.5% smaller to 2.2% larger than the output of
brotliCompressWithDict(), and at most 0.8% larger at qualities 1 to 11.BrotliCompressDictContextaccepts{ incremental: true }as a third
argument, which the streams use;flush()then returns all the output of
the input so far once the input has passed the first 4,194,288 bytes.
Without it, the context keeps its documented behaviour: it holds all of its
input and compresses it with the dictionary infinish(). -
0ef61e6: Give browsers the
*Asyncfunctions and@derodero24/comprs/streams, so
that code written for the native addon runs there too. In 2.0.x, the types
of thebrowsercondition declared the*Asyncfunctions, but the browser
entry did not export them, and@derodero24/comprs/streamsloaded the
native addon even in browser builds. The browser entry now exports all 23
*Asyncfunctions, and its declarations declare them. A browser has no
thread pool to run them on: each one runs its synchronous function on the
calling thread before it returns, and returns a Promise of the result,
which rejects on any error, including an invalid argument.
@derodero24/comprs/streamsnow has abrowsercondition for imports,
which resolves to the same Web Streams helpers on the WebAssembly build,
with their own declarations;require()keeps loading the native addon.
@derodero24/comprs/noderemains for Node.js only. -
b5fee8a: Add an optional
maxOutputSizeargument todecompress()and
decompressAsync(), in the native addon and the WASM build. Like the
maxOutputSizeofcreateDecompressStream(), it limits the decompressed
size in bytes whatever the detected format, and it defaults to 256 MB, so
code that auto-detects the format no longer has to calldetectFormat()
and dispatch to the*DecompressWithCapacity()functions to set another
limit. A second argument used to be silently ignored and is now the limit,
so code that passes one by accident, such asbuffers.map(decompress),
which passes the array index, must callbuffers.map((data) => decompress(data))instead. Values that are not integers from 0 to
Number.MAX_SAFE_INTEGERthrow. -
9ddb3cd: Stop building and publishing
@derodero24/comprs-wasm32-wasi, the WASI
build of the native addon; 2.0.2 is its last version. Since 2.0.2 it was no
longer an optional dependency of@derodero24/comprs, so it was only used
where it had been installed by hand. Node.js, Deno and Bun keep loading the
native addon for their platform. The wasm-bindgen build, which the
package'sbrowserentry loads, is now the only WebAssembly build.If you installed
@derodero24/comprs-wasm32-wasior set
NAPI_RS_FORCE_WASI, uninstall the package and unset the variable: on a
supported platform the native addon loads instead, and in browsers the
wasm-bindgen build. The generated loader still looks for the WASI package
when no native binary loads or whenNAPI_RS_FORCE_WASIistrueor
error, so a copy left innode_moduleswould be loaded with a newer
@derodero24/comprswhose API it does not match. On a platform without a
native binary, loading comprs in Node.js, Deno or Bun throwsCannot find native binding; if you relied on the WASI build there, stay on
@derodero24/comprs2.0.x. -
0a98674: LZ4 decompression streams now emit each block as soon as all of it has
arrived, and hold at most one block of their input (up to 4 MiB, or a
little over 8 MiB in a legacy frame), instead of the whole input until it
ends:
createLz4DecompressStream(),createLz4DecompressTransform(), and
createDecompressStream()andcreateDecompressTransform()for LZ4 input.
They report data after the last frame on the chunk that holds it.
Each block of a legacy frame (lz4 -l) is decoded into a buffer no larger
than its compressed size allows, rather than one of 8 MiB, so legacy frames
of small blocks also decode faster with the one-shot functions.Lz4DecompressContextaccepts{ incremental: true }as a second argument,
which the streams use:transform()then returns each block as it
arrives and throws as soon as the input is invalid,flush()returns
nothing,finish()throws unless the input ended between frames, and
maxOutputSizelimits the output of the whole stream. Without it, the
context keeps its documented buffering behaviour, including the output
limit that applies to eachflush()on its own. TheStreamContextOptions
type declares the options. -
fa0e184:
compress,decompressandtrainDictionaryfrom@derodero24/comprs/next
accept anAbortSignalas thesignaloption. An aborted call rejects with
thereasonof the signal. In Node.js, a call whose work no thread of the
libuv pool has started yet rejects at once, and the pool skips the work; work
that has started finishes, and its result is discarded. The*Async
functions of the package root take no signal. -
a971e09:
CompressionStreamandDecompressionStreamfrom@derodero24/comprs/next
are ponyfills of the classes of the Compression Streams standard, in Node.js
and in browsers. They cover zstd, gzip, deflate (zlib), deflate-raw, brotli
and lz4, and'auto'for decompression, with thelevel,dictionary,
gzipHeader,workersandmaxOutputSizeoptions and the error codes of
the unified API. In Node.js, chunks that take 2 ms or more run on the libuv
thread pool. The WebAssembly binary grows by about 6 KB with gzip. -
6b99ce5:
Dictionary.from(bytes, { format, level })in@derodero24/comprs/next
prepares a zstd or brotli dictionary once, which makes repeated zstd
compression and decompression of small messages with a dictionary much
cheaper: thedictionaryoption ofcompress,decompressand their
*Syncvariants takes it in place of the bytes of a dictionary, and
decompression defaults to its format.close(), or ausingdeclaration,
frees it early. -
971d4f9: New entry
@derodero24/comprs/nextfor Node.js and browsers:compress,
compressSync,decompress,decompressSync,detectFormat,
trainDictionaryandtrainDictionarySync. It takes options objects,
supports zlib ('deflate') and raw deflate ('deflate-raw'), returns plain
Uint8Arrayresults, gives every error a stablecode, and has aworkers
option for zstd in Node.js. The root API is unchanged. -
c40243c: Report the native memory of the stream contexts to V8 and allow releasing
it early. A context keeps its encoder or decoder state outside the
JavaScript heap, from a few hundred kilobytes for gzip to about 90 MB for
zstd at level 19, but V8 saw only a small object and had no reason to
collect it. A server that dropped contexts, such as the streams of aborted
responses, could grow to gigabytes before an unrelated garbage collection
freed them. The contexts now report their memory, so that V8 collects
abandoned contexts in time.Every context class gains a
close()method that releases the native state
right away; later calls throw<format> stream already closed. Contexts are
also disposable, sousing ctx = new ZstdCompressContext()closes the
context at the end of the scope.finish()releases the state too, and
Lz4DecompressContextgains thefinish()that the other contexts have.
The Web streams and Node.js Transforms close their context when they end,
fail, or are cancelled or destroyed. The contexts of the browser build gain
close()andLz4DecompressContext.finish()as well. -
c1a46b2: The 14 stream context classes gain
transformAsync(),flushAsync()and
finishAsync(), in the native and the browser build (where they run
synchronously). In Node.js, the Web streams and Node transforms run expensive
chunks on the libuv thread pool and yield between cheap ones, so the event
loop keeps turning. Output is byte-identical. A call while an asynchronous
call on the same context is in flight fails with<name> is busy: an asynchronous call has not finished, e.g.lz4 stream is busy: an asynchronous call has not finished.
Patch Changes
-
80da8a5: Reject the Promise of the
*Asyncfunctions on invalid arguments instead of
throwing synchronously. An invalid level, quality,capacity,
maxOutputSizeormaxDictSize, and an argument of the wrong type, such as
a string instead of aBuffer, used to throw before any Promise existed, so
promise.catch()missed the error. Every*Asyncfunction now returns a
Promise that rejects with the error that the synchronous function throws
for the same arguments, with the same message andcode. Code that awaits
the call insidetry/catchis unaffected; code that relied on a
synchronous throw from a call it does not await now gets a rejection
instead. -
b5fee8a: Detect every supported format reliably in
detectFormat(),decompress(),
decompressAsync(),createDecompressStream()and
createDecompressTransform(). The empty brotli stream that
brotliCompress()writes for empty input, zstd and LZ4 frames that follow
skippable frames, and LZ4 legacy frames (lz4 -l) used to be reported as
'unknown'; they are now detected. The auto-detecting streams decided on
the format after 4 bytes and failed on brotli input that arrived in small
chunks; they now buffer the input until the format is detected, up to
64 KiB.Brotli, which has no magic number, is now recognized by decoding up to the
first 64 KiB of the input instead of a single byte. Random data shorter than
64 KiB and raw deflate, some of which used to be reported as'brotli', are
now'unknown', and so is data that continues after the end of a brotli
stream shorter than 64 KiB, whichdecompress()used to decompress while
ignoring the rest. When data detected as brotli does not decode, including
a truncated brotli stream,decompress()throws theunable to detect compression formaterror instead of a brotli error. That error now also
points todeflateDecompress()for raw deflate. -
74cece7: Enforce
maxOutputSizewhile streaming decompression runs instead of after
each chunk. The gzip, deflate and brotli decompression streams and contexts
used to inflate a whole input chunk before checking the limit, so one small,
highly compressed chunk could allocate gigabytes before the size-limit error.
They now stop as soon as the output would exceed the limit, keeping memory
nearmaxOutputSize.finish()onGzipDecompressContextand
DeflateDecompressContextnow counts its output toward the limit too.Reject truncated input instead of returning a shorter result. The zstd,
brotli and raw deflate decompression streams,DeflateDecompressContextand
thedeflateDecompress*()functions used to succeed with whatever had been
decoded when the input ended mid-stream; they now throw<format> stream is truncated: unexpected end of input. The zstd and brotli decompression
contexts gain afinish()method that performs this check, and their
streams, including the auto-detecting ones, call it when their input ends.Empty input now throws for every format, in the one-shot functions and in
the streams alike, because no format has a valid zero-length encoding.
zstdDecompress*(),deflateDecompress*(),lz4Decompress*()and most
decompression streams used to return an empty result for it. -
23454f2:
BrotliDecompressContextandBrotliDecompressDictContextnow return all
the output of a chunk that ends mid-stream fromtransform(), in the
Node.js and browser builds, instead of 4 KiB of it per call, with the rest
held until more input arrived. The brotli decompression streams, and the
auto-detecting ones for brotli input, emit it right away too, so a reader of
a stream that is flushed after each message, such as a brotli response
flushed after each event, gets the whole message as it arrives. -
59055c4: Work around two bugs in the brotli 9.0.0 encoder that
brotliCompressWithDict(),
brotliCompressWithDictAsync()andBrotliCompressDictContexthit for some
inputs. At qualities 2 to 9, including the default, the encoder panicked,
which aborted the Node.js process; at qualities 10 and 11 it returned a
stream thatbrotliDecompressWithDict()rejected with "Invalid Data". When
the encoder panics, or when quality 10 or 11 output does not decode back to
the input, comprs now compresses that input again with neither the
dictionary nor brotli's built-in one, whose references a decoder given the
custom dictionary would misread. The result is a valid brotli stream that
decodes with or without the dictionary, only less compressed in these cases.
The native addon recovers from the panic without printing it to stderr;
every other panic still prints.
In the WebAssembly build, where panics cannot be caught, the panic still
traps until brotli is fixed. -
9305256:
brotliCompressWithDict(),brotliCompressWithDictAsync(), a
BrotliCompressDictContextwithout{ incremental: true }andcompress()
of@derodero24/comprs/nextwith a brotli dictionary now compress an input
of more than 8 MiB without the dictionary, and without brotli's built-in
one, into output that decodes the same with or without the dictionary. The
brotli encoder could panic on such inputs with a dictionary: the browser
build threwRuntimeError: unreachable, and the native addon compressed the
input twice. The output of shorter inputs does not change. -
933b413: Brotli stream compression at qualities 0 and 1 no longer compresses small
chunks far worse than large ones. At those qualities the encoder
compresses the input of each call on its own: written in chunks of 100
bytes, 300 KB of text that compresses to 120 bytes in one call took
204,335 bytes at quality 1, and 233,478 bytes of 300 KB of random words
that compress to 53,598.BrotliCompressContext, the streams and Node.js
Transforms built on it, and theCompressionStreamof
@derodero24/comprs/next, in the Node.js and browser builds, now pass
their input to the encoder in blocks of 64 KiB at qualities 0 and 1, so
that their output is the same however the input is split into chunks.
transform()holds up to 64 KiB of input until it completes a block, and
flush()andfinish()compress what it holds. Qualities 2 to 11 are
unchanged. -
118ed81: Reject Large Window Brotli streams in every brotli decompression function,
stream context anddecompress(), as RFC 7932 decoders and Node.js zlib
do. Their header declares a window of up to 1 GiB, and the decoder
reserved a buffer of that size before writing any output, whatever the
output limit: 12 bytes of input grew the memory of the WebAssembly build
to 1.5 GiB, which it never returns. comprs does not write these streams;
only encoders with the extension explicitly enabled do. -
694894d: The one-shot brotli decompression functions (
brotliDecompress(),
brotliDecompressWithDict(), theirWithCapacityandAsyncvariants,
anddecompress()anddecompressAsync()on brotli data) hand the decoder
the whole input at once. Small brotli decompressions no longer allocate a
4 MiB ring buffer: 10 KB of incompressible data decompresses about 40 times
faster, and 1 MB of incompressible data about twice as fast. -
fd2542f: Brotli compression of up to 256 KiB in one call keeps the ring buffer of
its encoder, at most 8.3 MiB, per thread instead of allocating and
zero-filling a new one for every call that fills it. Every call with a
dictionary filled it: compressing a 110-byte message with a 110 KiB
dictionary at quality 5 takes about 0.32 ms instead of 0.65 ms, with a
2 KiB dictionary 0.05 ms instead of 0.38 ms, inbrotliCompressWithDict()
and incompress()of@derodero24/comprs/nextwith a brotliDictionary
alike. Larger inputs and the streams allocate a fresh ring buffer, as
before: with glibc, keeping it made the compression of 1 MB and more
slower. The output does not change. -
6e5f891: Make the browser entry work with bundlers. In 2.0.x, browser builds of
import … from '@derodero24/comprs'failed with esbuild and webpack, and
Vite produced a bundle that threw on the first call: resolvers picked the
Node.js entry before thebrowserone, webpack refused to parse the
browser files as ES modules,"sideEffects": falselet bundlers drop the
WebAssembly initialisation, esbuild could not bundle the.wasmimport,
and the 2.0.2 browser entry imported the WASI package, which 2.0.2 does not
install.Imports under the
browsercondition now resolve to the WebAssembly build,
ahead of the Node.js entry, with type declarations that describe it rather
than the native addon. The browser files moved to abrowser/directory
that declares"type": "module", and the entry instantiates the
WebAssembly module with top-levelawait, from
new URL('./comprs-wasm_bg.wasm', import.meta.url): the functions are
ready once the import resolves, with no init call. Bundlers must support
top-levelawait. webpack 5 and Vite emit the.wasmfile as an asset;
with esbuild, copybrowser/comprs-wasm_bg.wasmnext to the bundle. On
Vite 7 and older,vite devneeds
optimizeDeps: { exclude: ['@derodero24/comprs'] }, and before Vite 7,
vite buildneedsbuild.target: 'es2022'.require()cannot load a module that uses top-levelawait, so
require('@derodero24/comprs')keeps loading the native addon even where
thebrowsercondition is set, as in Jest with a jsdom environment. Under
Jest's ES module support, that environment now imports the WebAssembly
build, which does not load in Jest; set
testEnvironmentOptions: { customExportConditions: ['node', 'node-addons'] }
to keep the native addon. The top-level browser files (browser-entry.js,
browser.js,browser-streaming.jsandcomprs-wasm*) and thebrowser
field ofpackage.json, which pointed atbrowser-entry.js, are gone. -
89a005b: Accept any
ArrayBufferorArrayBufferViewchunk, and a
SharedArrayBufferas well, in the browser build of the Web streams that
@derodero24/comprs/streamsprovides, as the streams of the native addon
now do. A bareArrayBufferorSharedArrayBufferused to fail, and
createDecompressStream()read aDataViewas empty and aUint16Array
element by element, so that it failed to detect the format; it could also
keep a view of anArrayBufferchunk, which the writer might overwrite,
while it waited for enough input to detect the format. Every chunk is now
read byte for byte, and a chunk that is not binary data errors the stream
with aTypeError. The browser typings accept these chunks too. -
0ef61e6: Make the stream contexts of the browser entry stream. They were JavaScript
stand-ins that kept every chunk until the end and then ran the one-shot
function: they held the whole input in memory, kept a view of each chunk
rather than a copy, so a caller that reused its buffer compressed
overwritten data, returned nothing fromflush()(or, when decompressing
zstd, brotli or LZ4, ended the stream there), reported corrupt input only
at the end, and passed a zstdmaxOutputSizeon as a capacity that grew
the WebAssembly memory to that size. The browser entry now exports the
stream contexts of the WebAssembly build, which work like the native
ones:transform()andflush()return output as soon as it is ready,
so code that kept only whatfinish()returned must keep what every call
returns, as on Node.js. Like those, they haveclose(), which
[Symbol.dispose]()calls, andLz4DecompressContext.finish(); they
also have afree()method, which frees the context object and its
WebAssembly memory before garbage collection does. -
8133e5b: Declare
CompressionFormatas a regular enum rather than aconst enum.
TypeScript cannot use the members of a declaredconst enumin projects
that enableisolatedModulesorverbatimModuleSyntax, as esbuild, swc
and Vite setups do: there,detectFormat(data) === CompressionFormat.Zstd,
an exhaustiveswitchwithcase CompressionFormat.Zstd:and, with
verbatimModuleSyntaxon TypeScript 5.9 and later, even importing the enum
failed with TS2748. They now type-check. Code thattsccompiles reads the
members from the exported object instead of inlining their strings.The browser entry now exports
CompressionFormatas well: in 2.0.x, the
declarations of thebrowsercondition declared it, but the entry did not
export it. The values are unchanged, the strings'zstd','gzip',
'brotli','lz4'and'unknown', so comparisons with strings keep
working. As in the object that TypeScript compiles an enum into, each
member is now a plain property in both builds:Object.keys(CompressionFormat)
returns the names,Object.values(CompressionFormat)the strings, and
JSON.stringify(CompressionFormat)shows them. In 2.0.x, the native addon
defined the members as read-only and not enumerable, so these returned[]
and{}. -
532cc79: gzip, zlib and raw deflate stream compression no longer compresses small
chunks worse than large ones. At levels 5 and 6 (6 is the default), the
output of zlib-rs depends on how its calls split the input: a stream
written in chunks of 1,000 bytes compressed repetitive text 17% larger
than in one call, and in chunks of 100 bytes 57% larger.
GzipCompressContext,DeflateCompressContext, the streams and Node.js
Transforms built on them, and theCompressionStreamof
@derodero24/comprs/next, in the Node.js and browser builds, now compress
their input in blocks of 32 KiB, so that their output is the same however
the input is split into chunks, and small chunks compress faster.
transform()holds up to 32 KiB of input until it completes a block, and
flush()andfinish()compress what it holds.A stream that is flushed when its input pauses, as the middleware does,
still sends each write promptly: a small write now reaches the client with
that flush, the gzip header and the response headers included, instead of
partly before it. -
e939fc2: Derive the ES module entry from the CommonJS one.
index.mjslisted every
export by hand and has drifted from the native addon before; it now
re-exports the CommonJS loader and the stream helpers withexport *, so
the two entries cannot drift apart. Node.js and Deno see the same exports
as before. Bundlers now follow the entry into the loader and bundle it,
instead of leaving arequire('./index.js')that fails once the bundle
moves; as before, the native addon itself must stay external. -
e939fc2: Fix the type declarations of the ES module entry.
index.d.mtsre-exported
declaration files, which TypeScript rejects with TS2846, so ES module
projects that type-check their dependencies (skipLibCheck: false) got two
errors fromnode_modules. It now re-exports./index.jsand
./streams.js, and the exported types are unchanged. -
3d0d9b2:
flush()ofGzipCompressContextandDeflateCompressContextnow emits
all the input written so far, in the Node.js and browser builds. After a
transform()of much poorly compressible input, such as random or already
compressed data, a flush could leave up to about 16 KiB of it in the
encoder until the next call, so a client decoding the output as it arrives,
such as that of a response the middleware flushes while its handler waits,
got less than had been written. A flush that was already complete returns
the same bytes as before. -
568419c: Update flate2 to 1.1.10 with its
runtime_detectionfeature, so zlib-rs
keeps using SIMD and CRC instructions and gzip and deflate keep their
speed in the native binary. One-shot gzip decompression (gzipDecompress,
its variants anddecompress) of input that ends inside the compressed
data can now report "incomplete deflate stream" instead of "unexpected end
of file"; the gzip streams and contexts report the same errors as before. -
23454f2:
transform()ofGzipDecompressContextnow returns all the output of its
chunk, in the Node.js and browser builds. It kept the last 32 KiB or less
until the next call, so a gzip stream that is flushed after each message,
such as a response the middleware flushes after each event, reached a
reader ofcreateGzipDecompressStream(),createGzipDecompressTransform()
or the auto-detecting streams one message late, and a short message not at
all until the next one. -
0f15e49: Reject gzip header filenames that cannot be stored.
gzipCompressWithHeader()
with afilenamethat contains a NUL character ('\u0000') aborted the
Node.js process, and trapped withRuntimeError: unreachablein the WASM
build. It now throwsgzip filename must not contain NUL characters. A
filename longer than 65535 bytes in UTF-8 was written, butgzipDecompress()
andgzipReadHeader()could not read the result back; it now throwsgzip filename must be at most 65535 bytes long. -
15b4d0b: Stop gzip decompression from reserving memory because of a forged size
trailer.gzipDecompress(),gzipDecompressWithCapacity(), their async
variants anddecompress()sized their initial output buffer from the
ISIZE trailer, which nothing verifies until decoding ends, so a few bytes of
input could reserve up to 4 GiB, capped only by the output limit (256 MB by
default). The initial buffer is now also capped by what the input can
expand to and by 64 MiB, and larger outputs grow the buffer as they decode.
In the native addon, gzip, brotli and LZ4 decompression also report a failed
reservation of the initial output buffer as an error instead of aborting the
process. -
18fb990: Write LZ4 frames in blocks of at most 256 KiB. For more than 256 KiB of
input,lz4Compress()andlz4CompressAsync()wrote 4 MiB blocks, as the
lz4CLI does by default, so the encoder and every decoder of the frame
needed 4 MiB buffers. They now write 256 KiB blocks: compressing 1 MB of
repetitive data is about 3 times faster, and the output is about 0.3%
larger on text. They also make room for the whole frame up front, so
compressing 1 MB that does not compress is about 30% faster.
Lz4CompressContextand the LZ4 compression streams sized their blocks
from the first chunk and now always write 64 KiB blocks, so a first chunk
of more than 256 KiB no longer makes the context hold 8 MiB that it did not
report to V8; their output is then about 1.5 to 2% larger on text than with
the 4 MiB blocks that such a chunk gave. The compressed bytes change for
more than 256 KiB of input, and for streams whose first chunk holds more
than 64 KiB; they are still standard LZ4 frames that any LZ4 decoder reads. -
3d7ddef: Write a content checksum in LZ4 frames.
lz4Compress(),
lz4CompressAsync(),Lz4CompressContextand the LZ4 compression streams
now add an xxHash32 of the data to each frame, as thelz4CLI does by
default, so decompression detects corrupted data instead of returning it.
The output is still a standard LZ4 frame that any LZ4 decoder reads, 4 bytes
longer than before, and its FLG byte changes from0x60to0x64.
Computing the checksum makes compression about 15% slower, and verifying it
makes decompressing these frames up to about 30% slower. Frames without a
checksum, as other encoders may write them, still decode. -
18fb990: Decode LZ4 frames that declare large blocks faster.
lz4Decompress(),
lz4DecompressWithCapacity(), their async variants,decompress()and
decompressAsync()on LZ4 input,Lz4DecompressContextand the LZ4
decompression streams zero-filled a buffer of the frame's block maximum
size on every call: 4 MiB for frames from thelz4CLI, which declares
4 MiB blocks by default even for small content. Each thread now keeps that
buffer for its next call while it holds at most 4 MiB, and only its growth
is zero-filled: decoding 10 KB from such a frame is about 35 times faster,
and 84 KB of JSON about 5 times. The 8 MiB buffer of a legacy frame
(lz4 -l) is not kept. -
3d7ddef: Decode every frame of LZ4 input and reject truncated or trailing data.
lz4Decompress(),lz4DecompressWithCapacity(), their async variants,
decompress(),Lz4DecompressContextand the LZ4 decompression streams
used to stop at the end of the first frame, or at the first empty block, and
silently drop whatever followed, and they accepted a frame cut short at a
block boundary. They now decode concatenated frames like thelz4CLI.
Input that ends inside a frame, including a frame without its end mark,
throwslz4 stream is truncated: unexpected end of input, and a frame
followed by data that is not a frame throws anunexpected data after the end of a frameerror. The output limit (capacity,maxOutputSize) covers
all frames together.The LZ4-specific functions, context and streams also skip skippable frames,
which used to fail withSkippableFrame, and accept a legacy frame
(lz4 -l) followed by further frames, or with blocks of data that does not
compress, which used to fail withBlockTooBig. -
b15d12d: The Node.js entry of the package root now also exports
__napiBindingTarget, throughrequire()andimport. The loader that
@napi-rs/cli 3.10 generates adds it to report which binding it loaded.
It is declared as'native' | 'wasm32-wasi' | 'wasm32-wasip1', and with
the native addons that this package publishes its value is'native'.
It comes from napi-rs and is not part of the comprs API. The browser
entry and thestreamsandnodesubpaths do not export it. -
ebb2190:
require('@derodero24/comprs')no longer lists 23 undeclared*Task
classes, such asZstdCompressTask, that napi-rs generated for the
*Asyncfunctions and that could not be constructed. The ES module entry,
which now re-exports the CommonJS loader, does not list them under Bun and
bundlers either. The*Asyncfunctions, their results and their errors
are unchanged. -
89a005b: Push the output of the Node.js transforms from
@derodero24/comprs/nodein
chunks of at mostreadableHighWaterMarkbytes (64 KiB by default, 16 KiB on
Windows). A small input chunk that decompressed to many megabytes used to
arrive as one chunk of that size, which downstream consumers had to take in
at once. The chunks are views of the native output, not copies. -
694894d: Buffers returned by the one-shot compression and decompression functions
no longer retain an allocation sized from the input. The encoders reserved
an output buffer as large as the input, and the decoders grew theirs by
doubling; the whole allocation lived as long as the returned Buffer, so 50
retainedbrotliCompress()results of an 8 MB input held about 400 MB of
memory for less than 1 KB of output. Results now release large spare
capacity before they are returned. The LZ4 compression streams also stop
copying each output chunk. -
43d32c8: Update Rust crates that compile into the published native binary and WASM
build, including zlib-rs 0.6.8 (the gzip and deflate backend),
brotli-decompressor 6.0.1, crc32fast 1.5.2, twox-hash 2.1.5 and thiserror
2.0.21, and for the WASM build wasm-bindgen 0.2.129 and js-sys 0.3.106. -
9fcb6c6:
DeflateDecompressContext,BrotliDecompressContextand
BrotliDecompressDictContextnow keep failing after an error, in the
Node.js and browser builds: oncetransform()has thrown, for example on
data after the end of the compressed stream,flush()andfinish()throw
the same error instead offinish()returning the output decoded before
that data, and the context releases its decoder state at once. The other
decompression contexts already failed there. The streams, which close their
context on any error, are not affected. -
89a005b: Accept any
ArrayBufferorArrayBufferViewchunk, asCompressionStream
does, and aSharedArrayBufferas well, in the Web streams that
@derodero24/comprs/streamsprovides with the native addon (Node.js, Deno
and Bun). A bareArrayBufferused to fail withValue is none of these types, andcreateDecompressStream()failed to detect the format of
DataViewandUint16Arraychunks, which it read as empty or element by
element. Every chunk is now read byte for byte, and a chunk that is not
binary data errors the stream with aTypeError. The typings accept these
chunks and remain assignable toTransformStream<Uint8Array, Uint8Array>. -
94945f4: The Web streams of the native and browser builds read the bytes of each
chunk rather than itsbyteLength,byteOffsetandbufferproperties,
as the other functions do. A typed array subclass or a view with own such
properties is now compressed and decompressed as the bytes it holds;
createDecompressStream()buffered such chunks at the wrong size, and a
DataViewwith an ownbyteOffsetwas read from the wrong place. -
e25f89a: The npm packages now include
THIRD_PARTY_LICENSES, the licenses of the Rust
crates and of the zstd C library that the native and WebAssembly builds link
statically, and the platform packages include comprs'sLICENSE. -
ccfb44d: No API change:
streams.js,node.js,index.mjsandbrowser/streams.js
are now compiled from TypeScript, and their declarations are generated from
the code. -
54bf460: Fix a crash in Bun and Deno when a method of a native context class is
called with an instance of another context class asthis, such as
ZstdCompressContext.prototype.transform.call(gzipContext, chunk). The
method used the other class's native state as its own, and the process
crashed with a segmentation fault. It now throws anInvalidArgerror;
Node.js already rejected such calls withIllegal invocation.The fix comes with the update of the native addon to napi 3.14.0,
napi-derive 3.6.10 and napi-build 2.6.0, which tag every class instance
and check the tag before a method uses it. The crates were held at 3.9.1 /
3.5.6 / 2.3.2 for the WASI build, which is no longer published. -
dc859ee: In the native addon, the synchronous functions and the stream contexts now
return results of up to 2 MiB in memory that the JavaScript engine owns,
instead of memory that Node.js frees only on a later turn of the event
loop. A synchronous loop no longer page-faults fresh memory on every call
or holds on to every result until it yields: on Linux with glibc,
deflateDecompress()calls that return 1 MB take about 0.4 ms instead of
1.1 ms, and 200 of them grow the resident set by about 12 MiB on Node.js 22
and 50 MiB on Node.js 24, instead of about 190 MiB. Such results can also
be transferred to workers, so the Web streams no longer copy each chunk a
second time, and a chunk of the Node.js transforms that holds a whole
result can be transferred too. Larger results, and the results of the
*Asyncfunctions, stay in the memory of the addon, which saves a copy. -
0f15e49: Reject invalid numeric arguments instead of silently converting them.
Compression levels and qualities, thecrc32()initial value and the gzip
headermtimewere converted to 32-bit integers before any check ran, so
NaNandInfinitybecame 0 (no compression for gzip, deflate and brotli),
1.9became 1,2 ** 32 + 1became 1 andcrc32(data, -1)used
0xFFFFFFFF. This applied to the one-shot and async functions and to the
stream contexts, in the native addon and in the WASM build, which also
turned non-numeric strings and objects into 0.maxOutputSizetruncated
fractions (0.5became a limit of 0 bytes), and acapacityof2 ** 64
passed validation. All of these now throw an error that names the argument
and its accepted range, for examplegzip compression level must be an integer between 0 and 9; the messages of the existing range errors changed
to this form.capacityandmaxOutputSizeaccept integers from 0 to
Number.MAX_SAFE_INTEGERon every platform (the WASM build used to reject a
capacityof2 ** 32or more), and a limit of 0 accepts only data that
decompresses to nothing.maxDictSizeaccepts integers from 0 to 16777216. -
5cead12: Read the bytes of input arrays in the browser (WebAssembly) build, rather
than trust theirlengthproperty, as the native addon does. A
Uint8Arraysubclass, or a typed array with an ownlengthproperty, whose
lengthdisagreed with its bytes made the build return wrong output that
held other data from its WebAssembly memory, or throw aRangeErrorthat
left a stream context unusable. Other views now read as in the native addon
too: a view of a detachedArrayBuffer, or one out of the bounds of a
resizableArrayBufferthat shrank, reads as empty rather than throwing a
TypeError, and a view whose prototype was swapped is read by its bytes
rather than throwing. An object that only inherits from
Uint8Array.prototypeis rejected with the sameErroras other values
that are not byte arrays, rather than aTypeError. -
7cf02d8: Make the WebAssembly build take and return what the native addon does.
ItsgzipCompressWithHeader()took(data, level, filename, mtime)
rather than the(data, header, level)of the published types, so
gzipCompressWithHeader(data, { filename }, 6)failed with
RuntimeError: memory access out of bounds. It now takes
(data, header, level), as the native addon does; the positional form was
never part of the published types, and calls that use it must pass the
header as an object.gzipReadHeader()now leaves out the header fields
that are absent, rather than setting them tonull, as the native addon
does.Byte array arguments are now checked as the native addon checks them:
anything but aUint8Array(aBufferis one) or anotherArrayBuffer
view throws anError. The WebAssembly build used to compress an
ArrayBufferas empty input and a string as one zero byte per character,
and its stream contexts took anArrayBufferor an array of numbers as a
chunk. -
7cf02d8: Log the message of a panic in the WebAssembly build. A panic aborts the
call with a trap, which throws a bareRuntimeError: unreachable; the
build now logs the panic message withconsole.error()first. -
2180aaf: Shrink the WebAssembly build that browsers load.
comprs-wasm_bg.wasm,
which 2.0.2 compiled for speed, is now compiled for size, which makes the
same code about a fifth smaller, 14% smaller to download with gzip (level
9) and 12% smaller with brotli (quality 11). The README lists the sizes of
the current build, under "WASM bundle size". In exchange, brotli
compression runs about 40% slower in the browser, brotli decompression
about 30%, gzip decompression about 20% and LZ4 compression about 10%
slower; zstd, gzip compression and LZ4 decompression keep their speed. The
native addon is unchanged. -
b683226: Fix a memory-safety bug in the browser (WebAssembly) build: a zstd operation
whose memory allocation cannot be satisfied — for example decompressing a
frame that declares a very large window in a memory-constrained tab — used to
corrupt WebAssembly linear memory and trap with an opaque out-of-bounds error.
It now fails deterministically at the point the allocation fails. The wasm
crate installs a global allocator that aborts on allocation failure, so
zstd-sys's WebAssembly allocation shim can never receive a null pointer and
hand zstd a buffer derived from it. The native addon uses the system allocator
and was never affected. -
bae7130: Link the C runtime statically into the Windows ARM64 binary
(@derodero24/comprs-win32-arm64-msvc), as the x64 one already did. Up to
2.0.2 it importedVCRUNTIME140.dlland theapi-ms-win-crt-*DLLs, so
loading comprs failed on Windows on Arm machines without the Visual C++
Redistributable for ARM64. It now loads without it. -
15b4d0b: Size zstd one-shot decompression output from the data instead of reserving
it up front.zstdDecompress(),zstdDecompressWithDict(), their async
variants anddecompress()reserved 256 MB for every frame without a
content size, which streaming encoders such asZstdCompressContextwrite,
and the*WithCapacity()variants allocated the wholecapacity, so a value
such as2 ** 40aborted the process. The output buffer now grows with the
decompressed data: frames that declare their size still decode straight
into a buffer of that size, as long as the input could actually expand to
it, and everything else goes through the streaming decoder.capacityis
only a limit, and output over it throwszstd decompress exceeded maximum size of <capacity> byteslike the other formats, instead ofDestination buffer is too small.The one-shot zstd functions now accept concatenated frames and skippable
frames, as the streaming API already did, and report truncated input as
zstd stream is truncated: unexpected end of input; data after the last
frame now usually reportsUnknown frame descriptorinstead ofSrc size is incorrect. Like the streaming API andzstd -d, they reject a frame
without a content size whose window exceeds 128 MiB (written by
zstd --long=28or higher on piped input) withFrame requires too much memory for decoding. -
15b4d0b: Reject a
maxDictSizeabove 16 MiB (16777216 bytes) in
zstdTrainDictionary()andzstdTrainDictionaryAsync(). The value was
allocated up front, so one that could not be allocated, such as2 ** 40,
aborted the process (a trap in the WASM build). zstd recommends
dictionaries of about 100 KB, and training allocates several buffers of
maxDictSizebytes. -
8f6967f: Speed up small one-shot zstd calls.
zstdCompress(),zstdDecompress(),
zstdDecompressWithCapacity(), their*Asyncvariants, anddecompress()
anddecompressAsync()for zstd input reuse one compression and one
decompression context per thread instead of creating one per call:
compressing messages of about 110 bytes is about 6 times faster, and
decompressing them about 12 times. A thread keeps a context only while it
holds at most 8 MiB, so a large or high-level call does not leave its
workspace behind. The output is unchanged, and the dictionary functions
still create a context per call. -
8f6967f: Speed up zstd stream compression with small chunks.
ZstdCompressContext,
ZstdCompressDictContextand the zstd compression streams no longer
zero-fill a 128 KiB output buffer on every call: 10 MiB written in 1 KiB
chunks compresses about 10 times faster for highly compressible data and
up to twice as fast for JSON lines. The output is unchanged. -
27a3926: Bound the window of the zstd decoders by the output limit. A zstd frame
header can declare a window, or for a single-segment frame a content size,
of up to 128 MiB, which the streaming decoder allocated as soon as it had
read the header, whatevercapacityormaxOutputSizesaid. Stream
contexts decode every frame with it, and the one-shot functions any input
whose frame headers do not size the output, such as frames without a
content size: 100ZstdDecompressContexts with amaxOutputSizeof 1024
bytes, fed the 6-byte header of such a frame, held 12.5 GiB of address
space and reported it to V8, and the WebAssembly build grew its memory by
128 MiB for good.zstdDecompressWithCapacity(),zstdDecompressWithDictWithCapacity(),
their async variants,decompress()anddecompressAsync()for zstd input,
ZstdDecompressContext,ZstdDecompressDictContextand the zstd
decompression streams now accept a window of at most the limit rounded up
to a power of two, but never less than 8 MiB, the most that zstd writes at
levels up to 19, nor more than zstd's default of 128 MiB. A frame whose
window exceeds the bound fails before the decoder allocates anything, in
one of two ways:- A limit of 64 MiB or less lowers zstd's bound, and the frame throws
zstd frame window exceeded maximum size of <limit> bytes, the size-limit
error. Raising the limit decodes the frame if its window is at most
128 MiB; a window over 128 MiB throws the same error, although no limit
decodes it. - A limit of more than 64 MiB, such as the default of 256 MB, keeps zstd's
bound of 128 MiB, and a frame whose window exceeds it still throws zstd's
Frame requires too much memory for decoding, a corrupt-data error.
Of the frames that zstd writes, only those without a content size whose
window exceeds 8 MiB can now fail under a limit that their output fits in:
streams thatZstdCompressContext, the streams built on it or
zstd --ultrareading a pipe compress at levels 20 to 22, and streams
compressed withzstd --long. Under an explicit limit, they now need one of
more than 16 MiB at level 20, 32 MiB at level 21, and 64 MiB at level 22 or
with--long. Frames that zstd compresses with a known size, such as those
ofzstdCompress(), declare no window larger than their content and decode
under any limit that their content fits in. - A limit of 64 MiB or less lowers zstd's bound, and the frame throws
Release assets
Each comprs.<abi>.node is the native addon of the npm package @derodero24/comprs-<abi>. THIRD_PARTY_LICENSES.<abi>.txt holds the license notices of the third-party code that it links statically, and LICENSE is the license of @derodero24/comprs.