VaptVupt v2.65.11

This release improves small-input encoding and makes entropy-table ownership
explicit. The wire layout and existing public entry points remain compatible.
It is a userspace codec release, not a Linux kernel port or a claim of
universal superiority over LZ4/Zstd.

• Fast one-shot inputs up to 4 KiB initialize only reachable full-width hash
  buckets and allocate a circular chain sized to the input. A 4 KiB input
  avoids 240 KiB of chain allocation at the default window; the 1 MiB primary
  map allocation remains. Streaming and larger-input setup are unchanged.
  Seven paired in-process runs measured about 2.7x faster encoding for 1 KiB
  text and +10.7% throughput for 4 KiB text, with identical compressed bytes.
  These allocation-sensitive measurements are separate from CLI/page-API
  comparisons; large-input controls were approximately unchanged.
• Fix explicit format_v2 with fast mode. Plain compressed blocks always
  decode matches with a four-byte minimum, but this combination previously
  emitted a three-byte bias and could produce undecodable data. Fast mode
  now uses ordinary v1 tokens; balanced/extreme retain the requested v2
  entropy format. Creation, chunk compression and reset apply the same
  rule, including mode changes while the requested format remains enabled.
• Add checked caller-owned workspace entry points for single/four-stream
  Huffman and ANS literal decoding, with size/alignment queries. Existing
  allocating wrappers remain available. SEQ/S/T decoding reuses its existing
  48 KiB sequence-table region during the earlier literal phase, removing
  one nested allocation and up to 17,412 bytes of peak table storage. Whole
  frame decoding still allocates, and legacy context tables are unchanged.
  Single/four-stream ANS literal decoding also uses the existing direct-table
  builder, removing its separate 4 KiB spread array from the stack. This
  reduces those individual frames, not the whole codec's worst-case stack.
• Add VV_DISABLE_SIMD=1 and a clean make SIMD=0 build mode. Scalar copy
  calls bypass runtime dispatch and its mutable state. make scalar-test
  separately builds the integer core with general-purpose registers only
  and runs eleven userspace regression suites. The normal x86-64 decoder
  still uses inline AVX2; it is not a runtime-portable binary for older CPUs.
• Add a caller-owned FAST context for independent inputs up to 64 KiB. It
  separates size/alignment/init from compression, performs no hot-path heap
  allocation, resets history on every call and preserves one-shot output
  bytes. Compact 16-bit matcher positions reduce queried workspace to 537,800
  bytes for a 4 KiB limit and 722,361 bytes for 64 KiB. Roots clear sparsely
  below 4 KiB and densely at and above 4 KiB. Even after this reduction the
  context is too large for a credible per-CPU zram proposal without more work.
• Check frame-footer capacity before writing, accept a null zero-length XXH64
  input without pointer arithmetic, and validate checksum tails and decoder
  spans by remaining length before advancing input pointers. Both AVX2
  prefetch phases validate literal capacity and match history before forming a
  lookahead pointer. New regressions exercise exact/truncated endpoints,
  oversized frame/block lengths, multiframe boundaries, streaming-capacity
  retry and both prefetch phases. These are invariant hardening changes, not a
  claim that an observed crash was repaired; valid encoded bytes are unchanged.
• Add workspace ownership/capacity/reuse regressions. On Linux, the normal
  test target injects allocation failures to require zero allocations in
  workspace literal calls and exactly one allocation in tested S/T decode
  paths. Exact-buffer, sanitizer and MemorySanitizer checks cover the new
  small-input initialization and format-selection paths.
• Add a reproducible page-sized in-process comparison harness with explicit
  one-shot API, framing, checksum and allocation caveats. Refresh English
  and Portuguese documentation while retaining historical evidence under
  its original version. Document the remaining licensing, stack, allocator,
  architecture and human-review requirements for a possible kernel port.
  No Linux patch is submitted and no new formal proof is claimed.
• Final scalar page-profile measurements cover 216 profiles across 4/16/64 KiB
  and preserve 2,304 frames byte-for-byte. On 4 KiB synthetic text, LZ4 remains
  substantially faster and Zstd level 1 remains smaller and faster to encode;
  VaptVupt has the faster decode of the two measured Zstd settings but retains
  much more context memory. Six paired builds found strong 4 KiB context-encode
  gains on random, records and repeating inputs, alongside a 2.36% 64 KiB
  records decode-batch loss and a 12.24% 4 KiB text decode-p95 loss. The release
  does not claim to supersede LZ4 or Zstd.

Source archive SHA-256: dbe268feaf120f295066fed85f9ef2c14f157bb96773aa5a46ece6c0198b7563

VaptVupt v2.65.10

Performance and correctness maintenance. The wire layout is unchanged, and
valid one-shot compressed output remains byte-identical on the paired
benchmark fixtures. Corrupt encodings accepted by earlier decoders may now
return an error. Streaming reset between format versions now produces the
same bytes as a newly created context with the selected options.

• Allocate secondary hash4 matcher tables only when adaptive binary encoding
  uses them. Fast encoding, text paths, adaptive trials, optimal-parser
  prepasses and streaming avoid an unused 256 KiB map plus a window-sized
  chain: 512 KiB per default-window matcher, or 64.25 MiB at window_log=24.
  These are allocation savings, not guaranteed RSS reductions.
• Seven alternating baseline/candidate in-process runs on CPU 4 with GCC
  14.3 measured fast encoding gains of 32.3% on 1 KiB text, 19.7% on 4 KiB
  text and 17.2% on 1 MiB random data; balanced random encoding improved
  39.6%. Large text was approximately unchanged. Fixture compressed sizes
  and hashes matched. These workload-specific measurements are separate
  from the generated-v1 CLI comparison table.
• Fixed data corruption when vv_cstream_reset changes from format v1 to
  v2: reset now updates the representable match-length limit and hash3
  configuration, allocating before accepting new options. Regression cases
  exercise repeated v1/v2 transitions in balanced and extreme modes, with
  long matches and three-byte records, against fresh-context output.
• Token length extensions now require a final byte below 255, including an
  explicit zero for a zero extension. Classic LZ and legacy entropy token
  decoders also enforce the frame's advertised history window. Regressions
  cover one-shot and fragmented streaming, scalar tails, AVX2 paths, and
  valid boundary controls. Completed streaming decode continues to report
  cumulative written on subsequent calls.
• Huffman encoding returns VVH_ERR_OVERFLOW when its bit writer exhausts
  the output buffer. The accumulator remains bounded, preventing the
  undefined shift reproduced with small destination buffers. Single- and
  four-stream decoding reject codewords whose bits extend beyond input.
• Validate normalized ANS frequencies sum to 4096 before constructing
  single-stream, four-stream, global-context or local-context literal
  tables. This rejects incomplete tables that could leave decoder state
  uninitialized, and rejects overfull tables before use.
• CLI numeric arguments now require a complete, nonnegative decimal value
  within range. Unknown modes, conflicting actions and multiple inputs are
  rejected. Buffered write/close errors return failure, so a full disk
  cannot be reported as successful compression or decompression. Empty
  files no longer depend on implementation-specific malloc(0) behavior.
• make test propagates failures from every Python check. Header changes
  rebuild CLI/test consumers, core changes rebuild fuzz harnesses, and the
  SIMD fuzz object is instrumented with the selected sanitizers.
• Corrected the ratio gate's ultra_fast JSON column to invoke CLI fast:
  the old unknown CLI name silently selected balanced mode. The gate now
  checks the emitted mode byte. Its fast baseline is corrected using
  v2.65.9 fast output, verified byte-identical to this release on all ten
  fixtures; balanced/extreme pins are unchanged. This is a measurement
  correction, not a compression-ratio regression.
• Updated English and Portuguese documentation and comparison evidence.
  The copied read_ext_len formal harnesses follow the changed helper;
  historical proofs for that helper are not claimed for the new code
  without rerunning the tools. The release artifact remains source-only.

Source archive: vaptvupt-2.65.10-src.tar.gz

SHA-256: 50cdb343010f3549d518cd5bb0b84e87aedc1e42de2940ba392b11b9ccc7591e

Signed tag: v2.65.10; commit: 658e226f0fd4f4b0a2600535ee55414fe328a326.

VaptVupt v2.65.9

Performance, correctness, and release-infrastructure maintenance. The frame
layout is unchanged from v2.65.8, and streams v2.65.8 encoded validly remain
compatible. The rare formerly undecodable SEQ candidate described below now
selects a different, lossless fallback representation.

• Sequence tANS decode tables are now built directly in their final storage
  rather than through a 4 KiB spread array. Per-block sequence-table scratch
  drops from 52 KiB to 48 KiB. A test compares the direct and historical
  builders entry-for-entry across 256 deterministic normalized tables, and the
  established ANS, roundtrip, SEQ, safe-zone, and exact-buffer regressions pass.
• Fixed a rare SEQ-encoder data-integrity defect. The wire carries one global
  match_count, so a literal run over 65,535 bytes before a later match cannot
  be split into a zero-match midstream entry: the decoder assigns real matches
  to the first match_count LL entries. The encoder now rejects that ambiguous
  SEQ candidate and losslessly falls back to another block representation;
  terminal long literal runs remain representable as trailing LL-only entries.
  test_seq_v2 is now 21/21 with a direct rejection case and a deterministic
  end-to-end sparse/random reproducer.
• Paired pinned in-process decode measurements found +0.40% on text and +1.21%
  on JSON, about +0.80% geometric mean. This is a modest, workload-dependent
  improvement; it does not change the stream or justify a universal speed
  claim.
• Streaming decode now performs the selected BCJ inverse exactly once when a
  frame completes: after a present footer and checksum have been validated, or
  immediately after the final block for a checksumless frame. Permanent
  roundtrips cover whole and split input for both x86 and AArch64 filters with
  checksum on and off.
• vv_compress now rejects mode values outside the three public enum constants
  and rejects simultaneous x86 and AArch64 filter requests, including for
  empty input, with VV_ERR_PARAM. The existing 10..24 window-range check now
  runs before any requested BCJ allocation or transform. One-shot decode,
  streaming decode, and frame-info parsing reject input headers that set both
  BCJ architecture bits. The streaming encoder, which cannot transform a whole
  frame while emitting incremental blocks, now rejects invalid modes and all
  BCJ options instead of silently accepting ignored filter requests.
• Restored reference-decoder parity: Python and JavaScript now consume trailing
  LL-only entries after all matches with the C-equivalent iteration bound,
  avoiding both premature termination and corrupt-input hangs. Both references
  reject contradictory dual-BCJ headers, implement the exact x86/AArch64 BCJ
  inverse after checksum validation, and reproduce current encoder output in
  checksum-on/off cross-language fixtures. C remains canonical for legacy
  H/A/I/C; Python retains limited A-tag support and JavaScript omits the legacy
  tags.
• The OOM sweep now requires its randomized baseline fixture to roundtrip before
  allocation failure is injected, so a codec correctness regression cannot be
  misreported as an injector-initialization failure.
• The one-shot BCJ path now calls vv_secure_zero on its private full-input
  working copy before free(), bringing it under the tracked plaintext-buffer
  hygiene policy. test_secure_zero exercises BCJ cleanup completion and
  byte-exact roundtrip under sanitizers; it does not directly inspect freed
  memory contents.
• Corrected CLI help for -A 0: zero selects the automatic factor, fast=2 and
  balanced/extreme=1; it is not the old hard-off setting.
• Added the deterministic generated-v1 competitive suite. It generates four
  dependency-free fixtures, requires the vv-fast/vv-balanced/lz4-1/zstd-1/
  zstd-3 matrix by default, reports medians after configurable warm-ups, checks
  every decode by SHA-256, and emits CSV plus provenance-rich JSON. The README
  and comparison guide carry a fresh seven-run pinned-core table separately
  from the retained July 2026 11-file history.
• Updated release, integration, format, security, and verification documents
  for v2.65.9. The formal-audit update records regression/dynamic validation of
  the delta and explicitly inherits the earlier formal baseline; it does not
  claim a new full formal-tool run. Release publication now lists only the
  source tar.gz artifact.

Source archive

• vaptvupt-2.65.9-src.tar.gz
• SHA-256: 436e988b53ce29d3ffa9c02e5763a78002b44b02ef78a8573a581e79f83e0a53
• The Git tag is signed; verify it with git tag -v v2.65.9.

v2.65.6 — Sprint 136: documentation refresh (docs only)

Documentation only; no source, wire-format, or output change (the tree
compiles to a v2.65.5-identical binary). Brings the docs current after
the v2.65.1-v2.65.5 releases:

• README: regenerated the head-to-head table and win/loss notes from a
  fresh v2.65.6 measurement (ratios unchanged since v2.65.0 — output is
  byte-identical — with corrected decode-speed comparisons and the
  vv-extreme encode-speed column now ~2 MB/s per v2.65.2); replaced the
  out-of-order, incomplete version-notes pile with a clean
  newest-first "recent releases" summary through v2.65.6.
• DEPLOY.md: corrected stale release-artifact names that were still
  pinned to 2.61.1 (tarball, bundle, binary, and vcpkg version) while
  the release is v2.65.x.
• bench/COMPARISON.md: added a currency note that the v2.65.0 tables
  hold through v2.65.6 (byte-identical output), with extreme encode
  speed the only moved column.
• SECURITY.md: document version to 2.11 / codebase v2.65.6.

Verified: reference-decoder guard passes, ratio gate +-0, current build
clean.

v2.65.5 — Sprint 135: reference-decoder default-format guard in make test

Test-infrastructure only; the shipping codec, wire format, and all
outputs are unchanged.

Sprint 134 fixed the reference decoders' missing HUFFMAN4 support — a
gap that had let the differential cross-check silently skip default
encoder output. This release makes that class of gap impossible to
reintroduce unnoticed: a new guard, tests/reference_roundtrip.py,
now runs in both make test and make python-test. It

• builds deterministic literal-heavy fixtures and compresses them in
  balanced and extreme mode,
• walks the .vv container and FAILS if no 'S'/'T' block with
  lit_fmt = 4 was produced (so the guard cannot pass vacuously if
  encoder format selection changes),
• decodes every stream with the Python reference decoder and compares
  byte-for-byte, and
• when node is available, repeats the byte-exact check with the
  JavaScript reference decoder via tests/reference_decode_check.js.

The guard's failure path is verified both ways: the pre-Sprint-134
Python reference fails it with NotImplementedError, and the JS checker
exits non-zero on an intentional plaintext mismatch.

Validation: guard passes 8/8 checks with 4 HUFFMAN4 blocks exercised;
22/22 C suites; 5,200 differential fuzz cases; JS reference suite
17/17; ratio gate ±0.

v2.65.4 — Sprint 134: fix broken amalgamation build + complete the reference decoders

Two correctness fixes to build and test infrastructure. The shipping
codec and wire format are unchanged.

Amalgamation (single-file build) was broken. The make amalg and
make amalg-verify recipes carried a hardcoded source/header list that
was never updated when the BCJ branch filter (vv_bcj.c / vv_bcj.h)
became a core source in v2.60.x. Since vv_compress calls
vv_bcj_detect / vv_bcj_x86 / vv_bcj_arm64 and uses
vv_filter_kind_t, the amalgamated build/vaptvupt.c failed to
compile (unknown type name 'vv_filter_kind_t', implicit-declaration
errors). The single-file build is a documented feature; both recipes
now include include/vv_bcj.h and src/vv_bcj.c. Verified: the
amalgamation compiles clean under -Wall -Wextra and round-trips all
three modes plus the --auto-filter BCJ path byte-exact.

Reference decoders now cover the default literal format. The Python
(reference/vv_huffman.py, reference/vv_ans.py) and JavaScript
(reference/vv_decoder.js) reference decoders raised
NotImplementedError on lit_fmt = 4 (HUFFMAN4, the 4-stream
interleaved Huffman format the encoder selects by default for blocks
with ≥1024 literals since v2.47.0). They therefore could not decode
typical modern output, and the differential fuzzer silently skipped
those cases — the independent cross-check did not cover the default
format. Both decoders now implement vvh_decode4 per FORMAT.md §3.4.1
(shared code table, 9-byte stream-size header, byte-aligned streams,
round-robin symbol interleave), reusing each implementation's existing
Huffman primitives. Validated byte-exact: both reference decoders
decode the full 11-file benchmark corpus in balanced and extreme mode
(22/22 each), independently confirming the C decoder on the default
format. The README's "byte-exact reference decoders … cross-check C
against Python" is now accurate for the default format.

Validation: 22/22 C test suites under -O3 -flto and under
-fsanitize=address,undefined -fno-sanitize-recover=all; ratio gate
±0 (C unchanged); 5,200 differential fuzz cases consistent; JS
reference suite 17/17; both reference decoders byte-exact on the corpus.

v2.65.3 — Sprint 133: cap the extreme prepass window allocation (memory hygiene)

Output byte-identical to v2.65.0/1/2 across the corpus, the ratio gate
(±0), and the fixture suite; wire format unchanged. This is a
memory-robustness fix, not a speed change.

The residual-literal prepass added in v2.65.0 (Sprint 130) allocated
its throwaway matcher at the full encode window — up to wlog=24, i.e.
2 × 2^24 × 4 = 128 MB of chain arrays reserved per block. But the
prepass compresses ONE block (≤ VV_MAX_BLOCK_SIZE = 2^20) with a fresh
matcher, so every match it can find is intra-block, distance < 2^20; a
wlog-20 window covers that exactly, with non-aliasing chain indices
across a ≤ 2^20-wide position span. The prepass matcher is now capped
at wlog=20, reserving 8 MB instead of up to 128 MB per block.

Because the chain arrays are lazily faulted (only the hash tables are
memset, and their size is window-independent), resident memory and
wall-clock are unchanged — the fix is to the VIRTUAL reservation.
That matters on overcommit-strict systems (vm.overcommit_memory=2,
where address-space reservation counts against the commit limit) and
in virtual-memory-limited containers, where a 128 MB-per-block reserve
on a large file could spuriously fail; it is dead weight everywhere
else. Output is identical (verified per file, by the ratio gate at ±0,
and by a byte-exact roundtrip on a 7 MB multi-block fixture).

Validation: 22/22 test suites under -O3 -flto and under
-fsanitize=address,undefined -fno-sanitize-recover=all; ratio gate
±0 bytes (output identity); 5,200 differential fuzz cases consistent;
27/27 negative-corpus cases; ASan+UBSan+LeakSanitizer roundtrip sweep
clean (11 corpus files + a 7 MB fixture × 3 modes, byte-exact).

v2.65.2 — Sprint 132: extreme-mode encode ~2x faster, byte-identical

Pure speedup of the extreme-mode optimal parser; output is
byte-identical to v2.65.0/1 (verified per file and by the ratio gate
at ±0), wire format unchanged.

Two changes in the candidate collector, both outcome-preserving:

• Rep-offset probes now extend through extend_match (8-byte xor/ctz
  stride) instead of a byte-at-a-time loop. The probe runs up to three
  times at every DP position.
• The hash-chain walk exits as soon as a LONG_MATCH-class candidate
  (>= 512) appears. The DP takes such a candidate immediately and
  ignores all others, so the remainder of the depth-256 walk — with an
  extend per prefix hit — was pure waste on repetitive regions.

Measured extreme encode (CLI, corpus rev 2): logs 1.1 -> 2.0 MB/s,
json 1.2 -> 2.2, xml 1.2 -> 2.2, text 4.0 -> 5.2, source 2.9 -> 4.3,
csv 1.3 -> 1.5. Lowering the LONG_MATCH threshold itself (256/128) was
also swept: size-neutral within ±130 bytes and only marginally faster,
so the threshold stays at 512 and the release stays byte-identical.

Validation: 22/22 test suites under -O3 -flto and under
-fsanitize=address,undefined -fno-sanitize-recover=all; ratio gate
±0 bytes (output identity); 5,200 differential fuzz cases consistent;
27/27 negative-corpus cases; ASan+UBSan+LeakSanitizer roundtrip sweep
clean (11 corpus files x 3 modes, byte-exact).

v2.65.1 — Sprint 131: OF-code price decomposition (negative result, default off)

Maintenance release. Output is byte-identical to v2.65.0 across the
full corpus, the ratio gate (±0), and the fixture suite; the wire
format is unchanged.

The optimal parser's match price is refactored into an explicit
decomposition — 8 (LL+ML sequence overhead) + OF-code bits + offset
extra bits — with the previous constants preserved exactly as the
prior (rep codes 2 bits, explicit codes 6 bits). The greedy prepass
now also classifies each match's OF code with the wire's exact rep
rules, producing a per-block OF-code histogram, and a new
VV_OPT_OF_BLEND knob (default 0) can blend measured code costs into
the price the way VV_OPT_LIT_BLEND does for literals.

The measured result is negative and is recorded so it is not re-tried
blind: blend 0 (the prior) beats every measured blend on the 11-file
corpus total (blends 2/4/6/8 of 8 land 0.01-0.08% worse). Measured OF
pricing helps plain text and source (~-1%) but hurts json and logs by
more — per-block OF distributions on this corpus do not deviate from
the prior enough to pay, unlike the literal distributions in v2.64.0/
v2.65.0. Blend 0 was verified to reproduce v2.65.0 byte-for-byte
before the sweep, so the refactor itself is exactly anchored.

Validation: 22/22 test suites under -O3 -flto; ratio gate ±0 bytes
(output identity); 5,200 differential fuzz cases consistent.

v2.65.0 — Sprint 130: residual-literal pricing via greedy prepass

Extreme-mode ratio release; balanced/fast output unchanged, wire format
version 1 unchanged. Completes the literal-repricing arc: v2.64.0
priced literals from the raw block histogram, which is dominated by
exactly the repetitive content that matches remove — it underestimates
the entropy of the RESIDUAL literal stream the coder actually sees.

The optimal parser now runs a depth-4 greedy prepass on a private
throwaway matcher (accel on, ~1% of the DP's runtime, no shared-state
pollution), histograms the literal bytes of its token stream, and
prices literals from that residual distribution (still blended with
the flat prior; the blend re-swept to 6/8 — the honest histogram
tolerates a stronger weight than the raw one did, and the mode
contract now holds with 20 bytes of headroom on the tightest fixture
instead of violating). Falls back to the raw histogram when the
prepass cannot run.

Measured, extreme mode (corpus rev 2, zero roundtrip mismatches; full
tables in bench/COMPARISON.md): json 485,218 -> 451,744 (-6.9%; now
4.6% smaller than zstd-9's 473,364), xml -1.9% (7.3% under zstd-9),
csv -0.8%, logs -0.2%, plain text +1.6% and source +1.9% (both cells
still beat balanced and were already zstd-9 losses). Extreme now takes
two of the six text-family files from zstd-9 outright. Encode speed
unchanged (prepass ~1%).

Ratio-gate baseline regenerated (documented --update flow): json-mixed
and csv improve again; text-simple/varied/large give back 28/80/128
bytes; the known source-like contract violation narrows (849 -> 845).

Validation: 22/22 test suites under -O3 -flto and under
-fsanitize=address,undefined -fno-sanitize-recover=all; 5,200
differential fuzz cases consistent; 27/27 negative-corpus cases;
ASan+UBSan+LeakSanitizer roundtrip sweep clean (11 corpus files x 3
modes, byte-exact).