Fix remaining_length accounting after Huffman decode in regression/composed predictors - #2
Conversation
…mposed predictors
RegressionPredictor::load decremented remaining_length by the uncompressed
index count (coeff_size * sizeof(int)) after Huffman decode(), but decode()
advances the read pointer only by the compressed stream size. The uncompressed
count overshoots the compressed stream, understating remaining_length for the
subsequent quantizer encoder.load(), whose bound check then spuriously rejects
valid data with "SZ3 Huffman: tree exceeds compressed buffer".
RegressionPredictor is only used by ALGO_LORENZO_REG, so a column compressed
with CODEC(SZ3('ALGO_LORENZO_REG', ...)) could be written on insert but fail
every subsequent read with CORRUPTED_DATA (effective data loss). Small element
counts are affected; large counts happen to leave enough slack to survive the
understated bound.
Account for exactly the bytes decode() consumed via pointer difference.
ComposedPredictor::load had the mirror defect (it never decremented
remaining_length for its selection stream); tightened the same way.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
|
|
|
cc @alexey-milovidov @scanhex12 — could you review? This fixes the ALGO_LORENZO_REG data-loss reported in ClickHouse/ClickHouse#110336 (small element counts fail every read with |
Advance the `contrib/sz3` submodule from `dda0caee` to `ff760cc`, the current tip of the `ClickHouse/v3.3.2` branch. In addition to the quantizer NaN cast fix already pinned, this brings in the fix for `remaining_length` accounting after Huffman `decode` in `RegressionPredictor`/`ComposedPredictor` (ClickHouse/SZ3#2), which otherwise made a column compressed with `CODEC(SZ3('ALGO_LORENZO_REG', ...))` fail to decompress after being written. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Fixes a
remaining_lengthbyte-accounting bug that causes silent data loss for theALGO_LORENZO_REGcompression algorithm when used via the ClickHouse SZ3 codec (ClickHouse/ClickHouse#110336).Root cause
HuffmanEncoder::decode()advances the read pointer past the encoded stream but does not update the caller-ownedremaining_lengthbound. Callers must account for the consumed bytes themselves.RegressionPredictor::load()did this incorrectly:coeff_size * sizeof(int)is the uncompressed index count in bytes, butdecode()only advancedcby the Huffman-compressed stream size (sizeof(size_t) + encodedLength). Since the compressed stream is smaller, this over-subtracts and understatesremaining_lengthfor the following main-quantizerencoder.load(), whose tree bound checktree_bytes > remaining_lengththen spuriously throwsSZ3 Huffman: tree exceeds compressed buffer.RegressionPredictoris only used byALGO_LORENZO_REG, soALGO_INTERP/ALGO_INTERP_LORENZOare unaffected. Small element counts fail deterministically; large counts happen to leave enough slack in the main stream to survive the understated bound (in ClickHouse this shows up as: n=500 fails, n=5000 works).Fix
Account for exactly the bytes
decode()consumed, via pointer difference:ComposedPredictor::load()had the mirror defect (it decoded its selection stream but never decrementedremaining_lengthat all, leaving a loose bound). Tightened the same way.