fix(vortex-row): decimal sort keys are not memcmp-comparable across chunks

[HarukiMoriarty]

The bug

ORDER BY / top-k on a decimal column silently returns wrongly ordered rows whenever the column spans chunks whose physical value widths differ. No error is raised; the output looks plausible.

Example

Take a DECIMAL(7, 5) tip column and ORDER BY tip DESC. Two rows land in different chunks, and compression picks a different physical width for each chunk:

row	unscaled value	chunk's physical type
tip = 0.00484	484	i16 (484 doesn't fit i8)
tip = 0.00091	91	i8 (every value in its chunk fits i8)

A descending sort key is built per chunk: write the value big-endian, flip the sign bit, invert the value bytes (for DESC), and prefix the non-null sentinel 0xFE. Before this fix, the number of value bytes came from the chunk's type:

tip = 0.00484, encoded at its chunk's width (i16 → 2 value bytes)

    big-endian bytes      01 E4
    flip sign bit         81 E4
    invert for DESC       7E 1B
    prepend sentinel   FE 7E 1B      ← 3-byte key

tip = 0.00091, encoded at its chunk's width (i8 → 1 value byte)

    big-endian bytes      5B
    flip sign bit         DB
    invert for DESC       24
    prepend sentinel   FE 24         ← 2-byte key

The sorter compares keys with memcmp:

key(0.00484) = FE 7E 1B
key(0.00091) = FE 24
               ─┬ ─┬
                │  └─ byte 1 decides: 0x24 < 0x7E, so key(0.00091) is the smaller key
                └─ byte 0: equal

But byte 1 means different things in the two keys: in the 3-byte key it is the high-order byte of a two-byte number, while in the 2-byte key it is the only byte of a one-byte number. The comparison is meaningless — and DESC encoding promises bigger value ⇒ smaller key, so the smaller key wins: the sorter ranks 0.00091 as a larger tip than 0.00484. Ascending breaks symmetrically.

With this fix, both chunks encode at the width the declared dtype implies (DECIMAL(7,5) → i32 → 4 value bytes), whatever their physical storage:

key(0.00484) = FE 7F FF FE 1B
key(0.00091) = FE 7F FF FF A4
                        ─┬
                         └─ byte 3 decides: 0xFE < 0xFF, so key(0.00484) is smaller

Equal-length keys, every byte position aligned — 0.00484 correctly sorts first in the descending order.

The fix

Derive the key width from the declared dtype, so every chunk of a column encodes identical-length keys:

• decimal_key_type (vortex-row): the dtype-derived width, used by both the sizing pass and the encode dispatch.
• converted_buffer<W> (vortex-array, next to widened_buffer): returns a chunk's values at exactly width W. Zero-copy when the chunk is already stored at W (the common case), lossless widening otherwise, and an error for any value that violates its declared precision, such values previously encoded a garbage key silently.

The encode loop itself is unchanged; only where it reads its values from changed.

[a10y]

thank you for finding this! Let's please add a regression test in vortex-row, just the case from the PR description is a good starting point

[a10y]

We probably should have a fuzz target too for vortex-row :/ Doesn't need to be in this PR but probably a good follow up

Additionally, had claude take a look, it notes

It also quietly fixes a second, worse latent bug the PR body doesn't mention: for decimals nested in structs/FSLs, the slot width was already computed from the declared dtype (row_width_for_dtype, codec.rs:797, codec.rs:846) while encode_decimal wrote bytes at the chunk's physical width. A nested decimal chunk stored at a non-declared width would desynchronize col_offset and corrupt every subsequent column's bytes in the row.

[codspeed-hq]

Merging this PR will not alter performance

✅ 1849 untouched benchmarks
⏩ 46 skipped benchmarks¹

---

_{Comparing nemo/row-decimal-key-width (e0a2a24) with develop (d9619b8)}

Footnotes

1. 46 benchmarks were skipped, so the baseline results were used instead. If they were deleted from the codebase, click here and archive them to remove them from the performance reports. ↩

[a10y]

is this true? this still forces a branch in the inner loop

[a10y]

i think we should handle when validity is alltrue or nonnullable without a branch, and then if the validity is real Mask then we do the branch

[a10y]

Looks good!