[Bug] Decimal z-order values are not order-preserving and zero collides with the null sentinel

[LuciferYang]

Search before asking

• I searched in the issues and found nothing similar.

Paimon version

master, 475be566f (2.1-SNAPSHOT).

Compute Engine

Flink and the Java API, wherever clustering or sort-compact runs: append/cluster/ZorderSorter on the write path and CALL sys.compact(..., order_strategy => 'zorder', ...).

Minimal reproduce step

Cluster a table by a DECIMAL column and compare the z-values two rows get. ZIndexer's decimal branch wrote the unscaled value as a minimal two's-complement byte array into the fixed 8-byte z-value buffer:

return o == null
        ? NULL_BYTES
        : ZOrderByteUtils.byteTruncateOrFill(
                        ((Decimal) o).toUnscaledBytes(), PRIMITIVE_BUFFER_SIZE, reuse)
                .array();

byteTruncateOrFill left-aligns and pads with 0x00, and z-values are compared unsigned, so for DECIMAL(20,2):

• -1.00 gives unscaled -100, one byte 0x9C, padded to 9C 00 00 00 00 00 00 00
• 1.00 gives unscaled 100, one byte 0x64, padded to 64 00 00 00 00 00 00 00

0x9C is 156 unsigned and 0x64 is 100, so -1.00 sorts above 1.00. Length makes it worse: 100.00 is unscaled 10000, two bytes 27 10, which sorts below the one-byte 1.00. And 0.00 gives a single 0x00 byte padded to eight zero bytes, which is byte-for-byte the NULL_BYTES sentinel, so zero and NULL are indistinguishable. toUnscaledBytes is toBigDecimal().unscaledValue().toByteArray() for every precision, so this is not limited to the wide decimals.

What doesn't meet your expectations?

Clustering by a DECIMAL column should put nearby values near each other. Instead the ordering is scrambled inside each byte-length class and reversed across the sign, so the clustering does not help data skipping, and min/max statistics on the sorted output are no better than random. Results stay correct, since a z-value is only a sort key, but the feature does nothing useful for decimals.

Every other numeric type in that visitor goes through the sign-flipped fixed-width transform in ZOrderByteUtils (intToOrderedBytes, longToOrderedBytes, and so on), which is order-preserving under unsigned comparison and keeps zero away from the null sentinel. The decimal branch is the one that does not.

Anything else?

HilbertIndexer encodes decimals as toBigDecimal().longValue(), which drops the fractional part entirely, so every value with magnitude below 1 collapses to the same key. Different bug, same area.

Are you willing to submit a PR?

• I'm willing to submit a PR!