Search before asking
Paimon version
master, 475be566f (2.1-SNAPSHOT).
Compute Engine
Flink and Spark. Reached from sys.compact with order_strategy => 'zorder', from write-path clustering, and from Spark's z-order sort, which calls the same two functions through SparkZOrderUDF.
Minimal reproduce step
Z-order clustering on a FLOAT or DOUBLE column. The transform that is supposed to make the bit pattern unsigned-comparable shifts by the wrong width:
public static ByteBuffer doubleToOrderedBytes(double val, ByteBuffer reuse) {
ByteBuffer bytes = reuse(reuse, PRIMITIVE_BUFFER_SIZE);
long lval = Double.doubleToLongBits(val);
lval ^= ((lval >> (Integer.SIZE - 1)) | Long.MIN_VALUE);
bytes.putLong(lval);
return bytes;
}lval is 64 bits, so the sign extension has to come from a shift of 63. Shifting right by 31 puts bits 31 through 61 of the pattern into the low half of the mask instead, so every value bit that lines up with a set bit there is inverted.
For negatives that means the magnitude is only partly inverted, and adjacent values come out reversed:
-2.5 encodes to 0x3ffbffff80080000-2.5000000000000004, a strictly smaller value, encodes to 0x3ffbffff80080001
Positives are hit too, wherever two values differ at one of those bit positions:
2.5 encodes to 0xc0040000800800002.5000000002328306, a strictly larger value, encodes to 0xc004000080000000
Walking 2.5 upward in steps of 2^19 ULPs, three of every five adjacent pairs come out inverted. floatToOrderedBytes widens to double and runs the same line, so floats are affected the same way.
What doesn't meet your expectations?
Values that sort one way have to encode the other way round for clustering to mean anything. With this transform, FLOAT and DOUBLE columns get a z-value ordering that is scrambled within each exponent range, so clustering and the min/max statistics computed over the sorted output do not help skipping. Results stay correct, since a z-value is only a sort key.
The same transform written correctly is a few files away, in SortUtil: putFloatNormalizedKey shifts an int by Integer.SIZE - 1 and putDoubleNormalizedKey shifts a long by Long.SIZE - 1. The z-order copy kept the int-sized constant on a long.
Anything else?
The existing tests testFloatOrdering and testDoubleOrdering only feed random.nextFloat() and random.nextDouble(), which land in [0, 1) and rarely differ at an affected bit, so nothing caught this.
Are you willing to submit a PR?
Search before asking
Paimon version
master,
475be566f(2.1-SNAPSHOT).Compute Engine
Flink and Spark. Reached from
sys.compactwithorder_strategy => 'zorder', from write-path clustering, and from Spark's z-order sort, which calls the same two functions throughSparkZOrderUDF.Minimal reproduce step
Z-order clustering on a FLOAT or DOUBLE column. The transform that is supposed to make the bit pattern unsigned-comparable shifts by the wrong width:
lvalis 64 bits, so the sign extension has to come from a shift of 63. Shifting right by 31 puts bits 31 through 61 of the pattern into the low half of the mask instead, so every value bit that lines up with a set bit there is inverted.For negatives that means the magnitude is only partly inverted, and adjacent values come out reversed:
-2.5encodes to0x3ffbffff80080000-2.5000000000000004, a strictly smaller value, encodes to0x3ffbffff80080001Positives are hit too, wherever two values differ at one of those bit positions:
2.5encodes to0xc0040000800800002.5000000002328306, a strictly larger value, encodes to0xc004000080000000Walking 2.5 upward in steps of 2^19 ULPs, three of every five adjacent pairs come out inverted.
floatToOrderedByteswidens to double and runs the same line, so floats are affected the same way.What doesn't meet your expectations?
Values that sort one way have to encode the other way round for clustering to mean anything. With this transform, FLOAT and DOUBLE columns get a z-value ordering that is scrambled within each exponent range, so clustering and the min/max statistics computed over the sorted output do not help skipping. Results stay correct, since a z-value is only a sort key.
The same transform written correctly is a few files away, in
SortUtil:putFloatNormalizedKeyshifts anintbyInteger.SIZE - 1andputDoubleNormalizedKeyshifts alongbyLong.SIZE - 1. The z-order copy kept the int-sized constant on a long.Anything else?
The existing tests
testFloatOrderingandtestDoubleOrderingonly feedrandom.nextFloat()andrandom.nextDouble(), which land in [0, 1) and rarely differ at an affected bit, so nothing caught this.Are you willing to submit a PR?