cast(double/float as decimal) loses precision: the native kernel scales the float by 10^scale

[ShreyeshArangath]

Describe the bug

Casting a double (or float) to a decimal whose scale exceeds roughly 17 produces wrong
digits from about the 18th significant digit onwards. There is no exception and no fallback — the
result is silently corrupted.

cast_impl in native-engine/datafusion-ext-commons/src/arrow/cast.rs has a dedicated arm for
Utf8 -> Decimal128, but none for Float32/Float64 -> Decimal128. Those therefore fall into the
catch-all _ => arrow::compute::kernels::cast::cast(array, cast_type), and Arrow's float-to-decimal
kernel computes value * 10^scale in IEEE-754 double arithmetic. A f64 holds only ~15–17
significant decimal digits, so everything past that is floating-point noise written into the
decimal as if it were data.

Spark never does floating-point arithmetic at the target scale. Cast.castToDecimal for a
FractionalType evaluates Decimal(fractional.toDouble(b)), and Decimal(Double) is
BigDecimal.valueOf(d) == new BigDecimal(Double.toString(d)) — the shortest decimal string
that round-trips. The rest of the target scale is zeros.

To Reproduce

SELECT CAST(CAST(0.034567890 AS DOUBLE) AS DECIMAL(38,33));

	result
Spark	0.034567890000000000000000000000000
Auron	0.034567890000000002231853683572736

The digits are not random: 0.034567890 * 1e33 evaluated in f64 is
3.4567890000000000223185368357273600e31, and truncating that to i128 gives exactly the
observed unscaled value. Note this is not the binary expansion of the double, which is
0.03456788999999999695... — the noise comes from the scaled multiply, not from the input.

Spark's own DataFrameSuite test "SPARK-22271: mean overflows and returns null for some decimal variables" fails on this.

Expected behavior

cast(<double> as decimal(p,s)) should match Spark: format the double to its shortest
round-trip decimal representation, then rescale that exactly.

Affected versions

Reproduced on Auron 7.0.0-incubating against Spark 3.1.1 and Spark 3.5.2. The gap is still present
on master (0f98674e) — git grep Decimal128 native-engine/datafusion-ext-commons/src/arrow/cast.rs returns only the Utf8 arm and test lines.

Affected code

native-engine/datafusion-ext-commons/src/arrow/cast.rs — the Utf8 -> Decimal128 arm (~line 222)
is where the float arm should sit; the catch-all _ => arrow::compute::kernels::cast::cast(...)
(~line 321) is what currently swallows it.

Suggested fix

Add a Float32 | Float64 -> Decimal128 arm that mirrors Spark:

• widen f32 to f64 first — Spark's fractional.toDouble(b) does the same, so
  cast(0.1f as decimal(20,18)) must remain 0.100000001490116120, not 0.1;
• format with Rust's shortest round-trip f64 Display, which never uses exponent notation;
• hand the resulting string to the existing string-to-decimal kernel, so rounding (half-up) and
  precision-overflow behaviour stay identical to the already-validated string cast path;
• map non-finite values to None. Spark's BigDecimal.valueOf(NaN) raises
  NumberFormatException, which Cast catches and turns into null.

Workaround

Cast the double to string first — cast(cast(d as string) as decimal(38,33)) — which routes
through the correct Utf8 -> Decimal128 arm.

[Sigma-Ma]

Hi @ShreyeshArangath, I'd like to work on this.

I think the cleanest fix is to handle Float32/Float64 -> Decimal128 explicitly in cast_impl: widen f32 to f64, format it as the shortest round-trip decimal string, then reuse the existing string-to-decimal path. This avoids floating-point scaling noise while preserving the current rounding and overflow behavior. I'll also add regression tests for high-scale casts, nulls, and non-finite values.

[ShreyeshArangath]

@Sigma-Ma assigned this to you

[Sigma-Ma]

Thanks for assigning this to me. I've opened #2496 with the fix.

The patch routes float-to-decimal casts through the existing string-to-decimal path, avoiding the floating-point scaling noise while preserving Spark's rounding and overflow behavior. It also adds coverage for high-scale casts, f32 widening, NULL/non-finite inputs, and re-enables the SPARK-22271 regression test for Spark 3.1–3.5.