Describe the bug
In ANSI mode, cast_float_to_int32_up (native/spark-expr/src/conversion_funcs/numeric.rs:395-396, same shape in cast_float_to_int16_down at :333) detects overflow as
let is_overflow = value.is_nan() || value.abs() as $rust_dest_type == $max_dest_val;
i.e. "the saturating conversion of |value| landed on MAX". That flags every double in [2147483647.0, 2147483648.0) and (-2147483649.0, -2147483647.0] as overflow, including INT_MAX and INT_MIN themselves, which are exactly representable doubles and valid ints. Spark's DoubleExactNumeric.toInt / toLong accept any x with Math.floor(x) <= MaxValue && Math.ceil(x) >= MinValue and return x.toInt / x.toLong; for BIGINT Long.MaxValue.toDouble == 2^63, so Spark accepts ±2^63 and saturates.
Steps to reproduce
SET spark.sql.ansi.enabled = true;
CREATE TABLE t USING parquet AS SELECT * FROM VALUES (2147483647.0D), (-2147483648.0D), (2147483647.5D) AS t(c);
SELECT CAST(c AS INT) FROM t;
SELECT CAST(9223372036854775808.0D AS BIGINT), CAST(-9223372036854775808.0D AS BIGINT);
Spark: 2147483647, -2147483648, 2147483647; 9223372036854775807, -9223372036854775808. Comet: CAST_OVERFLOW error for every one of them. Same for FLOAT sources.
Expected behavior
Match Spark's bound check: only values outside [MinValue, MaxValue] after floor/ceil overflow.
Proposed solution
let is_overflow = value.is_nan()
|| !(value.floor() <= $max_dest_val as $float_ty && value.ceil() >= $min_dest_val as $float_ty);
// then `value as $rust_dest_type` — Rust `as` saturates exactly like JVM d2i/d2l
Apply to both macros and add the boundary values to CometNativeCastSuite in ANSI mode.
Additional context
Default-on: CometCast.canCastFromDouble / canCastFromFloat mark Int/Long as Compatible in all eval modes. The check dates from #350; #4941 and the #5128 plan intend to keep "the saturation-based overflow detection exactly as written", so a performance rewrite would carry the bug forward.
Describe the bug
In ANSI mode,
cast_float_to_int32_up(native/spark-expr/src/conversion_funcs/numeric.rs:395-396, same shape incast_float_to_int16_downat:333) detects overflow asi.e. "the saturating conversion of |value| landed on MAX". That flags every double in
[2147483647.0, 2147483648.0)and(-2147483649.0, -2147483647.0]as overflow, includingINT_MAXandINT_MINthemselves, which are exactly representable doubles and valid ints. Spark'sDoubleExactNumeric.toInt/toLongaccept anyxwithMath.floor(x) <= MaxValue && Math.ceil(x) >= MinValueand returnx.toInt/x.toLong; for BIGINTLong.MaxValue.toDouble == 2^63, so Spark accepts±2^63and saturates.Steps to reproduce
Spark:
2147483647,-2147483648,2147483647;9223372036854775807,-9223372036854775808. Comet:CAST_OVERFLOWerror for every one of them. Same for FLOAT sources.Expected behavior
Match Spark's bound check: only values outside
[MinValue, MaxValue]after floor/ceil overflow.Proposed solution
Apply to both macros and add the boundary values to
CometNativeCastSuitein ANSI mode.Additional context
Default-on:
CometCast.canCastFromDouble/canCastFromFloatmark Int/Long as Compatible in all eval modes. The check dates from #350; #4941 and the #5128 plan intend to keep "the saturation-based overflow detection exactly as written", so a performance rewrite would carry the bug forward.