Skip to content

fix: fall back to Spark for negative-scale decimal casts and arithmetic - #5050

Closed
0lai0 wants to merge 2 commits into
apache:mainfrom
0lai0:fix-5013-negative-scale-decimal-cast
Closed

fix: fall back to Spark for negative-scale decimal casts and arithmetic#5050
0lai0 wants to merge 2 commits into
apache:mainfrom
0lai0:fix-5013-negative-scale-decimal-cast

Conversation

@0lai0

@0lai0 0lai0 commented Jul 27, 2026

Copy link
Copy Markdown
Contributor

Which issue does this PR close?

Closes #5013

Rationale for this change

With spark.sql.legacy.allowNegativeScaleOfDecimal=true, Spark allows DecimalType(p, s) with s < 0. Comet's native cast and decimal arithmetic kernels scale-align values by computing 10^|scale| (e.g. 10_i128.pow(scale as u32)). When scale is negative, casting it to u32 produces a huge value, so the power computation overflows: it panics (attempt to multiply/subtract with overflow) in debug builds and silently wraps to an incorrect value in release builds (the native workspace sets overflow-checks = false for release).

This is reachable from two places:

  • Casting an integer type (Byte/Short/Integer/Long) to/from a negative-scale DecimalType, or casting a negative-scale DecimalType to TimestampType, both hit the same 10^|scale| computation in numeric.rs / cast_decimal_to_timestamp.
  • Arithmetic on negative-scale decimals (Add/Subtract/Divide/IntegralDivide/Remainder), which scale-aligns operands the same way.

CometCast already special-cased negative-scale decimals for the Decimal -> String direction, but that guard was unreachable in practice because producing the negative-scale value in the first place would panic before a -> String cast could ever run.

Since allowNegativeScaleOfDecimal is a rarely-used legacy flag (default false), and the native kernels don't implement negative-scale-safe rescaling, the fix makes Comet detect these cases at planning time and fall back to Spark instead of attempting them natively.

What changes are included in this PR?

  • CometCast.scala: added negativeScaleDecimalCastReason and marked as Unsupported:

    • int -> Decimal(neg) via canCastFromByte/Short/Int/Long
    • Decimal(neg) -> int and Decimal(neg) -> Timestamp via canCastFromDecimal (now takes the source DecimalType so it can check fromType.scale)

    All other cast directions to/from negative-scale decimals (e.g. String <-> Decimal(neg), Decimal(neg) -> Float/Double/String/Boolean/wider-Decimal) don't perform integer scale-alignment and are unaffected, they keep running natively.

  • arithmetic.scala: added negScaleDecimalRejection(expr: BinaryArithmetic) in MathBase, returning Unsupported when the left operand, right operand, or result type is a negative-scale DecimalType. Wired into CometAdd, CometSubtract, CometDivide, CometIntegralDivide, CometRemainder.

    CometMultiply and UnaryMinus are intentionally left unguarded, neither scale-aligns, so they stay native

How are these changes tested?

Added tests to CometCastSuite and CometExpressionSuite
make core
./mvnw -pl spark test -Dsuites='org.apache.comet.CometCastSuite'
./mvnw -pl spark test -Dsuites='org.apache.comet.CometExpressionSuite'

Note: the TimestampType case is not called out in the linked issue's reproducer, it was found while auditing the same 10^|scale| code path during this fix and is included here since it shares the exact same root cause and fix.

@0lai0
0lai0 force-pushed the fix-5013-negative-scale-decimal-cast branch from f4a027d to e767210 Compare July 27, 2026 14:35
@andygrove

Copy link
Copy Markdown
Member

This may overlap with #4799

@andygrove
andygrove requested a review from comphead July 27, 2026 14:53
@0lai0

0lai0 commented Jul 27, 2026

Copy link
Copy Markdown
Contributor Author

Thanks @andygrove and @comphead, sorry for missing #4799! That would be great if it covers it.
I'd be glad to help review #4799.

@0lai0 0lai0 closed this Jul 27, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Native panic casting to negative-scale decimal when spark.sql.legacy.allowNegativeScaleOfDecimal=true

2 participants