Skip to content

[SPARK-58235][SQL] Fix Double/Float to Timestamp silent overflow clamping in non-ANSI mode - #57399

Open
jiangxt2 wants to merge 2 commits into
apache:masterfrom
jiangxt2:fix/double-to-timestamp-overflow
Open

[SPARK-58235][SQL] Fix Double/Float to Timestamp silent overflow clamping in non-ANSI mode#57399
jiangxt2 wants to merge 2 commits into
apache:masterfrom
jiangxt2:fix/double-to-timestamp-overflow

Conversation

@jiangxt2

Copy link
Copy Markdown

What changes were proposed in this pull request?

Add Long range validation for the Double/Float-to-Timestamp cast path in both interpreted and codegen execution.

When a Double or Float value multiplied by MICROS_PER_SECOND (1,000,000) exceeds Long range, the unchecked cast to long silently produces a clamped value (Long.MAX_VALUE or Long.MIN_VALUE). The fix detects this overflow and either returns NULL (non-ANSI mode) or throws CAST_OVERFLOW (ANSI mode).

The interpreted path (Cast.doubleToTimestamp) now checks the multiplication result against Long range before calling .toLong. The codegen path (castToTimestampCode Double/Float branches) applies the same range check before the (long) cast.

Why are the changes needed?

CAST(1e20 AS TIMESTAMP) silently produces +294247-01-10 04:00:54.775807 (clamped to Long.MAX_VALUE) instead of NULL in non-ANSI mode. This is inconsistent with other overflow behavior:

SET spark.sql.ansi.enabled=false;
SELECT CAST(1e20 AS TIMESTAMP);
-- Before fix: +294247-01-10 04:00:54.775807  (clamped to Long.MAX_VALUE)
-- After fix:  NULL

SELECT CAST(-1e20 AS TIMESTAMP);
-- Before fix: -290308-12-21 19:59:05.224192  (clamped to Long.MIN_VALUE)
-- After fix:  NULL

The ANSI mode path (doubleToTimestampAnsiDoubleExactNumeric.toLong) already correctly throws CAST_OVERFLOW. This bug only affects non-ANSI mode.

Does this PR introduce any user-facing change?

Yes. Previously, CAST(large_double AS TIMESTAMP) silently returned an incorrect value caused by numeric overflow. Now:

  • Non-ANSI mode: returns NULL (consistent with other overflow casts)
  • ANSI mode: throws CAST_OVERFLOW error (unchanged)

How was this patch tested?

Added unit tests in CastWithAnsiOffSuite and CastWithAnsiOnSuite:

Non-ANSI mode (CastWithAnsiOffSuite):

  • Positive overflow: 1E20 → null
  • Negative overflow: -1E20 → null
  • Boundary just under: 9223372036854.0 → passes through
  • Boundary just over: 9223372036855.0 → null
  • Negative boundary: -9223372036854.0 / -9223372036855.0
  • Finite input overflowing to +-Infinity: Double.MaxValue / -Double.MaxValue
  • NaN, Infinity, normal values, zero, Float overflow

ANSI mode (CastWithAnsiOnSuite):

  • Positive overflow: throws CAST_OVERFLOW with correct error parameters
  • Negative overflow: throws CAST_OVERFLOW with correct error parameters
  • Boundary values, normal values, Float overflow

Was this patch authored or co-authored using generative AI tooling?

Generated-by: Claude Code (Opus 4.8)

Comment thread docs/sql-migration-guide.md Outdated

@uros-b uros-b left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thank you @jiangxt2, please address comment before merge - otherwise LGTM! cc @MaxGekk @gengliangwang

@uros-b
uros-b requested review from MaxGekk and gengliangwang July 21, 2026 18:29
@jiangxt2

Copy link
Copy Markdown
Author

Thanks for catching this! Moved the entry to the Upgrading from Spark SQL 4.2 to 4.3 section and corrected the version to Since Spark 4.3. @uros-b

@jiangxt2
jiangxt2 force-pushed the fix/double-to-timestamp-overflow branch from ce0ebe2 to 2e93c6a Compare July 22, 2026 08:55
@jiangxt2
jiangxt2 requested a review from uros-b July 22, 2026 13:03
jiangxt2 added 2 commits July 23, 2026 09:31
…ping in non-ANSI mode

In non-ANSI mode, a Double or Float cast to Timestamp that overflows
the Long range is silently clamped to Long.MAX_VALUE or Long.MIN_VALUE
instead of returning NULL. This is inconsistent with other overflow
casts (e.g., Timestamp to integral types) which return NULL in non-ANSI
mode.

Fix: add overflow detection in doubleToTimestamp (interpreted path)
and castToTimestampCode Double/Float branches (codegen path). In
non-ANSI mode, overflow returns NULL; in ANSI mode, it throws
CAST_OVERFLOW via doubleToTimestampAnsi.

Signed-off-by: jiangxt2 <jiangxt2@vip.qq.com>
@jiangxt2
jiangxt2 force-pushed the fix/double-to-timestamp-overflow branch from 2e93c6a to 2955922 Compare July 23, 2026 01:39
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants