Skip to content

fix(spark): match Spark abs overflow errors - #24156

Open
goutamadwant wants to merge 2 commits into
apache:mainfrom
goutamadwant:fix-spark-abs-overflow
Open

fix(spark): match Spark abs overflow errors#24156
goutamadwant wants to merge 2 commits into
apache:mainfrom
goutamadwant:fix-spark-abs-overflow

Conversation

@goutamadwant

Copy link
Copy Markdown
Contributor

Which issue does this PR close?

Rationale for this change

When ANSI mode is enabled, datafusion-spark reports type-specific DataFusion errors for integral abs overflow. The scalar and array paths also produce different messages. Spark 4.2 uses one canonical ARITHMETIC_OVERFLOW message for both paths.

What changes are included in this PR?

  • Add a Spark-local checked abs array kernel so Spark-specific error text does not change core DataFusion behavior.
  • Use the Spark 4.2 ARITHMETIC_OVERFLOW message for scalar and array overflows.
  • Update the existing Spark SQL logic tests for all signed integer widths and both input shapes.

Are these changes tested?

Yes. The following checks pass:

  • cargo fmt --all -- --check
  • cargo clippy -p datafusion-spark --all-targets --all-features -- -D warnings
  • cargo test -p datafusion-spark --all-features
  • cargo test --profile=ci --test sqllogictests -- spark/math/abs.slt
  • cargo test --profile=ci --test sqllogictests -- math.slt
  • RUST_BACKTRACE=1 cargo test --profile ci --exclude datafusion-examples --exclude datafusion-benchmarks --exclude datafusion-cli --workspace --lib --tests --bins --features avro,json,backtrace,extended_tests,recursive_protection,parquet_encryption

Are there any user-facing changes?

Yes. With datafusion.execution.enable_ansi_mode = true, Spark abs overflow errors now use Spark 4.2's canonical ARITHMETIC_OVERFLOW message for scalar and array inputs. There are no public API changes.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

spark sqllogictest SQL Logic Tests (.slt)

Projects

None yet

Development

Successfully merging this pull request may close these issues.

[Bug] abs reports a DataFusion-specific ANSI overflow message instead of Spark's ARITHMETIC_OVERFLOW text

1 participant