Skip to content

Conversation

@andygrove
Copy link
Member

@andygrove andygrove commented Feb 10, 2026

Which issue does this PR close?

Closes #.

Rationale for this change

DataFusion 52's arrow-arith kernels only support Date32 +/- Interval types, not raw integers. When Spark sends Date32 + Int8/Int16/Int32 arithmetic, the planner now routes these operations to the Spark date_add/date_sub UDFs which handle integer types directly.

What changes are included in this PR?

How are these changes tested?

DataFusion 52's arrow-arith kernels only support Date32 +/- Interval
types, not raw integers. When Spark sends Date32 + Int8/Int16/Int32
arithmetic, the planner now routes these operations to the Spark
date_add/date_sub UDFs which handle integer types directly.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
@andygrove andygrove requested a review from comphead February 10, 2026 17:10
Copy link
Contributor

@comphead comphead left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks @andygrove LGTM would you care to describe what tests this PR addresses?

@andygrove
Copy link
Member Author

Thanks @andygrove LGTM would you care to describe what tests this PR addresses?

Generated by AI:

The PR fixes Date32 +/- Int8/Int16/Int32 arithmetic which broke when upgrading to DataFusion 52, whose arrow-arith kernels only support Date32 +/- Interval types, not raw integers.

The following tests would fail without this fix:

CometExpressionSuite.scala

  • "date_add with int scalars" (line ~301) — SELECT _20 + CAST(2 as TINYINT/SHORT/INT)
  • "date_add with int arrays" (line ~335) — SELECT _20 + _2/_3/_4
  • "date_sub with int scalars" (line ~349) — SELECT _20 - CAST(2 as TINYINT/SHORT/INT)
  • "date_sub with int arrays" (line ~383) — SELECT _20 - _2/_3/_4

SQL file-driven tests (via CometSqlFileTestSuite)

  • spark/src/test/resources/sql-tests/expressions/datetime/date_add.sql
  • spark/src/test/resources/sql-tests/expressions/datetime/date_sub.sql

Fuzz tests (CometFuzzTemporalExpressions.scala)

  • "date arithmetic - date_add"
  • "date arithmetic - date_sub"

All of these tests perform Date32 + Int or Date32 - Int operations, which DataFusion 52's arrow-arith kernels no longer support. The fix routes those operations to the Spark date_add/date_sub UDFs
that handle integer types directly.

@comphead comphead merged commit 451ae69 into apache:df52 Feb 10, 2026
59 of 113 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants