Skip to content

feat: full native make_interval with 2.6x faster than spark and 1.1x faster than codegen dispatch - #5292

Open
peterxcli wants to merge 4 commits into
apache:mainfrom
peterxcli:feat/full-native-make-interval
Open

feat: full native make_interval with 2.6x faster than spark and 1.1x faster than codegen dispatch#5292
peterxcli wants to merge 4 commits into
apache:mainfrom
peterxcli:feat/full-native-make-interval

Conversation

@peterxcli

@peterxcli peterxcli commented Aug 7, 2026

Copy link
Copy Markdown
Member

Which issue does this PR close?

Closes #5279.
Closes #5131.

Rationale for this change

Comet represented Spark CalendarIntervalType as Arrow IntervalMonthDayNano. Converting Spark's microseconds to nanoseconds reduced the valid elapsed-time range by 1,000x, while the native datafusion-spark kernel also coerced Decimal(18,6) seconds to Float64, losing microsecond precision.

Spark represents calendar intervals losslessly as separate months, days, and microseconds. Comet needs the same representation across JVM/native boundaries and exact microsecond arithmetic in the native kernel.

What changes are included in this PR?

  • Represent CalendarIntervalType as a Spark-tagged Arrow struct containing months: Int32, days: Int32, and microseconds: Int64.
  • Preserve that logical type through Arrow conversion, protobuf serde, FFI, JVM readers/writers, codegen input/output, Scala UDF codegen, and native execution.
  • Replace the datafusion-spark make_interval wrapper with an exact Decimal(18,6) microsecond kernel with Spark-compatible NULL and ANSI/TRY overflow behavior.
  • Remove the obsolete incompatibility gate, ignored regressions, and unused datafusion-spark dependency.
  • Port Spark 4.2.0 make_interval boundary and arity cases with source permalinks.
  • Remove the obsolete duplicate codegen-dispatch benchmark case now that make_interval is fully native.

How are these changes tested?

  • cargo test --manifest-path native/Cargo.toml -p datafusion-comet-spark-expr preserves_spark_microsecond_range_and_overflow
  • cargo check --manifest-path native/Cargo.toml -p datafusion-comet-spark-expr
  • cargo check --manifest-path native/Cargo.toml -p datafusion-comet
  • cargo fmt --manifest-path native/Cargo.toml --all -- --check
  • make core
  • Focused CometArrowStreamSuite CalendarInterval round-trip test
  • Focused CometCodegenSuite CalendarInterval codegen test
  • CometSqlFileTestSuite make_interval: 6/6 passed, 0 ignored
  • CometSqlFileTestSuite calendar_interval: 1/1 passed
  • Spotless, Scalastyle, and git diff --check

Benchmark

CometDatetimeExpressionBenchmark, 1,048,576 rows, Apple M4, JDK 17. The codegen-dispatch result is from parent commit 268849c0c; the full-native and Spark results are from this PR at 72997e9e8. Both Comet revisions used optimized native builds and the same query and input.

Execution path Best time Average time Throughput Per row Relative to dispatch
Comet codegen dispatch 31 ms 33 ms 33.8 M rows/s 29.6 ns 1.00x
Comet full native 28 ms 30 ms 37.3 M rows/s 26.8 ns 1.11x
Spark 74 ms 81 ms 14.1 M rows/s 70.7 ns 0.42x

Full native is about 10% faster than codegen dispatch and 2.6x faster than Spark by best time.

@peterxcli peterxcli changed the title fix: preserve CalendarInterval microseconds across Comet boundaries and native kernels feat: full native make_interval with 2.x faster than spark Aug 7, 2026
@peterxcli
peterxcli marked this pull request as ready for review August 7, 2026 15:15
@peterxcli peterxcli changed the title feat: full native make_interval with 2.x faster than spark feat: full native make_interval with 2.6x faster than spark and 1.1x faster than codegen dispatch Aug 7, 2026
@peterxcli
peterxcli marked this pull request as draft August 7, 2026 17:13
@peterxcli
peterxcli marked this pull request as ready for review August 7, 2026 20:18
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

1 participant