feat: full native make_interval with 2.6x faster than spark and 1.1x faster than codegen dispatch - #5292
Open
peterxcli wants to merge 4 commits into
Open
feat: full native make_interval with 2.6x faster than spark and 1.1x faster than codegen dispatch#5292peterxcli wants to merge 4 commits into
peterxcli wants to merge 4 commits into
Conversation
peterxcli
marked this pull request as ready for review
August 7, 2026 15:15
peterxcli
marked this pull request as draft
August 7, 2026 17:13
peterxcli
marked this pull request as ready for review
August 7, 2026 20:18
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Which issue does this PR close?
Closes #5279.
Closes #5131.
Rationale for this change
Comet represented Spark
CalendarIntervalTypeas ArrowIntervalMonthDayNano. Converting Spark's microseconds to nanoseconds reduced the valid elapsed-time range by 1,000x, while the nativedatafusion-sparkkernel also coercedDecimal(18,6)seconds toFloat64, losing microsecond precision.Spark represents calendar intervals losslessly as separate months, days, and microseconds. Comet needs the same representation across JVM/native boundaries and exact microsecond arithmetic in the native kernel.
What changes are included in this PR?
CalendarIntervalTypeas a Spark-tagged Arrow struct containingmonths: Int32,days: Int32, andmicroseconds: Int64.datafusion-sparkmake_intervalwrapper with an exactDecimal(18,6)microsecond kernel with Spark-compatible NULL and ANSI/TRY overflow behavior.datafusion-sparkdependency.make_intervalboundary and arity cases with source permalinks.make_intervalis fully native.How are these changes tested?
cargo test --manifest-path native/Cargo.toml -p datafusion-comet-spark-expr preserves_spark_microsecond_range_and_overflowcargo check --manifest-path native/Cargo.toml -p datafusion-comet-spark-exprcargo check --manifest-path native/Cargo.toml -p datafusion-cometcargo fmt --manifest-path native/Cargo.toml --all -- --checkmake coreCometArrowStreamSuiteCalendarInterval round-trip testCometCodegenSuiteCalendarInterval codegen testCometSqlFileTestSuite make_interval: 6/6 passed, 0 ignoredCometSqlFileTestSuite calendar_interval: 1/1 passedgit diff --checkBenchmark
CometDatetimeExpressionBenchmark, 1,048,576 rows, Apple M4, JDK 17. The codegen-dispatch result is from parent commit268849c0c; the full-native and Spark results are from this PR at72997e9e8. Both Comet revisions used optimized native builds and the same query and input.Full native is about 10% faster than codegen dispatch and 2.6x faster than Spark by best time.