[AURON #2126] Add native support for acosh function#2127
Closed
weimingdiit wants to merge 2 commits intoapache:masterfrom
Closed
[AURON #2126] Add native support for acosh function#2127weimingdiit wants to merge 2 commits intoapache:masterfrom
weimingdiit wants to merge 2 commits intoapache:masterfrom
Conversation
Add Spark ascii expression conversion in NativeConverters so it maps to the existing backend Ascii scalar function, and add a regression test for ascii in AuronFunctionSuite. Signed-off-by: weimingdiit <weimingdiit@gmail.com>
Contributor
There was a problem hiding this comment.
Pull request overview
Adds Spark ascii() expression support to the Spark extension by converting Catalyst Ascii expressions into the already-existing native backend ScalarFunction::Ascii, and introduces a regression test to ensure the plan remains fully native and results match vanilla Spark.
Changes:
- Convert Spark
Asciiexpressions inNativeConverterstopb.ScalarFunction.Ascii. - Add an
asciiregression test toAuronFunctionSuiteusingcheckSparkAnswerAndOperator.
Reviewed changes
Copilot reviewed 2 out of 2 changed files in this pull request and generated 1 comment.
| File | Description |
|---|---|
| spark-extension/src/main/scala/org/apache/spark/sql/auron/NativeConverters.scala | Maps Catalyst Ascii to the native scalar function enum so it can be planned/executed natively. |
| spark-extension-shims-spark/src/test/scala/org/apache/auron/AuronFunctionSuite.scala | Adds a regression test validating ascii() correctness vs Spark and enforcing native operator usage. |
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
spark-extension-shims-spark/src/test/scala/org/apache/auron/AuronFunctionSuite.scala
Outdated
Show resolved
Hide resolved
…ronFunctionSuite.scala Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>
weimingdiit
changed the title
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Which issue does this PR close?
Closes # #2126
Rationale for this change
Spark
Acoshexpressions were not wired into Auron’s standard builtin scalar function conversion path, soacosh(expr)could not be planned through the native backend.This change follows the existing
ScalarFunctionflow used by other builtin math functions such asacos,asin, andatan: Spark expression conversion inNativeConverters, protobuf enum registration inauron.proto, and planner mapping inplanner.rs. This keepsacoshaligned with the current architecture instead of introducing a custom extension function path.What changes are included in this PR?
This PR:
Acoshexpression conversion inNativeConvertersScalarFunction::Acoshinauron.protoScalarFunction::Acoshinplanner.rsacosh(expr)through the standard builtinScalarFunctionchainAre there any user-facing changes?
NO.
How was this patch tested?
CI.