-
Notifications
You must be signed in to change notification settings - Fork 28k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
[SPARK-46036][SQL] Removing error-class from raise_error function #44004
Conversation
|
||
SELECT raise_error(NULL, NULL); | ||
-- TODO: Need feedback on proper behaviour here: |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Looking for feedback here.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
The implicit cast does not support converting complex type to string type. Actually we can skip these 3 tests as implicit cast is orthogonal to the raise_error function
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
done
* @group misc_funcs | ||
* @since 4.0.0 | ||
*/ | ||
def raise_error(c: Column, e: Column): Column = Column.fn("raise_error", c, e) |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
It seems this API added no long.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Not sure that I understand the comment. Can you please elaborate?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
cc @srielau
since = "3.1.0", | ||
group = "misc_funcs") | ||
// scalastyle:on line.size.limit | ||
object RaiseErrorExpressionBuilder extends ExpressionBuilder { |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
RaiseError
works good, why you create the builder?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
In my offline discussion with @cloud-fan we tried to come with a way to achieve following:
- Leave RaiseError class intact so internally we can throw errors of custom class.
- Externally allow only raise_error('message') API.
My understanding is that expression[RaiseError]("raise_error")
ends up doing reflection that would expose a function for every constructor signature in RaiseError, meaning that we will expose both def this(str: Expression)
and def this(errorClass: Expression, errorParms: Expression)
from RaiseError
.
Instead of doing this, we take ExpressionBuilder path that will expose only single argument version.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Got it. It hears good to me. We can use this builder to wrap the visibility.
// scalastyle:on line.size.limit | ||
object RaiseErrorExpressionBuilder extends ExpressionBuilder { | ||
override def build(funcName: String, expressions: Seq[Expression]): Expression = { | ||
// for some reason pattern matching doesn't work here... |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I think explicit length check is better than pattern match here. We don't need this comment.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
right
cc @srielau |
* @group misc_funcs | ||
* @since 4.0.0 | ||
*/ | ||
def raise_error(c: Column, e: Column): Column = Column.fn("raise_error", c, e) |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
We should remove this API from connector/connect/client/jvm/src/main/scala/org/apache/spark/sql/functions.scala
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
@cloud-fan This PR will forbid to expose the errorParms
?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
done.
@beliefer Yes, idea is to forbid errorParams exposure, given that we will always throw USER_RAISED_EXCEPTION with single arg (string representing the message). @cloud-fan, please confirm if I got things right.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
@dbatomic Thank you for the explanation. Please wait the confirm.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
LGTM except some comments.
since = "3.1.0", | ||
group = "misc_funcs") | ||
// scalastyle:on line.size.limit | ||
object RaiseErrorExpressionBuilder extends ExpressionBuilder { |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Got it. It hears good to me. We can use this builder to wrap the visibility.
* @group misc_funcs | ||
* @since 4.0.0 | ||
*/ | ||
def raise_error(c: Column, e: Column): Column = Column.fn("raise_error", c, e) |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
@cloud-fan This PR will forbid to expose the errorParms
?
…-error update of master
thanks, merging to master! |
What changes were proposed in this pull request?
ExpressionBuilder
interface (vs. reflection based lookup).Why are the changes needed?
This PR is a follow up of raise_error improvement PR. As described in jira ticket, raise_error with custom error-class parameter was deemed to be too powerful with concern that user might throw internal system errors which shouldn't be exposed to external layers.
With this change, from external perspective, we revert to the old behaviour, where raise_error only accepts single string argument which will represent message of
USER_RAISED_EXCEPTION
exception object.Does this PR introduce any user-facing change?
Yes, we introduce change in raise_error signature. With this change it is no longer allowed to call:
Instead, only allowed version will be:
Which will result in
USER_RAISED_EXCEPTION
being thrown with provided message.How was this patch tested?
The change updates existing sql tests under misc-functions.sql and adds some new checks.
Was this patch authored or co-authored using generative AI tooling?
No.