[SPARK-47200][SS] Error class for Foreach batch sink user function error#45299
Closed
micheal-o wants to merge 4 commits intoapache:masterfrom
Closed
[SPARK-47200][SS] Error class for Foreach batch sink user function error#45299micheal-o wants to merge 4 commits intoapache:masterfrom
micheal-o wants to merge 4 commits intoapache:masterfrom
Conversation
MaxGekk
reviewed
Feb 28, 2024
Contributor
Author
|
cc @HeartSaVioR |
HeartSaVioR
approved these changes
Feb 29, 2024
Contributor
HeartSaVioR
left a comment
HeartSaVioR
left a comment
There was a problem hiding this comment.
+1
I'll wait for a day (or over the weekend) for @MaxGekk to chime in. I'll merge the PR once he approves or no major comment is made from him.
MaxGekk
approved these changes
Feb 29, 2024
Contributor
|
Thanks! Merging to master. |
TakawaAkirayo
pushed a commit
to TakawaAkirayo/spark
that referenced
this pull request
Mar 4, 2024
### What changes were proposed in this pull request? Any exception can be thrown from the user provided function for ForEachBatchSink. We want to classify this class of errors. Including errors from Python (Py4JException) and Scala functions. ### Why are the changes needed? The user provided function can throw any type of error. Using the new error framework for better error messages and classification. ### Does this PR introduce _any_ user-facing change? Yes, better error message with error class for ForeachBatchSink user function failures. ### How was this patch tested? Updated existing tests and added a new one. Covers python and Scala. ### Was this patch authored or co-authored using generative AI tooling? No Closes apache#45299 from micheal-o/ForeachBatchSinkUserCodeError. Authored-by: micheal-o <micheal.okutubo@gmail.com> Signed-off-by: Jungtaek Lim <kabhwan.opensource@gmail.com>
ericm-db
pushed a commit
to ericm-db/spark
that referenced
this pull request
Mar 5, 2024
### What changes were proposed in this pull request? Any exception can be thrown from the user provided function for ForEachBatchSink. We want to classify this class of errors. Including errors from Python (Py4JException) and Scala functions. ### Why are the changes needed? The user provided function can throw any type of error. Using the new error framework for better error messages and classification. ### Does this PR introduce _any_ user-facing change? Yes, better error message with error class for ForeachBatchSink user function failures. ### How was this patch tested? Updated existing tests and added a new one. Covers python and Scala. ### Was this patch authored or co-authored using generative AI tooling? No Closes apache#45299 from micheal-o/ForeachBatchSinkUserCodeError. Authored-by: micheal-o <micheal.okutubo@gmail.com> Signed-off-by: Jungtaek Lim <kabhwan.opensource@gmail.com>
HeartSaVioR
pushed a commit
that referenced
this pull request
Mar 10, 2024
…g backward compatible ### What changes were proposed in this pull request? I checked in a previous PR (#45299), that handles and classifies exceptions thrown in user provided functions for foreach batch sink. This change is to make it backward compatible in order not to break current users, since users may be depending on getting the user code error from the `StreamingQueryException.cause` instead of `StreamingQueryException.cause.cause` ### Why are the changes needed? To prevent breaking existing usage pattern. ### Does this PR introduce _any_ user-facing change? Yes, better error message with error class for ForeachBatchSink user function failures. ### How was this patch tested? updated existing tests ### Was this patch authored or co-authored using generative AI tooling? No Closes #45449 from micheal-o/ForeachBatchExBackwardCompat. Authored-by: micheal-o <micheal.okutubo@gmail.com> Signed-off-by: Jungtaek Lim <kabhwan.opensource@gmail.com>
HeartSaVioR
pushed a commit
that referenced
this pull request
Sep 5, 2024
… error ### What changes were proposed in this pull request? Similar with classification that micheal-o did for ForeachBatch sink PR: #45299, any exception can be thrown from the user provided function for ForEach Sink. We want to classify this class of errors. Including errors from Python (Py4JException) and Scala functions. ### Why are the changes needed? The user provided function can throw any type of error. Using the new error framework for better error messages and classification. ### Does this PR introduce _any_ user-facing change? Yes, better error message with error class for Foreach sink user function failures. ### How was this patch tested? Updated existing tests. Covers Python and Scala. ### Was this patch authored or co-authored using generative AI tooling? No. Closes #47819 from jingz-db/classify-foreach-error. Authored-by: jingz-db <jing.zhan@databricks.com> Signed-off-by: Jungtaek Lim <kabhwan.opensource@gmail.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
3 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What changes were proposed in this pull request?
Any exception can be thrown from the user provided function for ForEachBatchSink. We want to classify this class of errors. Including errors from Python (Py4JException) and Scala functions.
Why are the changes needed?
The user provided function can throw any type of error. Using the new error framework for better error messages and classification.
Does this PR introduce any user-facing change?
Yes, better error message with error class for ForeachBatchSink user function failures.
How was this patch tested?
Updated existing tests and added a new one. Covers python and Scala.
Was this patch authored or co-authored using generative AI tooling?
No