New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
[SPARK-35176][PYTHON] Standardize input validation error type #32368
Conversation
Test build #138000 has finished for PR 32368 at commit
|
Kubernetes integration test starting |
Kubernetes integration test status failure |
cc @HyukjinKwon |
At first glance it looks good (I'll try to do more thorough scan if I have access to larger screen). We might have to document this as a change of behaviour though, as it might break some 3rd party code. |
Oh yeah, that's a very good point. @Yikun feel free to create a new page for Spark 3.1 -> 3.2 at https://github.com/apache/spark/tree/master/python/docs/source/migration_guide and document this one |
@HyukjinKwon @zero323 Thanks for reminder, the migration doc has been added in 02080b0 |
Test build #138060 has finished for PR 32368 at commit
|
Kubernetes integration test unable to build dist. exiting with code: 1 |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Looks okay as TypeError should be more accurate for the kind of exceptions.
Test build #138069 has finished for PR 32368 at commit
|
Kubernetes integration test starting |
Kubernetes integration test status failure |
Merged to master. |
What changes were proposed in this pull request?
This PR corrects some exception type when the function input params are failed to validate due to TypeError.
In order to convenient to review, there are 3 commits in this PR:
Why are the changes needed?
As suggestion from Python exception doc [1]: "Raised when an operation or function is applied to an object of inappropriate type.", but there are many Value error are raised in some pyspark code, this patch fix them.
[1] https://docs.python.org/3/library/exceptions.html#TypeError
Note that: this patch only addresses the exsiting some wrong raise type for input validation, the input validation decorator/framework which mentioned in SPARK-35176, would be submited in a speparated patch.
Does this PR introduce any user-facing change?
Yes, code can raise the right TypeError instead of ValueError.
How was this patch tested?
Existing test case and UT