Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[SPARK-46565][PYTHON] Refine error classes and error messages for Python data sources #44560

Conversation

allisonwang-db
Copy link
Contributor

What changes were proposed in this pull request?

This PR improves error classes and messages associated with Python data sources. It removes unnecessary error handling in Python and makes error class names more user-friendly.

Why are the changes needed?

To make the error messages clearer and more user-friendly. For instance, current stack traces has redundant information

AnalysisException: [PYTHON_DATA_SOURCE_FAILED_TO_PLAN_IN_PYTHON] Failed to create Python data source instance in Python: Traceback (most recent call last):
 ...
pyspark.errors.exceptions.base.PySparkNotImplementedError: [NOT_IMPLEMENTED] schema is not implemented.

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
    raise PySparkRuntimeError(
pyspark.errors.exceptions.base.PySparkRuntimeError: [PYTHON_DATA_SOURCE_METHOD_NOT_IMPLEMENTED] Unable to create the Python data source instance because the 'schema' method hasn't been implemented.
 SQLSTATE: 38000

After this PR, this During handling of the above exception, another exception occurred: will not show up.

Does this PR introduce any user-facing change?

No

How was this patch tested?

Existing unit tests

Was this patch authored or co-authored using generative AI tooling?

No

@HyukjinKwon
Copy link
Member

Merged to master.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
2 participants