Skip to content

[SPARK-42672][PYTHON][DOCS] Document error class list #40282

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Closed
wants to merge 3 commits into from

Conversation

itholic
Copy link
Contributor

@itholic itholic commented Mar 5, 2023

What changes were proposed in this pull request?

This PR proposes to document the PySpark error class list into a part of Development page.

Screen Shot 2023-03-05 at 5 49 28 PM

Why are the changes needed?

To make it easier for developers to find error classes.

Does this PR introduce any user-facing change?

No, it's documentation,

How was this patch tested?

Manually build docs by running make clean html

@github-actions github-actions bot added the PYTHON label Mar 5, 2023
@HyukjinKwon
Copy link
Member

cc @MaxGekk and @srielau

@itholic
Copy link
Contributor Author

itholic commented Mar 8, 2023

Just created ticket for SQL side: SPARK-42706 FYI.

@itholic
Copy link
Contributor Author

itholic commented Mar 10, 2023

Documentation for SQL side is get merged from #40336.

Note that Python side are simpler compared to SQL side because we do not have SQLSTATE, and there is currently no main error class with sub-error classes. Also, the overall volume of errors is not as high as in SQL documents.

@itholic
Copy link
Contributor Author

itholic commented Mar 14, 2023

Reminder for @HyukjinKwon @srielau @MaxGekk for error class document for PySpark.

Error conditions in PySpark
===========================

This is a list of common, named error conditions returned by PySpark which are defined at `error_classes.py <https://github.com/apache/spark/blob/master/python/pyspark/errors/error_classes.py>`_.
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I just wonder how error classes become error conditions?

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Actually yeah. Is this the proper term we use for error framework?

Copy link
Contributor Author

@itholic itholic Mar 15, 2023

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I referred this term from Error conditions in Databricks because they also support error classes in a similar form.

From my guess, it seems that "error conditions" are used as a concept that also includes error classes and their corresponding parameterized messages.

@srielau do you happen to have any context for this naming?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Maybe do we just update to use term "error classes" instead of "error conditions" ?

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yep, let's use the term "error classes", at least for consistency.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Sounds good. Just applied changes.


When writing PySpark errors, developers must use an error class from the list. If an appropriate error class is not available, add a new one into the list. For more information, please refer to `Contributing Error and Exception <https://spark.apache.org/docs/latest/api/python/development/contributing.html#contributing-error-and-exception>`_.

+------------------------------------------------------------+--------------------------------------------------------------------------------------------------------------+
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Would be great if you add a comment here that how you generated this table.

I am merging this to proceed but would be great if you can make a followup.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Sure. Will automate the doc generation as follow-up in SPARK-43047.

@HyukjinKwon
Copy link
Member

Merged to master.

@itholic itholic deleted the document_error branch April 22, 2023 05:47
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Projects
None yet
Development

Successfully merging this pull request may close these issues.

4 participants