-
Notifications
You must be signed in to change notification settings - Fork 28.2k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
[SPARK-41721][CONNECT][TESTS] Enable doctests in pyspark.sql.connect.catalog #39224
Conversation
97f478e
to
39a96bb
Compare
39a96bb
to
b1119f5
Compare
@@ -273,15 +273,15 @@ def databaseExists(self, dbName: str) -> bool: | |||
|
|||
>>> spark.catalog.databaseExists("test_new_database") | |||
False | |||
>>> _ = spark.sql("CREATE DATABASE test_new_database") | |||
>>> _ = spark.sql("CREATE DATABASE test_new_database").collect() |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Hmmm ... this is orthogonal issue but just to make sure we don't forget. I think we should make sql
method in Spark Connect to be analyzed per every call when they execute such commands. Otherwise, we can't make it compatible with the existing PySpark usage.
I change the existing doctest here for now to make the tests pass but should change this back ideally.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
cc @hvanhovell @grundprinzip FYI
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
what about create a jira ticket for this (should change this back ideally)?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
👌
b1119f5
to
ea6fad0
Compare
Merged to master. |
What changes were proposed in this pull request?
This PR proposes to enable doctests in
pyspark.sql.connect.catalog
that is virtually the same aspyspark.sql.catalog
.Why are the changes needed?
To make sure on the PySpark compatibility and test coverage.
Does this PR introduce any user-facing change?
No, test-only.
How was this patch tested?
CI in this PR should test this out.