[SPARK-41721][CONNECT][TESTS] Enable doctests in pyspark.sql.connect.catalog #39224
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
HyukjinKwon
changed the title
HyukjinKwon
changed the title
HyukjinKwon
force-pushed
the
SPARK-41721
branch
from
97f478e
to
39a96bb
Compare
HyukjinKwon
force-pushed
the
SPARK-41721
branch
from
39a96bb
to
b1119f5
Compare
HyukjinKwon
commented
Dec 27, 2022
| @@ -273,15 +273,15 @@ def databaseExists(self, dbName: str) -> bool: | |||
|
|
|||
| >>> spark.catalog.databaseExists("test_new_database") | |||
| False | |||
| >>> _ = spark.sql("CREATE DATABASE test_new_database") | |||
| >>> _ = spark.sql("CREATE DATABASE test_new_database").collect() | |||
There was a problem hiding this comment.
Hmmm ... this is orthogonal issue but just to make sure we don't forget. I think we should make sql method in Spark Connect to be analyzed per every call when they execute such commands. Otherwise, we can't make it compatible with the existing PySpark usage.
I change the existing doctest here for now to make the tests pass but should change this back ideally.
There was a problem hiding this comment.
cc @hvanhovell @grundprinzip FYI
There was a problem hiding this comment.
what about create a jira ticket for this (should change this back ideally)?
HyukjinKwon
force-pushed
the
SPARK-41721
branch
from
b1119f5
to
ea6fad0
Compare
zhengruifeng
approved these changes
Dec 27, 2022
|
Merged to master. |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
What changes were proposed in this pull request?
This PR proposes to enable doctests in
pyspark.sql.connect.catalogthat is virtually the same aspyspark.sql.catalog.Why are the changes needed?
To make sure on the PySpark compatibility and test coverage.
Does this PR introduce any user-facing change?
No, test-only.
How was this patch tested?
CI in this PR should test this out.