Skip to content

Commit

Permalink
[SPARK-43527][PYTHON] Fix catalog.listCatalogs in PySpark
Browse files Browse the repository at this point in the history
### What changes were proposed in this pull request?
Fix `catalog.listCatalogs` in PySpark

### Why are the changes needed?
existing implementation outputs incorrect results

### Does this PR introduce _any_ user-facing change?
yes

before this PR:
```
In [1]: spark.catalog.listCatalogs()
Out[1]: [CatalogMetadata(name=<py4j.java_gateway.JavaMember object at 0x1031f08b0>, description=<py4j.java_gateway.JavaMember object at 0x1049ac2e0>)]
```

after this PR:
```
In [1]: spark.catalog.listCatalogs()
Out[1]: [CatalogMetadata(name='spark_catalog', description=None)]
```

### How was this patch tested?
added doctest

Closes #41186 from zhengruifeng/py_list_catalog.

Authored-by: Ruifeng Zheng <ruifengz@apache.org>
Signed-off-by: Hyukjin Kwon <gurwls223@apache.org>
  • Loading branch information
zhengruifeng authored and HyukjinKwon committed May 16, 2023
1 parent 5573505 commit a232083
Showing 1 changed file with 8 additions and 1 deletion.
9 changes: 8 additions & 1 deletion python/pyspark/sql/catalog.py
Expand Up @@ -128,12 +128,19 @@ def listCatalogs(self) -> List[CatalogMetadata]:
-------
list
A list of :class:`CatalogMetadata`.
Examples
--------
>>> spark.catalog.listCatalogs()
[CatalogMetadata(name='spark_catalog', description=None)]
"""
iter = self._jcatalog.listCatalogs().toLocalIterator()
catalogs = []
while iter.hasNext():
jcatalog = iter.next()
catalogs.append(CatalogMetadata(name=jcatalog.name, description=jcatalog.description))
catalogs.append(
CatalogMetadata(name=jcatalog.name(), description=jcatalog.description())
)
return catalogs

def currentDatabase(self) -> str:
Expand Down

0 comments on commit a232083

Please sign in to comment.