[SPARK-22369][PYTHON][DOCS] Exposes catalog API documentation in PySpark

[HyukjinKwon]

What changes were proposed in this pull request?

This PR proposes to add a link from spark.catalog(..) to Catalog and expose Catalog APIs in PySpark as below:

Note that this is not shown in the list on the top - https://spark.apache.org/docs/latest/api/python/pyspark.sql.html#module-pyspark.sql

This is basically similar with DataFrameReader and DataFrameWriter.

How was this patch tested?

Manually built the doc.

[SparkQA]

Test build #83170 has finished for PR 19596 at commit 82fe78b.

• This patch passes all tests.
• This patch merges cleanly.
• This patch adds no public classes.

[HyukjinKwon]

cc @rxin, do you think it's okay to expose PySpark's Catalog class API doc?

[rxin]

Yea definitely.

[HyukjinKwon]

cc @holdenk and @viirya, mind taking a look please? I remember I had few talks about Sphinx and __all__ and I believe you guys are right reviewers.

[viirya]

Don't we like to add it to the list of Important classes of Spark SQL and DataFrames?

[viirya]

I've generated the Python docs. Looks good.

[HyukjinKwon]

Thanks, @viirya. I wasn't sure if I should add it to the list. My intention was .. this one is like DataFrameReader and DataFrameWriter (supposed to be used via spark.read) and I wanted to.. like hide the package path pyspark.sql.Catalog (instead of pyspark.sql.catalog.Catalog) in the doc, and I just decided the smallest change I could think for this issue.

I am fine with adding it too. It's easy to add it if anyone feels strongly about this. I guess it is also true that it might be one of the "Important classes" anyway. Please let me know.

[SparkQA]

Test build #83334 has finished for PR 19596 at commit 82fe78b.

• This patch passes all tests.
• This patch merges cleanly.
• This patch adds no public classes.

[rxin]

Merging in master.

[HyukjinKwon]

Thank you @rxin and @viirya.