Skip to content

[MINOR][DOCS][CONNECT] Update notes about supported modules in PySpark API reference#43013

Closed
HyukjinKwon wants to merge 1 commit intoapache:masterfrom
HyukjinKwon:minor-docs-py
Closed

[MINOR][DOCS][CONNECT] Update notes about supported modules in PySpark API reference#43013
HyukjinKwon wants to merge 1 commit intoapache:masterfrom
HyukjinKwon:minor-docs-py

Conversation

@HyukjinKwon
Copy link
Member

What changes were proposed in this pull request?

This PR proposes to add a couple of notes about which modules are supported by Spark Connect.

Why are the changes needed?

In order for users to explicitly know which ones are supported in PySpark with Spark Connect.

Does this PR introduce any user-facing change?

Yes, this exposes some notes for PySpark API with Spark Connect.

How was this patch tested?

Manually built the site, and checked.

Screenshot 2023-09-20 at 6 05 54 PM

Screenshot 2023-09-20 at 6 05 28 PM

Screenshot 2023-09-20 at 6 05 32 PM

Was this patch authored or co-authored using generative AI tooling?

No.

@HyukjinKwon
Copy link
Member Author

@zhengruifeng
Copy link
Contributor

/mllib in scala, pyspark.ml and pyspark.mllib in python, don't work on connect.

only new module pyspark.ml.connect works on connect.

pyspark.ml contains many classification/clustering/etc, they are not supported in pyspark.ml.connect

cc @WeichenXu123

@zhengruifeng
Copy link
Contributor

pyspark.ml.connect only supports a small subset of pyspark.ml

Copy link
Member

@dongjoon-hyun dongjoon-hyun left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

+1, LGTM.

@HyukjinKwon
Copy link
Member Author

Yeah. We should probably document pyspark.ml.connect separately but there haven't been examples out yet. Let's document that separately.

@HyukjinKwon
Copy link
Member Author

Merged to master.

@HyukjinKwon HyukjinKwon deleted the minor-docs-py branch January 15, 2024 00:48
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants