New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
[SPARK-40027][PYTHON][SS][DOCS] Add self-contained examples for pyspark.sql.streaming.readwriter #37461
Conversation
cc @viirya and @HeartSaVioR mind taking a look please when you find some time? |
c916bd4
to
e4daee1
Compare
>>> df = spark.readStream.format("rate").load() | ||
>>> df.writeStream.format("text") | ||
<pyspark.sql.streaming.readwriter.DataStreamWriter object ...> |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Is this redundant? Looks not related to the example below.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
the purpose is to show the type of DataStreamWriter
?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Just to make sure I understand correctly, either 1) you've tested these examples manually or 2) these examples will be automatically tested via CI?
Both, yes :-). |
Merged to master. |
I am touching these examples a lot. So all posthoc reviews are very appreciated! |
What changes were proposed in this pull request?
This PR proposes to improve the examples in
pyspark.sql.streaming.readwriter
by making each example self-contained with a brief explanation and a bit more realistic example.Why are the changes needed?
To make the documentation more readable and able to copy and paste directly in PySpark shell.
Does this PR introduce any user-facing change?
Yes, it changes the documentation
How was this patch tested?
Manually ran each doctest.