Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[SPARK-25972][PYTHON] Missed JSON options in streaming.py #22973

Closed
wants to merge 3 commits into from

Conversation

MaxGekk
Copy link
Member

@MaxGekk MaxGekk commented Nov 8, 2018

What changes were proposed in this pull request?

Added JSON options for json() in streaming.py that are presented in the similar method in readwriter.py. In particular, missed options are dropFieldIfAllNull and encoding.

@MaxGekk
Copy link
Member Author

MaxGekk commented Nov 8, 2018

@HyukjinKwon Thank you for pointing out the missed options. Please, have a look at the PR.

@HyukjinKwon
Copy link
Member

Oh, let's name it [PYTHON] BTW.

@SparkQA
Copy link

SparkQA commented Nov 8, 2018

Test build #98589 has finished for PR 22973 at commit da697ac.

  • This patch passes all tests.
  • This patch merges cleanly.
  • This patch adds no public classes.

@MaxGekk MaxGekk changed the title [SPARK-25972][SQL] Missed JSON options in streaming.py [SPARK-25972][PYTHON] Missed JSON options in streaming.py Nov 8, 2018
@MaxGekk
Copy link
Member Author

MaxGekk commented Nov 8, 2018

Oh, let's name it [PYTHON] BTW.

I see some of on-going PR have tag [PYSPARK]. Is this somewhere documented?

@HyukjinKwon
Copy link
Member

Yea .. actually that's documented in https://spark.apache.org/contributing.html . Strictly it should be PYTHON

The PR title should be of the form [SPARK-xxxx][COMPONENT] Title, where SPARK-xxxx is the relevant JIRA number, COMPONENT is one of the PR categories shown at spark-prs.appspot.com and Title may be the JIRA’s title or a more specific title describing the PR itself.

screen shot 2018-11-08 at 7 19 07 pm

I tried to be clear for myself so I took a look before but .. not a big deal at all :D.

@SparkQA
Copy link

SparkQA commented Nov 9, 2018

Test build #98639 has finished for PR 22973 at commit 4ca71fc.

  • This patch passes all tests.
  • This patch merges cleanly.
  • This patch adds no public classes.

@SparkQA
Copy link

SparkQA commented Nov 11, 2018

Test build #98695 has finished for PR 22973 at commit 5b132ce.

  • This patch passes all tests.
  • This patch merges cleanly.
  • This patch adds no public classes.

@HyukjinKwon
Copy link
Member

Merged to master.

@asfgit asfgit closed this in aec0af4 Nov 11, 2018
jackylee-ch pushed a commit to jackylee-ch/spark that referenced this pull request Feb 18, 2019
## What changes were proposed in this pull request?

Added JSON options for `json()` in streaming.py that are presented in the similar method in readwriter.py. In particular, missed options are `dropFieldIfAllNull` and `encoding`.

Closes apache#22973 from MaxGekk/streaming-missed-options.

Authored-by: Maxim Gekk <max.gekk@gmail.com>
Signed-off-by: hyukjinkwon <gurwls223@apache.org>
@MaxGekk MaxGekk deleted the streaming-missed-options branch August 17, 2019 13:32
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
3 participants