[SPARK-17029] make toJSON not go through rdd form but operate on dataset always #14615

robert3005 · 2016-08-11T23:44:34Z

What changes were proposed in this pull request?

Don't convert toRdd when doing toJSON

How was this patch tested?

Existing unit tests

rxin · 2016-08-12T06:48:21Z

sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/json/JsonFileFormat.scala

can you add classdoc to explain what boundGen (and other ctor parameters) does.

robert3005 · 2016-08-17T00:39:22Z

@rxin anything else? I added docs to the best of my understanding let me know if you meant something else.

ash211 · 2016-09-20T00:10:52Z

This seems pretty reasonable, assuming test coverage already exists on that toJSON method.

Jenkins, this is ok to test.

robert3005 · 2016-10-20T12:33:55Z

@rxin any chance you or someone else can take a look?

ash211 · 2017-03-17T19:47:49Z

Jenkins, this is ok to test.

SparkQA · 2017-03-17T20:59:28Z

Test build #74755 has finished for PR 14615 at commit 0043190.

This patch fails Spark unit tests.
This patch merges cleanly.
This patch adds no public classes.

ash211 · 2017-03-17T22:04:26Z

@robert3005 looks like this has unit test failures on org.apache.spark.sql.hive.orc.OrcSourceSuite.SPARK-19459/SPARK-18220: read char/varchar column written by Hive -- is that a flake?

ash211 · 2017-03-17T22:04:35Z

Jenkins, this is ok to test.

robert3005 · 2017-03-17T22:06:32Z

It indeed does look like a flake.

SparkQA · 2017-03-18T00:06:27Z

Test build #74759 has finished for PR 14615 at commit 0043190.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

SparkQA · 2017-03-19T18:45:34Z

Test build #74828 has finished for PR 14615 at commit 6250699.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

robert3005 · 2017-05-10T16:41:30Z

ping? @rxin

gatorsmile · 2017-05-10T18:38:19Z

sql/core/src/main/scala/org/apache/spark/sql/Dataset.scala

Revert this change?

gatorsmile · 2017-05-10T18:39:54Z

sql/core/src/main/scala/org/apache/spark/sql/Dataset.scala

Instead of importing it, we can explicitly pass the encoder.

mapPartitions { // func } (sparkSession.implicits.newStringEncoder)

robert3005 · 2017-05-10T20:25:24Z

thanks @gatorsmile, updated

SparkQA · 2017-05-10T22:37:17Z

Test build #76757 has finished for PR 14615 at commit 274be08.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

cloud-fan · 2017-05-11T00:07:02Z

sql/core/src/main/scala/org/apache/spark/sql/Dataset.scala

-    }
-    import sparkSession.implicits.newStringEncoder
-    sparkSession.createDataset(rdd)
+    } (sparkSession.implicits.newStringEncoder)


nit: Encoders.STRING

cloud-fan · 2017-05-11T00:08:22Z

sql/core/src/test/scala/org/apache/spark/sql/execution/datasources/json/JsonSuite.scala

+      case _ => false
+    }
+
+    if (containsRDD.isDefined) {


nit: assert(containsRDD.isEmpty, "xxx")

cloud-fan · 2017-05-11T00:08:33Z

LGTM

SparkQA · 2017-05-11T04:45:53Z

Test build #76766 has finished for PR 14615 at commit 723ef1e.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

cloud-fan · 2017-05-11T07:27:23Z

thanks, merging to master!

…set always ## What changes were proposed in this pull request? Don't convert toRdd when doing toJSON ## How was this patch tested? Existing unit tests Author: Robert Kruszewski <robertk@palantir.com> Closes apache#14615 from robert3005/robertk/correct-tojson.

robert3005 changed the title ~~make toJSON not go through rdd form but operate on dataset always~~ [SPARK-17209] make toJSON not go through rdd form but operate on dataset always Aug 11, 2016

robert3005 changed the title ~~[SPARK-17209] make toJSON not go through rdd form but operate on dataset always~~ [SPARK-17029] make toJSON not go through rdd form but operate on dataset always Aug 11, 2016

rxin reviewed Aug 12, 2016
View reviewed changes

robert3005 force-pushed the robertk/correct-tojson branch 4 times, most recently from def3cf0 to a2e8ebf Compare August 17, 2016 00:32

robert3005 force-pushed the robertk/correct-tojson branch from a2e8ebf to 6f286c1 Compare September 3, 2016 02:43

robert3005 force-pushed the robertk/correct-tojson branch from 6f286c1 to 48342c2 Compare September 13, 2016 23:01

robert3005 mentioned this pull request Oct 31, 2016

tojson fix palantir/spark#48

Merged

robert3005 force-pushed the robertk/correct-tojson branch from 48342c2 to 0043190 Compare March 17, 2017 19:43

gatorsmile reviewed May 10, 2017

View reviewed changes

sql/core/src/main/scala/org/apache/spark/sql/Dataset.scala Outdated

Copy link

Member

gatorsmile May 10, 2017

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Revert this change?

gatorsmile reviewed May 10, 2017

View reviewed changes

Robert Kruszewski added 3 commits May 10, 2017 16:23

make toJSON not go through rdd form but operate on dataset always

e3c9772

unbound encoder is enough since we're serializing

9cf1d55

revert unnecessary doc changes

078732b

Robert Kruszewski added 3 commits May 10, 2017 16:23

and a simple test

e1cc27f

imports

23a785f

address comments

274be08

robert3005 force-pushed the robertk/correct-tojson branch from 6250699 to 274be08 Compare May 10, 2017 20:24

cloud-fan reviewed May 11, 2017

View reviewed changes

style

723ef1e

asfgit closed this in 65accb8 May 11, 2017

robert3005 deleted the robertk/correct-tojson branch December 6, 2019 12:50

[SPARK-17029] make toJSON not go through rdd form but operate on dataset always #14615

[SPARK-17029] make toJSON not go through rdd form but operate on dataset always #14615

Uh oh!

Conversation

robert3005 commented Aug 11, 2016

What changes were proposed in this pull request?

How was this patch tested?

Uh oh!

rxin Aug 12, 2016

Choose a reason for hiding this comment

Uh oh!

robert3005 commented Aug 17, 2016

Uh oh!

ash211 commented Sep 20, 2016

Uh oh!

robert3005 commented Oct 20, 2016

Uh oh!

ash211 commented Mar 17, 2017

Uh oh!

SparkQA commented Mar 17, 2017

Uh oh!

ash211 commented Mar 17, 2017

Uh oh!

ash211 commented Mar 17, 2017

Uh oh!

robert3005 commented Mar 17, 2017

Uh oh!

SparkQA commented Mar 18, 2017

Uh oh!

SparkQA commented Mar 19, 2017

Uh oh!

robert3005 commented May 10, 2017

Uh oh!

gatorsmile May 10, 2017

Choose a reason for hiding this comment

Uh oh!

gatorsmile May 10, 2017

Choose a reason for hiding this comment

Uh oh!

robert3005 commented May 10, 2017

Uh oh!

SparkQA commented May 10, 2017

Uh oh!

cloud-fan May 11, 2017

Choose a reason for hiding this comment

Uh oh!

cloud-fan May 11, 2017

Choose a reason for hiding this comment

Uh oh!

cloud-fan commented May 11, 2017

Uh oh!

SparkQA commented May 11, 2017

Uh oh!

cloud-fan commented May 11, 2017

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

6 participants