Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[SPARK-12404] [SQL] Ensure objects passed to StaticInvoke is Serializable #10357

Closed
wants to merge 4 commits into from

Conversation

sarutak
Copy link
Member

@sarutak sarutak commented Dec 17, 2015

Now StaticInvoke receives Any as a object and StaticInvoke can be serialized but sometimes the object passed is not serializable.

For example, following code raises Exception because RowEncoder#extractorsFor invoked indirectly makes StaticInvoke.

case class TimestampContainer(timestamp: java.sql.Timestamp)
val rdd = sc.parallelize(1 to 2).map(_ => TimestampContainer(System.currentTimeMillis))
val df = rdd.toDF
val ds = df.as[TimestampContainer]
val rdd2 = ds.rdd                                 <----------------- invokes extractorsFor indirectory

I'll add test cases.

@SparkQA
Copy link

SparkQA commented Dec 17, 2015

Test build #47922 has finished for PR 10357 at commit ae7fdc1.

  • This patch fails PySpark unit tests.
  • This patch merges cleanly.
  • This patch adds no public classes.

@sarutak
Copy link
Member Author

sarutak commented Dec 17, 2015

retest this please.

@SparkQA
Copy link

SparkQA commented Dec 17, 2015

Test build #47923 has finished for PR 10357 at commit ae7fdc1.

  • This patch passes all tests.
  • This patch merges cleanly.
  • This patch adds no public classes.

@yu-iskw
Copy link
Contributor

yu-iskw commented Dec 17, 2015

@sarutak thank you for sending this PR.

@marmbrus @rxin could you review it? I think this is a little big issue. We should fix it before releasing Spark 1.6.

@marmbrus
Copy link
Contributor

Regression test please, for any patch that includes a bug fix.

@marmbrus
Copy link
Contributor

So we can avoid delay cutting RC4, I've opened #10360 with tests added.

@SparkQA
Copy link

SparkQA commented Dec 17, 2015

Test build #47947 has finished for PR 10357 at commit d485a95.

  • This patch fails Spark unit tests.
  • This patch merges cleanly.
  • This patch adds no public classes.

@SparkQA
Copy link

SparkQA commented Dec 17, 2015

Test build #47948 has finished for PR 10357 at commit 39a536a.

  • This patch fails Spark unit tests.
  • This patch merges cleanly.
  • This patch adds no public classes.

@SparkQA
Copy link

SparkQA commented Dec 17, 2015

Test build #47954 has finished for PR 10357 at commit f81852d.

  • This patch passes all tests.
  • This patch merges cleanly.
  • This patch adds no public classes.

@marmbrus
Copy link
Contributor

Thanks, merging to master and 1.6

asfgit pushed a commit that referenced this pull request Dec 18, 2015
Now `StaticInvoke` receives `Any` as a object and `StaticInvoke` can be serialized but sometimes the object passed is not serializable.

For example, following code raises Exception because `RowEncoder#extractorsFor` invoked indirectly makes `StaticInvoke`.

```
case class TimestampContainer(timestamp: java.sql.Timestamp)
val rdd = sc.parallelize(1 to 2).map(_ => TimestampContainer(System.currentTimeMillis))
val df = rdd.toDF
val ds = df.as[TimestampContainer]
val rdd2 = ds.rdd                                 <----------------- invokes extractorsFor indirectory
```

I'll add test cases.

Author: Kousuke Saruta <sarutak@oss.nttdata.co.jp>
Author: Michael Armbrust <michael@databricks.com>

Closes #10357 from sarutak/SPARK-12404.

(cherry picked from commit 6eba655)
Signed-off-by: Michael Armbrust <michael@databricks.com>
@asfgit asfgit closed this in 6eba655 Dec 18, 2015
@sarutak sarutak deleted the SPARK-12404 branch June 4, 2021 20:46
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
4 participants