New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
[SPARK-12404] [SQL] Ensure objects passed to StaticInvoke is Serializable #10357
Conversation
Test build #47922 has finished for PR 10357 at commit
|
retest this please. |
Test build #47923 has finished for PR 10357 at commit
|
Regression test please, for any patch that includes a bug fix. |
So we can avoid delay cutting RC4, I've opened #10360 with tests added. |
Test build #47947 has finished for PR 10357 at commit
|
Test build #47948 has finished for PR 10357 at commit
|
Test build #47954 has finished for PR 10357 at commit
|
Thanks, merging to master and 1.6 |
Now `StaticInvoke` receives `Any` as a object and `StaticInvoke` can be serialized but sometimes the object passed is not serializable. For example, following code raises Exception because `RowEncoder#extractorsFor` invoked indirectly makes `StaticInvoke`. ``` case class TimestampContainer(timestamp: java.sql.Timestamp) val rdd = sc.parallelize(1 to 2).map(_ => TimestampContainer(System.currentTimeMillis)) val df = rdd.toDF val ds = df.as[TimestampContainer] val rdd2 = ds.rdd <----------------- invokes extractorsFor indirectory ``` I'll add test cases. Author: Kousuke Saruta <sarutak@oss.nttdata.co.jp> Author: Michael Armbrust <michael@databricks.com> Closes #10357 from sarutak/SPARK-12404. (cherry picked from commit 6eba655) Signed-off-by: Michael Armbrust <michael@databricks.com>
Now
StaticInvoke
receivesAny
as a object andStaticInvoke
can be serialized but sometimes the object passed is not serializable.For example, following code raises Exception because
RowEncoder#extractorsFor
invoked indirectly makesStaticInvoke
.I'll add test cases.