[SPARK-5494][SQL] SparkSqlSerializer Ignores KryoRegistrators

[hkothari]

SparkSqlSerializer completely ignores custom KryoRegistrators (unlike the regular Spark serializer). I've applied a change to call super.newKryo before registering their custom classes, and removed duplicate registrations/settings.

[ash211]

Jenkins this is ok to test

[SparkQA]

Test build #27723 has finished for PR 4693 at commit 8e756cc.

• This patch fails Spark unit tests.
• This patch merges cleanly.
• This patch adds no public classes.

[marmbrus]

There is no reason to allow custom kryo registration. The SQL serializer is only ever used to serialize SQL types. Have you seen some bug with us missing a type?

[pwoody]

I believe the bug is due to inconsistency between KryoSerializer and SqlSerializer in the order of registration. I don't know if there is a way to manage multiple Serializers, but mixed applications (sql/raw spark) can run into issues.

[marmbrus]

@pwoody I don't think that is possible. As long as the two sides are using copies of kryo with the same order or registration then things should work. As far as I know we never mix them up. Please let me know if you have a test case that shows otherwise.

If there are no examples of failures, then I suggest we close this issue.

[mccheah]

Talked with @pwoody offline and he's working on a test case, but I'll summarize what I think will break. If you use an SQL Context wrapping a Spark Context where you set spark.serializer, then when you do some things in Spark SQL and try to collect the RDD, the operations in computing the SQL will be fine, but when you collect the RDD it uses the SQL serializer to serialize the results but the driver will kryo-deserialize using spark.serializer.

This was the issue that originally prompted this change.

[marmbrus]

@mccheah we set the serializer in SQL on a per-shuffle basis, so that would surprise me. However, if you can show it happening we should certainly fix it.

[yhuai]

@pwoody @mccheah any update on it? If you saw an error, what was the exception and error message?