[SPARK-5470][Core]use defaultClassLoader to load classes in KryoSerializer

[lianhuiwang]

Now KryoSerializer load classes of classesToRegister at the time of its initialization. when we set spark.kryo.classesToRegister=class1, it will throw SparkException("Failed to load class to register with Kryo".
because in KryoSerializer's initialization, classLoader cannot include class of user's jars.
we need to use defaultClassLoader of Serializer in newKryo(), because executor will reset defaultClassLoader of Serializer after Serializer's initialization.
thank @zzcclp for reporting it to me.

[SparkQA]

Test build #26281 has started for PR 4258 at commit 64cf306.

• This patch merges cleanly.

[zzcclp]

LGTM, I test this with spark-branch 1.2, it works correctly. :)

[SparkQA]

Test build #26281 has finished for PR 4258 at commit 64cf306.

• This patch passes all tests.
• This patch merges cleanly.
• This patch adds the following public classes (experimental):
  • class LogisticRegression(JavaEstimator, HasFeaturesCol, HasLabelCol, HasPredictionCol, HasMaxIter,
  • class LogisticRegressionModel(JavaModel):
  • class Tokenizer(JavaTransformer, HasInputCol, HasOutputCol):
  • class HashingTF(JavaTransformer, HasInputCol, HasOutputCol, HasNumFeatures):
  • class Param(object):
  • class Params(Identifiable):
  • template = '''class Has$Name(Params):
  • class HasMaxIter(Params):
  • class HasRegParam(Params):
  • class HasFeaturesCol(Params):
  • class HasLabelCol(Params):
  • class HasPredictionCol(Params):
  • class HasInputCol(Params):
  • class HasOutputCol(Params):
  • class HasNumFeatures(Params):
  • class Estimator(Params):
  • class Transformer(Params):
  • class Pipeline(Estimator):
  • class PipelineModel(Transformer):
  • class Identifiable(object):
  • class JavaWrapper(Params):
  • class JavaEstimator(Estimator, JavaWrapper):
  • class JavaTransformer(Transformer, JavaWrapper):
  • class JavaModel(JavaTransformer):

[AmplabJenkins]

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/26281/
Test PASSed.

[sryza]

Can we do the splitting and filtering during initialization so that we don't incur them for each task?

[SparkQA]

Test build #26447 has started for PR 4258 at commit 73b719f.

• This patch merges cleanly.

[lianhuiwang]

@sryza thank you. I have updated for your comment. Can you review again?

[SparkQA]

Test build #26447 has finished for PR 4258 at commit 73b719f.

• This patch passes all tests.
• This patch merges cleanly.
• This patch adds no public classes.

[AmplabJenkins]

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/26447/
Test PASSed.

[sryza]

LGTM

[SparkQA]

Test build #26528 has started for PR 4258 at commit 73b719f.

• This patch merges cleanly.

[SparkQA]

Test build #26528 has finished for PR 4258 at commit 73b719f.

• This patch fails Spark unit tests.
• This patch merges cleanly.
• This patch adds no public classes.

[AmplabJenkins]

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/26528/
Test FAILed.

[lianhuiwang]

Jenkins, retest this please.

[SparkQA]

Test build #26707 has started for PR 4258 at commit 73b719f.

• This patch merges cleanly.

[SparkQA]

Test build #26707 has finished for PR 4258 at commit 73b719f.

• This patch passes all tests.
• This patch merges cleanly.
• This patch adds no public classes.

[AmplabJenkins]

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/26707/
Test PASSed.

[srowen]

This looks like a logical change, since it makes sense to use the provided default classloader to register user classes, just as it's already used to create the serializer. Given reviews from 3 other people and confirmation this does what's intended, it seems like a good fix to merge.

[JoshRosen]

I've cherry-picked this into branch-1.3 (1.3.0) as well.