Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[SPARK-5470][Core]use defaultClassLoader to load classes in KryoSerializer #4258

Closed
wants to merge 2 commits into from
Closed

[SPARK-5470][Core]use defaultClassLoader to load classes in KryoSerializer #4258

Changes from 1 commit
Commits
Show all changes
2 commits
Select commit Hold shift + click to select a range
64cf306
use defaultClassLoader to load classes of classesToRegister in KryoSe…
lianhuiwang Jan 29, 2015
73b719f
do the splitting and filtering during initialization
lianhuiwang Jan 31, 2015
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Jump to
Jump to file
Failed to load files.
Diff view
Diff view
14 changes: 3 additions & 11 deletions core/src/main/scala/org/apache/spark/serializer/KryoSerializer.scala
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -56,16 +56,6 @@ class KryoSerializer(conf: SparkConf)
private val registrationRequired = conf.getBoolean("spark.kryo.registrationRequired", false)
private val userRegistrator = conf.getOption("spark.kryo.registrator")
private val classesToRegister = conf.get("spark.kryo.classesToRegister", "")
.split(',')
.filter(!_.isEmpty)
.map { className =>
try {
Class.forName(className)
} catch {
case e: Exception =>
throw new SparkException("Failed to load class to register with Kryo", e)
}
}

def newKryoOutput() = new KryoOutput(bufferSize, math.max(bufferSize, maxBufferSize))

Expand Down Expand Up @@ -97,7 +87,9 @@ class KryoSerializer(conf: SparkConf)
// Use the default classloader when calling the user registrator.
Thread.currentThread.setContextClassLoader(classLoader)
// Register classes given through spark.kryo.classesToRegister.
classesToRegister.foreach { clazz => kryo.register(clazz) }
classesToRegister.split(',')
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Can we do the splitting and filtering during initialization so that we don't incur them for each task?

.filter(!_.isEmpty)
.foreach { className => kryo.register(Class.forName(className, true, classLoader)) }
// Allow the user to register their own classes by setting spark.kryo.registrator.
userRegistrator
.map(Class.forName(_, true, classLoader).newInstance().asInstanceOf[KryoRegistrator])
Expand Down