Low-level helpers for Spark libraries and tests
Clone or download
Fetching latest commit…
Cannot retrieve the latest commit at this time.
Permalink
Failed to load latest commit information.
project
src
.gitignore
.travis.yml
LICENSE
README.md
build.sbt

README.md

spark-util

Build Status Coverage Status Maven Central

Spark, Hadoop, and Kryo utilities

Kryo registration

Classes that implement the Registrar interface can use various shorthands for registering classes with Kryo.

Adapted from RegistrationTest:

register(
  cls[A],                  // comes with an AlsoRegister that loops in other classes
  arr[Foo],                // register a class and an Array of that class
  cls[B]  BSerializer(),  // use a custom Serializer
  CDRegistrar              // register all of another Registrar's registrations
)
  • custom Serializers and AlsoRegisters are picked up implicitly if not provided explicitly.
  • AlsoRegisters are recursive, allowing for much easier and more robust accountability about what is registered and why, and ensurance that needed registrations aren't overlooked.

Configuration/Context wrappers

  • Configuration: serializable Hadoop-Configuration wrapper
  • Context: SparkContext wrapper that is also a Hadoop Configuration, for unification of "global configuration access" patterns
  • Conf: load a SparkConf with settings from file(s) specified in the SPARK_PROPERTIES_FILES environment variable

Spark Configuration

Misc