Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[SPARK-746][CORE] Added Avro Serialization to Kryo #7004

Closed
wants to merge 13 commits into from
Closed

[SPARK-746][CORE] Added Avro Serialization to Kryo #7004

wants to merge 13 commits into from

Commits on Jun 24, 2015

  1. Added a custom Kryo serializer for generic Avro records to reduce the… 

    … network IO

involved during a shuffle. This compresses the schema and allows for users to
register their schemas ahead of time to further reduce traffic.

Changed how the records were serialized to reduce both the time and memory used

removed unused variable
    Joseph Batchik committed Jun 24, 2015
    Configuration menu
    Copy the full SHA
    97fba62 View commit details
    Browse the repository at this point in the history

Commits on Jun 25, 2015

  1. started working on fixes for pr

    Joseph Batchik committed Jun 25, 2015
    Configuration menu
    Copy the full SHA
    2b545cc View commit details
    Browse the repository at this point in the history

Commits on Jun 26, 2015

  1. fixed serialization error in that SparkConf cannot be serialized

    Joseph Batchik committed Jun 26, 2015
    Configuration menu
    Copy the full SHA
    f4ae251 View commit details
    Browse the repository at this point in the history

Commits on Jun 29, 2015

  1. Changed Avro dependency to be similar to parent

    Joseph Batchik committed Jun 29, 2015
    Configuration menu
    Copy the full SHA
    ab46d10 View commit details
    Browse the repository at this point in the history

Commits on Jul 1, 2015

  1. updated pom to removed versions

    Joseph Batchik committed Jul 1, 2015
    Configuration menu
    Copy the full SHA
    d421bf5 View commit details
    Browse the repository at this point in the history

Commits on Jul 14, 2015

  1. fixed to changes suggested by @squito

    Joseph Batchik committed Jul 14, 2015
    Configuration menu
    Copy the full SHA
    6d1925c View commit details
    Browse the repository at this point in the history

  2. implemented @squito suggestion to use a codec that is already in spark

    Joseph Batchik committed Jul 14, 2015
    Configuration menu
    Copy the full SHA
    0f5471a View commit details
    Browse the repository at this point in the history

Commits on Jul 15, 2015

  1. implemented @squito suggestion

    Joseph Batchik committed Jul 15, 2015
    Configuration menu
    Copy the full SHA
    c5fe794 View commit details
    Browse the repository at this point in the history

  2. forgot a couple of fixes

    Joseph Batchik committed Jul 15, 2015
    Configuration menu
    Copy the full SHA
    fa9298b View commit details
    Browse the repository at this point in the history

  3. updated codec settings

    Joseph Batchik committed Jul 15, 2015
    Configuration menu
    Copy the full SHA
    1183a48 View commit details
    Browse the repository at this point in the history

  4. fixed bug with serializing

    Joseph Batchik committed Jul 15, 2015
    Configuration menu
    Copy the full SHA
    dd71efe View commit details
    Browse the repository at this point in the history

Commits on Jul 16, 2015

  1. implemented @squito suggestion for SparkEnv

    Joseph Batchik committed Jul 16, 2015
    Configuration menu
    Copy the full SHA
    c0cf329 View commit details
    Browse the repository at this point in the history

Commits on Jul 22, 2015

  1. updated per feedback

    @JDrit
    JDrit committed Jul 22, 2015
    Configuration menu
    Copy the full SHA
    8158d51 View commit details
    Browse the repository at this point in the history