Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[SPARK-9043] Serialize key, value and combiner classes in ShuffleDependency #7403

Closed
wants to merge 4 commits into from

Commits on Sep 9, 2015

  1. [SPARK-9043] Serialize key, value and combiner classes in ShuffleDepe…

    …ndency
    
    ShuffleManager implementations are currently not given type information for
    the key, value and combiner classes. Serialization of shuffle objects relies
    on objects being JavaSerializable, with methods defined for reading/writing
    the object or, alternatively, serialization via Kryo which uses reflection.
    
    Serialization systems like Avro, Thrift and Protobuf generate classes with
    zero argument constructors and explicit schema information
    (e.g. IndexedRecords in Avro have get, put and getSchema methods).
    
    By serializing the key, value and combiner class names in ShuffleDependency,
    shuffle implementations will have access to schema information when
    registerShuffle() is called.
    massie committed Sep 9, 2015
    Configuration menu
    Copy the full SHA
    ed1afac View commit details
    Browse the repository at this point in the history
  2. Configuration menu
    Copy the full SHA
    eccb0ed View commit details
    Browse the repository at this point in the history
  3. Scaladoc cleanup

    massie committed Sep 9, 2015
    Configuration menu
    Copy the full SHA
    2906e74 View commit details
    Browse the repository at this point in the history

Commits on Sep 10, 2015

  1. Configuration menu
    Copy the full SHA
    adcdfaf View commit details
    Browse the repository at this point in the history