New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[Test Only][not ready for review][SPARK-6235][CORE]Address various 2G limits #14977

Closed
wants to merge 1 commit into
base: master
from

Conversation

Projects
None yet
2 participants
@witgo
Contributor

witgo commented Sep 6, 2016

What changes were proposed in this pull request?

Design

Setup for eliminating the various 2G limit in Spark.

Replace ByteBuffer with ChunkedByteBuffer.
  1. Move the ChunkedByteBuffer class to common/network-common/src/main/java/org/apache/spark/network/buffer/
  2. Modify ManagedBuffer.nioByteBuffer's return value to ChunkedByteBuffer instance.
  3. Modify the parameter of SerializerInstance.deserialize and the return value of SerializerInstance.serialize ChunkedByteBuffer instance.
  4. Other changes.
Replace ByteBuf with InputStream.
  1. Add InputStreamManagedBuffer class, used to convert InputStream instance to ManagedBuffer instance.
  2. Modify NioManagedBuffer.convertToNetty method returns InputStream instances when data is larger than Integer.MAX_VALUE.
  3. Modify the parameters of the Encodable.encode method to OutputStream instance.
  4. Modify the parameters of the decode method of the classes who implement the Encodable interface to InputStream instance.
  5. Modify TransportFrameDecoder class, use LinkedList<ByteBuf> to represent the Frame, remove the size limit of Frame.
  6. Add ByteBufInputStream class, used to convert LinkedList<ByteBuf> instance to InputStream instance.
  7. Modify the parameters of RpcHandler.receive method to InputStream instance.

How was this patch tested?

TODO ...

@SparkQA

This comment has been minimized.

Show comment
Hide comment
@SparkQA

SparkQA Sep 6, 2016

Test build #64999 has finished for PR 14977 at commit 5259ecf.

  • This patch fails MiMa tests.
  • This patch merges cleanly.
  • This patch adds no public classes.

SparkQA commented Sep 6, 2016

Test build #64999 has finished for PR 14977 at commit 5259ecf.

  • This patch fails MiMa tests.
  • This patch merges cleanly.
  • This patch adds no public classes.

@witgo witgo closed this Sep 7, 2016

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment