Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[SPARK-24307][CORE] Support reading remote cached partitions > 2gb #21440

Closed
wants to merge 6 commits into from

Commits on May 25, 2018

  1. [SPARK-24307][CORE] Support reading remote cached partitions > 2gb

    (1) Netty's ByteBuf cannot support data > 2gb.  So to transfer data from a
    ChunkedByteBuffer over the network, we use a custom version of
    FileRegion which is backed by the ChunkedByteBuffer.
    
    (2) On the receiving end, we need to expose all the data in a
    FileSegmentManagedBuffer as a ChunkedByteBuffer.  We do that by memory
    mapping the entire file in chunks.
    
    Added unit tests.  Also tested on a cluster with remote cache reads >
    2gb (in memory and on disk).
    squito committed May 25, 2018
    Configuration menu
    Copy the full SHA
    4373e27 View commit details
    Browse the repository at this point in the history

Commits on May 29, 2018

  1. review feedback

    squito committed May 29, 2018
    Configuration menu
    Copy the full SHA
    a9cfe29 View commit details
    Browse the repository at this point in the history

Commits on Jun 27, 2018

  1. review feedback

    squito committed Jun 27, 2018
    Configuration menu
    Copy the full SHA
    6c57e4d View commit details
    Browse the repository at this point in the history

Commits on Jun 28, 2018

  1. review feedback

    squito committed Jun 28, 2018
    Configuration menu
    Copy the full SHA
    4b53667 View commit details
    Browse the repository at this point in the history

Commits on Jul 2, 2018

  1. fix

    squito committed Jul 2, 2018
    Configuration menu
    Copy the full SHA
    65b7d87 View commit details
    Browse the repository at this point in the history

Commits on Jul 18, 2018

  1. review feedback

    squito committed Jul 18, 2018
    Configuration menu
    Copy the full SHA
    4664942 View commit details
    Browse the repository at this point in the history