Feat kryo max buffersize

[koertkuipers]

No description provided.

[AmplabJenkins]

Can one of the admins verify this patch?

[mateiz]

Hey, so is this a new feature that was recently added to Kryo? Seems super useful, but in this case, I'd actually make the max buffer size higher by default. Or we can use the old setting as a max, and create a new setting for the initial buffer size.

[mateiz]

Jenkins, this is ok to test

[AmplabJenkins]

Merged build triggered.

[AmplabJenkins]

Merged build started.

[AmplabJenkins]

Merged build finished. All automated tests passed.

[AmplabJenkins]

All automated tests passed.
Refer to this link for build results: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/14895/

[koertkuipers]

hey matei,
i think they always had this feature in kryo, at least in 2.x.

created jira here:
https://issues.apache.org/jira/browse/SPARK-1811

only reason to be careful with this feature is that there is some overhead on buffer resizing (each time a new array is created with twice the size and the data is copied), which is why i was reluctant to make the old setting as a max.

how about defaults 2MB for initial and 16MB for max?

[AmplabJenkins]

Merged build triggered.

[AmplabJenkins]

Merged build started.

[AmplabJenkins]

Merged build finished. All automated tests passed.

[AmplabJenkins]

All automated tests passed.
Refer to this link for build results: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/14960/

[pwendell]

This looks good to me! @koertkuipers mind updating the docs to include this (maybe wait until after #880 is merged tomorrow)? Also, mind prefixing the title here with SPARK-1811, this allows our merge script to detect the JIRA and link it.

[mateiz]

Hey one other thought, is there a reason to have the max this low? It might be good to make it even higher to deal with the odd large object (e.g. people working with large text documents). I'd go for 64 MB.

[pwendell]

@koertkuipers any interest in updating the docs and bumping the default? This would be a good change to have.

[koertkuipers]

hey sorry somehow misses this conversation thread. sure will update
defaults and docs

On Wed, Jun 4, 2014 at 1:48 AM, Patrick Wendell notifications@github.com
wrote:

@koertkuipers https://github.com/koertkuipers any interest in updating
the docs and bumping the default? This would be a good change to have.

—
Reply to this email directly or view it on GitHub
#735 (comment).

[AmplabJenkins]

Merged build triggered.

[AmplabJenkins]

Merged build started.

[AmplabJenkins]

Merged build finished.

[AmplabJenkins]

Refer to this link for build results: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/15984/

[koertkuipers]

not sure if i am supposed to deal with these failures?

On Sat, Jun 21, 2014 at 1:52 PM, UCB AMPLab notifications@github.com
wrote:

Refer to this link for build results:
https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/15984/

—
Reply to this email directly or view it on GitHub
#735 (comment).

[pwendell]

Jenkins, retest this please.

[AmplabJenkins]

Merged build triggered.

[AmplabJenkins]

Merged build started.

[AmplabJenkins]

Merged build finished. All automated tests passed.

[AmplabJenkins]

All automated tests passed.
Refer to this link for build results: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/16372/

[mingyukim]

Hi all, is there any update on this PR?

[koertkuipers]

i updated docs and defaults as requested. currently waiting for feedback or
a merge

On Wed, Jul 9, 2014 at 6:46 PM, mingyukim notifications@github.com wrote:

Hi all, is there any update on this PR?

—
Reply to this email directly or view it on GitHub
#735 (comment).

[pwendell]

Jenkins, retest this please. @koertkuipers mind creating a JIRA for this (we're trying to track all changes now). I can merge this in once we have an issue for it.

[SparkQA]

QA tests have started for PR 735. This patch merges cleanly.
View progress: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/16760/consoleFull

[koertkuipers]

https://issues.apache.org/jira/browse/SPARK-2543

On Wed, Jul 16, 2014 at 9:53 PM, Apache Spark QA notifications@github.com
wrote:

QA tests have started for PR 735. This patch merges cleanly.
View progress:
https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/16760/consoleFull

—
Reply to this email directly or view it on GitHub
#735 (comment).

[SparkQA]

QA results for PR 735:
- This patch PASSES unit tests.
- This patch merges cleanly
- This patch adds no public classes

For more information see test ouptut:
https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/16760/consoleFull

[mateiz]

Thanks for the update, Koert!

@pwendell @rxin after this, what do you guys think of lowering the initial buffer size to 1 MB or 512K? We've had some efforts to remove and reuse buffers, but not everything is doing it yet.