PARQUET-321: Default maximum block padding to 8MB.

[rdblue]

No description provided.

[rdblue]

@isnotinvain and @julienledem, I think we should set a default max padding size before we release the row group padding in 1.8.0. There is more discussion about the 8MB limit on the issue, PARQUET-321. Thoughts?

[isnotinvain]

So this would no longer be an opt-in feature? It would be on by default?

[rdblue]

Yeah, it would be on by default. I think we need to do something like this so we don't end up with tiny row groups squeezed into the end of HDFS blocks. That either means starting a full-size block or padding.

The main question is: what should happen if there is 2MB left before the end of the block? Should we start a 2MB row group, pad, or ignore the situation? I think the right thing to do is to pad. Our expectation is for this to be less than ~2% overhead because we're pretty good at targeting a specific row group size, but we need to decide on the minimum size we are willing to use for a row group.

[zivanfi]

This seems like a very reasonable default to me, much better than the current default of 0. Any chance to get it in the codebase?

[rdblue]

I'm still +1 on this. @julienledem, any thoughts?

[julienledem]

+1
@isnotinvain ?

[isnotinvain]

Have you been using the padding in production? Do we have any numbers to show that it helps / doesn't hurt etc?

[rdblue]

CDH has been using this as the default for more than a year. I don't have numbers since we don't use it for S3, but I think it's a good thing to avoid remote reads.

[isnotinvain]

OK, yeah sounds fine to me. If there's trouble with it, it can be set back to 0 by the user right?

[rdblue]

Yeah, users can always override this.

@Zicl, would you like to update this and submit a new PR?

[zivanfi]

@rdblue, sure, I'll do that.

[zivanfi]

@rdblue I created #391 as you suggested.

[rdblue]

Merged #391.