PARQUET-2257: Add bloom_filter_length to ColumnMetaData

[wgtmac]

The specs has only added bloom_filter_offset to locate the bloom filter. The reader cannot load the bloom filter in a single shot until it parses the bloom filter header to get the total size.

struct ColumnMetaData {
  /** Byte offset from beginning of file to Bloom filter data. **/
  14: optional i64 bloom_filter_offset;
}

This patch proposes to add an optional bloom_filter_length field to track the size of bloom filter to facilitate I/O scheduling. The specs already do the similar things for column index and offset index.

[wgtmac]

@emkornfield @pitrou @gszadovszky @shangxinli @mapleFU

[mapleFU]

Seems that if length exists, offset must exists. However, if offset not exists, length should not exist?

And reader should check that:

1. Does it have offset
2. If has, can it go fast path that using bloom_filter_length to read?

[wgtmac]

On the writer side:

• Old writer only writes offset.
• New writer should write length as well.

On the reader side:

• Old reader only checks offset.
• New reader checks offset and then checks if length exists.

[emkornfield]

for future implementers we should add a comment noting that existing of one element does not indicate existence of the other (even though it is already implied by mapping).

[wgtmac]

Agreed.

I have thought to add a comment to say that bloom_filter_length is introduced by v2.10.0 (the next unreleased version) and readers may not expect its existence from legacy files or some implementations.

[wgtmac]

@emkornfield @pitrou Do you have any comment?

[emkornfield]

One minor comment otherwise looks OK to me, apologies for the late review.

[wgtmac]

One minor comment otherwise looks OK to me, apologies for the late review.

I have updated the comment. Please take a look, thanks! @emkornfield