Reduce bloom filter size by using the optimal count for hash functions.

[jfboeuf]

BloomFilteringPostingsFormat currently relies on a bloom filter with one hash function (k=1). For a target false positive probability of 10%, 1 is never the optimal value for k. Using the best value for k would either:

• achieve a much better fpp with the same bitset size, or
• achieve the same fpp with a reduced size (half of what is currently used.

From tests:

• targeting a better false positive probability doesn't bring significant enough better performance for the increased size;
• Targeting a smaller size by degrading the false positive probability comes with a significant performance hit.

As consequence, a target false positive probability of about 10% seems to be a good trade-off. I slightly raised this value (to 0.1023f) so the size of newly allocated bloom filters is always half the size of what they used to be. The effective false positive probability varies from significantly better in most cases to slightly worse in rare cases. This graph compares both size and effective false positive probability of the current and proposed implementations. Overall performance remains comparable (slightly but not significantly better); the reduced size and the improved false positive probability compensate for the cost of having additional hashes. You can find in branch bloomPerfBench the class BloomBench I used to check for performance.

In addition, the implementation of the bitset is based on a long array, so picking up a size lower than 64 bits is pointless.

API change:

• HashFunction.hash(BytesRef) returns a long: more accuracy with a 64bits hash useful to derivate additional hashes from the original one.

The proposed implementation remains compatible with existing/persisted bloom filters.

[jpountz]

Nice, thanks for improving this. I left some minor comments.

[jpountz]

You can drop versions 2 and 3: non-default codecs do not have to remain backward compatible.

[jfboeuf]

Awesome, breaking backward compatibility will allow lots of cleaning up!

[jpountz]

You can use BitUtil.VH_LE_LONG instead.

[jfboeuf]

Thanks, I will do it.

[jpountz]

@jfboeuf I took a stab at removing the versioning logic to simplify the change, I plan on merging it soon if this works for you.

[jfboeuf]

Thank you very much for taking care of this and sorry for the late reply: I had been far from home for long vacations for the past months.