A couple of performance optimizations: HyperLogLog and BloomFilter #426
Conversation
…gLogMonoid (streamline and re-use pre-computed hashes for performance).
Bekbolatov
changed the title
|
|
Basically, current changes try to save time by re-using hashed values, in two useful situations. BloomFilter checkAndAdd will allow for a little error during partition merges (reduce): when the same item appears in two different partitions once. Worst case is when all partitions contain exactly same sets of items and items appears at most once in each partition. |
| val contained = this.contains(item) | ||
| (BFInstance(hashes, bits ++ bitsToActivate, width), contained) | ||
| } | ||
|
|
||
| def bitSetContains(bs: BitSet, il: Int*): Boolean = { | ||
| il.map{ i: Int => bs.contains(i) }.reduce{ _ && _ } |
There was a problem hiding this comment.
this could be optimized by stopping at the first false:
val ilit = il.iterator
var contains = true
while(ilit.hasNext && contains) {
contains = contains && bs.contains(ilit.next)
}
containsit is mutable, but it might speed things up measurably.
There was a problem hiding this comment.
Defintiely. I considered this ealier - performance measurements showed hash calculations and bit array conversions trumped everything small like this. If we do add this, how about:
il.foreach { i => if (!bs.contains(i)) return false }
return true
Not sure how we feel about break-out returns on Scala, but we will avoid extra assignments.
| @@ -150,6 +154,8 @@ case class BFItem(item: String, hashes: BFHash, width: Int) extends BF { | |||
|
|
|||
| def +(other: String) = this ++ BFItem(other, hashes, width) | |||
There was a problem hiding this comment.
in ++ should we optimize the case where other.item == item and not allocate? I guess that's rare.
There was a problem hiding this comment.
An extra "if " shortcut is definitely cheaper than hashes calculations - benefit of this trade-off will depend on data consumed. Probably doesn't hurt to put the check there - but as you said, it would happen pretty rarely - data values must be same. Your call.
…ions will require conversions anyway)
|
Thanks. This will be of help! |
A couple of performance optimizations: HyperLogLog and BloomFilter
Added possibility of HLL creation from pre-computed hashes to HyperLogLogMonoid.
This makes it possible to re-use pre-computed hashes and increase performance.