[SPARK-12937][SQL] bloom filter serialization #10920
Conversation
| /** | ||
| * Version number of the serialized binary format for bloom filter or count-min sketch. | ||
| */ | ||
| public enum Version { |
There was a problem hiding this comment.
bloom filter and count-min sketch can have different version values, but we can share same version class.
There was a problem hiding this comment.
I think we should move it back, because:
- The version enum is actually the best place to document the binary protocol.
- This will be really confusing when bloomfilter has v2 and yet count-min sketch has only v1.
- The amount of code duplication you save is teeny (actually you probably added more loc by having an apache licensing header).
There was a problem hiding this comment.
cc @liancheng on point 1 - the best place to document the binary protocol is in Version!
|
cc @rxin @liancheng |
| @@ -24,6 +27,9 @@ | |||
| private long bitCount; | |||
|
|
|||
| static int numWords(long numBits) { | |||
| if (numBits <= 0) { | |||
| throw new IllegalArgumentException("numBits must be positive"); | |||
There was a problem hiding this comment.
also include the current value
|
Test build #50081 has finished for PR 10920 at commit
|
| return versionNumber; | ||
| } | ||
| } | ||
|
|
||
| public abstract Version version(); |
There was a problem hiding this comment.
cc @liancheng , I removed this as the design doc says users should not care about the version being used.
|
Test build #50084 has finished for PR 10920 at commit
|
|
Test build #50086 has finished for PR 10920 at commit
|
|
I'm going to merge this first. Please move the num hash function thing in your next pr. Thanks. |
This PR adds serialization support for BloomFilter.
A version number is added to version the serialized binary format.