Add ability to pass in Bloom filter from Hive Queries #6222

nishantmonu51 · 2018-08-23T13:15:59Z

This PR adds a BloomDimFilter which can be used by Apache Hive to pass in BloomFilters.

Use Case -
We have fact table in druid and slowly changing dimension/lookup tables in Apache Hive and need to join those tables.
e.g. Consider the case of SSB Benchmark when lineorder is stored in Druid and parts table is in hive For following query from SSB Benchmark -

select sum(total_revenue) from druid.ssb_lineorder_100, hive.ssb_lineorder_100 WHERE lo_partkey = p_partkey and p_category = 'MFGR#14';

In the above query Hive can scan parts table, create a bloom filter for possible values for p_part_key where p_category = 'MFGR#14'. This bloom filter can then be pushed to Druid reducing the data that needs to scanned and transferred between Druid and Hive.
Since BloomFilter is probablistic data structure and can have false positives. Hive will still need to do filtering while processing joins.

gianm · 2018-08-23T16:00:56Z

Hi @nishantmonu51, this is really cool and something that seems useful outside of Hive / Druid interop too. Is there a way to do this without making it so Hive specific? Could the core bloom filter code be extracted into a library and put into a druid-bloom-filter extension?

I am thinking of use cases like approximate x IN (SELECT y FROM … WHERE …) filters done like:

Run the subquery with a bloom filter aggregator that builds a bloom filter object for all y matching the inner query.
Turn around and apply that bloom filter object as a filter on the main query.

It's not exact but with appropriately sized filters it would still be useful for some use cases.

nishantmonu51 · 2018-08-23T16:04:30Z

@gianm: sure, I agree its a useful feature in general.
I can rename the extension to druid-bloom-filter, will leave the dependency on hive-storage-api for bloom Filter implementation, I don't think we need to copy the implementation as hive-storage-api is just has few classes and does not have many transitive dependencies.
Does this sounds good to you ?

QiuMM · 2018-08-23T17:00:20Z

A cool extension.IMO it's better to extract the implementation of BloomKFilter to avoid any possible problems.

nishantmonu51 · 2018-09-08T00:23:07Z

@gianm: Updated based on your comments, please review.

QiuMM · 2018-09-08T02:19:38Z

@nishantmonu51 first there is no doc about how to use this extension. Secondly, I think you forget to add this extension to distribution/pom.xml.

jon-wei · 2018-09-14T01:23:41Z

...ions-core/druid-bloom-filter/src/main/java/org/apache/druid/query/filter/BloomDimFilter.java

+ */
+public class BloomDimFilter implements DimFilter
+{
+  public static byte BLOOM_DIM_FILTER_CACHE_ID = 0x10;


would it be better to put this in DimFilterUtils? I think it'd be easier to have all the filter cache IDs in one place, and this is a core extension

jon-wei · 2018-09-14T01:24:43Z

...re/druid-bloom-filter/src/main/java/org/apache/druid/guice/BloomFilterSerializersModule.java

+    addDeserializer(BloomKFilter.class, new BloomKFilterDeserializer());
+  }
+
+  public static class BloomKFilterserializer extends StdSerializer<BloomKFilter>


BloomKFilterserializer -> BloomKFilterSerializer

jihoonson · 2018-09-17T21:08:28Z

Hi @nishantmonu51, do you think this should be a blocking issue for 0.13.0 release?

jon-wei · 2018-09-17T23:28:11Z

@nishantmonu51 Can you add docs for this? An example of what a serialized BloomFilter input looks like and what it represents would be helpful IMO.

clintropolis

Tests for virtual columns are failing for me, but after making changes mentioned in my review comments (and adding to distribution pom), this extension works for me and is pretty rad!

clintropolis · 2018-09-21T21:19:02Z

.../druid-bloom-filter/src/main/resources/META-INF/services/io.druid.initialization.DruidModule

@@ -0,0 +1 @@
+org.apache.druid.guice.BloomFilterExtensionModule


this file has the wrong name, should be org.apache

clintropolis · 2018-09-21T21:19:51Z

extensions-core/druid-bloom-filter/src/test/java/io/druid/query/filter/BloomDimFilterTest.java

@@ -0,0 +1,388 @@
+/*


This test is located in a folder that does't match the package structure (io/druid/..)

extensions-core/druid-bloom-filter/pom.xml

fjy · 2018-09-25T00:50:27Z

@nishantmonu51 can you address comments? Otherwise we can move this to 0.13.1

nishantmonu51 · 2018-09-25T11:51:11Z

Will update the PR today with the fixes. It will be good to get this in 0.13.0

nishantmonu51 · 2018-09-25T16:36:09Z

@jon-wei @clintropolis @QiuMM Thanks for the review. Added docs and handled review comments.

clintropolis · 2018-09-25T19:36:29Z

This travis failure looks legitimate, but 👍 after travis and conflicts are resolved

fix checkstyle review comments Fix wierd failure review comments Revert "Fix wierd failure" This reverts commit a13a83a.

clintropolis · 2018-09-25T19:51:08Z

By the way, I built a bloom filter aggregator on top of this last weekend once I figured out BloomKFilter had a merge function. This way we can generate the bloom filters to feed into this directly from queries, which I'll submit in a PR once this gets merged. It's pretty similar to the cardinality agg. It's currently backported to a 0.12.3 branch of ours, and is missing tests, but it shouldn't be much to get it into master.

himanshug · 2018-09-25T20:16:05Z

docs/content/development/extensions-core/bloom-filter.md

+|-------------------------|------------------------------|----------------------------------|
+|`type`                   |Filter Type. Should always be `bloom`|yes|
+|`dimension`              |The dimension to filter over. | yes |
+|`bloomKFilter`           |Binary representation of `org.apache.hive.common.util.BloomKFilter`| yes |


nits:
"Base64 encoded Binary representation....." ?

Also, It appears to me that this module(and property) are really usable only if someone was using hive util BloomKFilter class on the client side because serialized representation is specific to that class and not something standard, so maybe we could add "hive" in the names everywhere e.g. hive-bloom-filter module and this property being hiveUtilBloomKFilter .
or I'm missing something :)

updated, yes, its true that the implementation is currently shared from hive-storage-api jar, but the implementation as such does not depend on any of the hive specific functionality.
I initially made this a hive-bloom-filter but later on renamed to bloom-filter based on review comments.

I think it's better to not have hive in the names. I can imagine evolving this extension to support some non-hive format too, and make it more divorced from hive. Especially since with the aggregator @clintropolis mentioned, people could create serialized bloom filters without really needing to worry about the fact that they came from hive.

To be clear, if we did evolve it that way, it should still support the hive format (to work with the original use case from @nishantmonu51). I just think there is a lot of potential for this extension to be useful beyond hive integration.

ah, I understand the naming now. ( I knew I was missing something ).

jihoonson

Looks that teamcity fail isn't related to this PR.

nishantmonu51 added this to the 0.13.0 milestone Aug 23, 2018

nishantmonu51 added Performance Area - Querying Area - Extension labels Aug 23, 2018

nishantmonu51 force-pushed the test-bloom-filter branch from 6458fd3 to 59d6a07 Compare August 23, 2018 15:36

nishantmonu51 force-pushed the test-bloom-filter branch from 59d6a07 to ea066c1 Compare September 8, 2018 00:15

nishantmonu51 added the Feature label Sep 8, 2018

jon-wei reviewed Sep 14, 2018

View reviewed changes

clintropolis requested changes Sep 23, 2018

View reviewed changes

nishantmonu51 force-pushed the test-bloom-filter branch from ea066c1 to e8aec5c Compare September 25, 2018 16:34

nishantmonu51 added 2 commits September 26, 2018 01:10

Bloom filter initial implementation

9aeccab

fix checkstyle review comments Fix wierd failure review comments Revert "Fix wierd failure" This reverts commit a13a83a.

fix test

305d083

nishantmonu51 force-pushed the test-bloom-filter branch from e8aec5c to 305d083 Compare September 25, 2018 19:50

himanshug reviewed Sep 25, 2018

View reviewed changes

clintropolis approved these changes Sep 25, 2018

View reviewed changes

review comment

b48ee5c

himanshug approved these changes Sep 26, 2018

View reviewed changes

jihoonson approved these changes Sep 26, 2018

View reviewed changes

jon-wei merged commit c9d281a into apache:master Sep 26, 2018

clintropolis mentioned this pull request Sep 28, 2018

Adds bloom filter aggregator to 'druid-bloom-filters' extension #6397

Merged

dclim mentioned this pull request Oct 10, 2018

Druid 0.13.0-incubating release notes #6442

Closed

gianm mentioned this pull request Oct 30, 2018

Modify BloomDimFilter to use BloomFilter instead of BloomKFilter which is not thread safe #6547

Closed

clintropolis mentioned this pull request Nov 7, 2018

fix druid-bloom-filter thread-safety #6584

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Add ability to pass in Bloom filter from Hive Queries #6222

Add ability to pass in Bloom filter from Hive Queries #6222

nishantmonu51 commented Aug 23, 2018

gianm commented Aug 23, 2018

nishantmonu51 commented Aug 23, 2018

QiuMM commented Aug 23, 2018

nishantmonu51 commented Sep 8, 2018

QiuMM commented Sep 8, 2018 •

edited

Loading

jon-wei Sep 14, 2018

jon-wei Sep 14, 2018

jihoonson commented Sep 17, 2018

jon-wei commented Sep 17, 2018

clintropolis left a comment •

edited

Loading

clintropolis Sep 21, 2018

nishantmonu51 Sep 25, 2018

clintropolis Sep 21, 2018

nishantmonu51 Sep 25, 2018

fjy commented Sep 25, 2018

nishantmonu51 commented Sep 25, 2018

nishantmonu51 commented Sep 25, 2018

clintropolis commented Sep 25, 2018

clintropolis commented Sep 25, 2018

himanshug Sep 25, 2018

nishantmonu51 Sep 26, 2018

gianm Sep 26, 2018 •

edited

Loading

gianm Sep 26, 2018

himanshug Sep 26, 2018

jihoonson left a comment

		@@ -0,0 +1 @@
		org.apache.druid.guice.BloomFilterExtensionModule

Add ability to pass in Bloom filter from Hive Queries #6222

Add ability to pass in Bloom filter from Hive Queries #6222

Conversation

nishantmonu51 commented Aug 23, 2018

gianm commented Aug 23, 2018

nishantmonu51 commented Aug 23, 2018

QiuMM commented Aug 23, 2018

nishantmonu51 commented Sep 8, 2018

QiuMM commented Sep 8, 2018 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

jihoonson commented Sep 17, 2018

jon-wei commented Sep 17, 2018

clintropolis left a comment • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

fjy commented Sep 25, 2018

nishantmonu51 commented Sep 25, 2018

nishantmonu51 commented Sep 25, 2018

clintropolis commented Sep 25, 2018

clintropolis commented Sep 25, 2018

Choose a reason for hiding this comment

Choose a reason for hiding this comment

gianm Sep 26, 2018 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

jihoonson left a comment

Choose a reason for hiding this comment

QiuMM commented Sep 8, 2018 •

edited

Loading

clintropolis left a comment •

edited

Loading

gianm Sep 26, 2018 •

edited

Loading