Allow filters to use extraction functions #2690

jon-wei · 2016-03-19T01:56:04Z

This PR allows the user to define an extraction function on each of the filters (except SpatialFilter); the extraction function will be applied to the input before the filter criteria is applied.

This merges the SelectorFilter and ExtractionFilter into a single class (ExtractionDimFilter is deprecated in this patch, ExtractionFilter has been deleted).

Fixes #2643

Also fixes #2772, #2775, #2776

jon-wei · 2016-03-21T18:49:57Z

updated docs

jon-wei · 2016-03-23T22:12:59Z

rebased, removed conflicts with #2711

navis · 2016-03-24T00:53:46Z

I think JavaScriptFilter and SelectorFilter also can extend DimensionPredicateFilter, removing some duplicated codes.

navis · 2016-03-24T00:55:07Z

processing/src/main/java/io/druid/segment/filter/DimensionPredicateFilter.java


  public DimensionPredicateFilter(
      String dimension,
-      Predicate<String> predicate
+      Predicate<String> predicate,
+      ExtractionFn extractionFn


cause extraction itself is handled in predicate, I think we don't need this here (can be confusing).

@navis thanks, extractionFn was unnecessary here, removed it

but now it's necessary again, right?

yes, it's being used now

gianm · 2016-03-24T01:21:32Z

sorry @jon-wei, more conflicts!

jon-wei · 2016-03-24T03:03:53Z

@navis

re: changing SelectorFilter and JavascriptFilter to implement DimensionPredicateFilter

There are some performance considerations if that change were made:

getBitmapIndex() for DimensionPredicateFilter requires a scan through the column's dictionary values to determine which values matched the predicate. For SelectorFilter without an extraction function, the current getBitmapIndex() is much simpler/faster since it can just call selector.getBitmapIndex(dimension, value).
For JavascriptFilter, the current implementation allows getBitmapIndex() to reuse the Context object across the scan of the dictionary. Using the JavascriptPredicate with DimensionPredicateFilter's getBitmapIndex() function would result in a new Context object being created on every apply() call. There's a note regarding this performance consideration at:
https://github.com/druid-io/druid/blob/master/processing/src/main/java/io/druid/segment/filter/JavaScriptFilter.java#L92

I think, if that change is desirable, it's better handled in a separate PR.

jon-wei · 2016-03-24T03:05:25Z

@gianm np, rebasing now

jon-wei · 2016-03-24T03:47:08Z

rebased

navis · 2016-03-24T06:52:30Z

@jon-wei Pair enough. 👍

gianm · 2016-03-24T16:57:58Z

processing/src/main/java/io/druid/query/filter/BoundDimFilter.java

+    this.extractionFn = extractionFn;
+  }
+
+  public BoundDimFilter(


Is this constructor needed? If it's only used for tests then usually we modify the tests rather than add another constructor.

(similar comment for all the other filters)

IMO one nice thing about only having one constructor is that it makes it harder to accidentally create an object with a critical parameter missing.

@gianm removed extra constructors

jon-wei · 2016-04-01T23:49:46Z

Addressed comments, rebasing now

This PR also fixes a few issues related to filtering, I've opened issues for them:
#2772
#2775
#2776

jon-wei · 2016-04-02T00:18:51Z

rebased

gianm · 2016-04-04T18:37:35Z

processing/src/main/java/io/druid/query/filter/ExtractionDimFilter.java

@@ -98,11 +102,25 @@ public DimFilter optimize()
  {
    if (this.getExtractionFn() instanceof LookupExtractionFn


why not return new SelectorDimFilter(dimension, value, extractionFn).optimize();?

@gianm good idea, I'll change that

@gianm changed to use SelectorDimFilter optimize()

gianm · 2016-04-04T19:23:56Z

processing/src/main/java/io/druid/query/filter/InDimFilter.java


  @JsonCreator
-  public InDimFilter(@JsonProperty("dimension") String dimension, @JsonProperty("values") List<String> values)


This could be optimized in a similar way to the SelectorDimFilter.

@gianm Can we handle the InDimFilter optimization in a separate PR?

sure, you can file an issue for anything you don't think is appropriate for this PR.

gianm · 2016-04-05T06:23:45Z

processing/src/main/java/io/druid/query/filter/SelectorDimFilter.java

+      // in the lookup map. Match on the selector value as well.
+      // If the selector value is overwritten in the lookup map, don't add selector value to keys.
+      if (exFn.isRetainMissingValue() && mappingForValue == null) {
+        keys.add(convertedValue);


keys isn't guaranteed to be mutable, so it'd be better to copy it and add convertedValue to the copy.

this looks good to me other than that

gianm · 2016-04-05T20:46:03Z

lgtm 👍

jon-wei force-pushed the filter_support branch from f3aadf6 to 1683163 Compare March 21, 2016 18:49

jon-wei added this to the 0.9.1 milestone Mar 21, 2016

jon-wei added the Improvement label Mar 21, 2016

gianm mentioned this pull request Mar 23, 2016

All Filters should work with FilteredAggregators. #2711

Merged

jon-wei force-pushed the filter_support branch from 1683163 to ea0ac74 Compare March 23, 2016 22:02

jon-wei changed the title ~~Allow all filters to use extraction functions, allow FilteredAggregators to use all filter types~~ Allow filters to use extraction functions Mar 23, 2016

jon-wei force-pushed the filter_support branch from ea0ac74 to a92afd5 Compare March 23, 2016 22:05

navis reviewed Mar 24, 2016
View reviewed changes

jon-wei force-pushed the filter_support branch from a92afd5 to 6372f0a Compare March 24, 2016 03:42

gianm reviewed Mar 24, 2016
View reviewed changes

jon-wei force-pushed the filter_support branch 2 times, most recently from 60a896a to a34c2f1 Compare April 1, 2016 23:11

jon-wei added the Bug label Apr 1, 2016

jon-wei force-pushed the filter_support branch from a34c2f1 to 56664b7 Compare April 2, 2016 00:18

gianm reviewed Apr 4, 2016
View reviewed changes

jon-wei mentioned this pull request Apr 4, 2016

missing value handling on LookupExtractionFn, Fix for #2775 #2785

Closed

gianm reviewed Apr 4, 2016
View reviewed changes

jon-wei mentioned this pull request Apr 5, 2016

Optimize InDimFilter #2787

Closed

gianm reviewed Apr 5, 2016
View reviewed changes

Allow filters to use extraction functions

0e481d6

jon-wei force-pushed the filter_support branch from be746c0 to 0e481d6 Compare April 5, 2016 20:25

fjy merged commit 289bb6f into apache:master Apr 5, 2016

fjy mentioned this pull request May 31, 2016

[WIP] Druid 0.9.1 Release Notes #2999

Closed

gianm mentioned this pull request Jun 21, 2016

Large change in cache hit rate for 0.9.1 #3169

Closed

jon-wei deleted the filter_support branch October 6, 2017 22:23

seoeun25 pushed a commit to seoeun25/incubator-druid that referenced this pull request Jan 10, 2020

apache#2690 Stream query with order by on metric is not working

36aeed5

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Allow filters to use extraction functions #2690

Allow filters to use extraction functions #2690

jon-wei commented Mar 19, 2016

jon-wei commented Mar 21, 2016

jon-wei commented Mar 23, 2016

navis commented Mar 24, 2016

navis Mar 24, 2016

jon-wei Mar 24, 2016

gianm Mar 31, 2016

jon-wei Apr 1, 2016

gianm commented Mar 24, 2016

jon-wei commented Mar 24, 2016

jon-wei commented Mar 24, 2016

jon-wei commented Mar 24, 2016

navis commented Mar 24, 2016

gianm Mar 24, 2016

gianm Mar 24, 2016

jon-wei Mar 25, 2016

jon-wei commented Apr 1, 2016

jon-wei commented Apr 2, 2016

gianm Apr 4, 2016

jon-wei Apr 4, 2016

jon-wei Apr 4, 2016

gianm Apr 4, 2016

jon-wei Apr 4, 2016

gianm Apr 4, 2016

gianm Apr 5, 2016

gianm Apr 5, 2016

jon-wei Apr 5, 2016

gianm commented Apr 5, 2016

		@@ -98,11 +102,25 @@ public DimFilter optimize()
		{
		if (this.getExtractionFn() instanceof LookupExtractionFn


		@JsonCreator
		public InDimFilter(@JsonProperty("dimension") String dimension, @JsonProperty("values") List<String> values)

Allow filters to use extraction functions #2690

Allow filters to use extraction functions #2690

Conversation

jon-wei commented Mar 19, 2016

jon-wei commented Mar 21, 2016

jon-wei commented Mar 23, 2016

navis commented Mar 24, 2016

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

gianm commented Mar 24, 2016

jon-wei commented Mar 24, 2016

jon-wei commented Mar 24, 2016

jon-wei commented Mar 24, 2016

navis commented Mar 24, 2016

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

jon-wei commented Apr 1, 2016

jon-wei commented Apr 2, 2016

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

gianm commented Apr 5, 2016