Implement Filter::mergeWith for MultiRange #299

amaliujia · 2021-09-24T22:17:37Z

Implement Filter::mergeWith for MultiRange.
Filter clone() now allows overriding nullAllowed.

facebook-github-bot · 2021-09-24T22:17:41Z

Thank you for your pull request and welcome to our community.

Action Required

In order to merge any pull request (code, docs, etc.), we require contributors to sign our Contributor License Agreement, and we don't seem to have one on file for you.

Process

In order for us to review and merge your suggested changes, please sign at https://code.facebook.com/cla. If you are contributing on behalf of someone else (eg your employer), the individual CLA may not be sufficient and your employer may need to sign the corporate CLA.

Once the CLA is signed, our tooling will perform checks and validations. Afterwards, the pull request will be tagged with CLA signed. The tagging process may take up to 1 hour after signing. Please give it that time before contacting us about it.

If you have received this in error or have any questions, please contact us at cla@fb.com. Thanks!

amaliujia · 2021-09-24T22:19:19Z

cc @mbasmanova

facebook-github-bot · 2021-09-24T23:06:47Z

Thank you for signing our Contributor License Agreement. We can now accept your code for this (and any) Facebook open source project. Thanks!

mbasmanova

Looks nice. I made some comments. Adding a test would help flesh our corner cases.

velox/type/Filter.cpp

mbasmanova · 2021-09-25T02:08:29Z

velox/type/Filter.cpp

+      // null, then this filter should be treated as isNULL.
+      if (bothNullAllowed && !merged[0]->testNull()) {
+        auto replaceFilter =
+            merged[0]->mergeWith(std::make_unique<IsNull>().get());


This is not quite right. We want OR semantics here, not AND. We may need to modify the clone() method to take an optional new value for the nullAllowed flag. Also, we don't want to wrap a single filter in a MultiRange, rather return it as is.

I see. I think I have to return IsNull directly.

mbasmanova · 2021-09-25T02:09:03Z

velox/type/Filter.cpp

+    return std::unique_ptr<Filter>(
+        new MultiRange(std::move(merged), bothNullAllowed, nanAllowed_));
+  } else {
+    return this->clone();


This is not quite right as we still need to merge with the other filter. See existing implementations of mergeWith.

Tried to add more cases. PLAT

amaliujia · 2021-09-27T22:08:13Z

@mbasmanova

I tried to address your comments and added some unit tests. Can you take a look?

meanwhile, can you share some examples of testing nulless? For multirange nullness tests, it's a bit complicated:

there are difference between top-level and inner-level nullness, there are also left/right multi-range that have not the same nullAllowed.

Until this moment, I am actually not sure the rule of dealing with null in Velox? If one filter allows NULL and another filter does not allow it, does their AND allow or not allow NULL? how about OR?

amaliujia · 2021-09-27T22:39:58Z

cc @atanu1991 @kgpai

mbasmanova

Some comments below.

MultiRange represents an OR of 2 or more filters. The nullAllowed flag in MultiRange is used to determine whether a null value passes the filter or not. The nullAllowed flags of the contained filters are not considered.

mbasmanova · 2021-09-28T23:35:46Z

velox/type/Filter.cpp

@@ -423,6 +423,18 @@ bool MultiRange::testFloat(float value) const {
  return false;
 }

+bool MultiRange::testInt64(int64_t value) const {


MultiRange filters are not used for integral types. For these types, we only use BigintMultiRange. Hence, MultiRange::testInt64 should just throw.

mbasmanova · 2021-09-28T23:37:42Z

velox/type/Filter.cpp

+          auto innerMerged = filter->mergeWith(filterOther.get());
+          switch (innerMerged->kind()) {
+            case FilterKind::kAlwaysFalse:
+              break;


Shouldn't this be "continue;"? E.g. (a or b) AND (c or d) = (a and c) or (a and d) or (b and c) or (b and d). If one of these is false, we just drop it but keep the rest.

This break is to break the switch, not the nested loop? Thus this is doing what you are saying If one of these is false, we just drop it but keep the rest..

mbasmanova · 2021-09-28T23:40:33Z

velox/type/Filter.cpp

+            case FilterKind::kIsNull:
+              // the merged filter is valid when it matches with top-level
+              // nullAllowed.
+              if (innerMerged->testNull() && bothNullAllowed) {


I think kIsNotNull is not possible. kIsNull can be treated same as kAlwaysFalse, since, bothNullAllowed alone will determine whether null value passes the combined filter.

mbasmanova · 2021-09-28T23:41:16Z

velox/type/Filter.cpp

+      if (merged.empty()) {
+        return nullOrFalse(bothNullAllowed);
+      } else if (merged.size() == 1) {
+        // If both sides allow null while the merged inner filter does not allow


We should not consider nullAllowed flags of the inner filters, only top-level nullAllowed flag should be used. Hence, we need to make a clone of merged[0] with nullAllowed = bothNullAllowed

mbasmanova · 2021-09-28T23:43:27Z

velox/type/Filter.cpp

+    case FilterKind::kIsNull:
+      return other->mergeWith(this);
+    case FilterKind::kIsNotNull: {
+      std::unique_ptr<Filter> clonedFilter = this->clone();


can we do this only if this filter's nullAllowed is false?

I see. So null AND not null should be null still. Only not null AND not null will be not null.

Sure but what to return when it is null AND not null?

I am thinking it should just return this->clone() when null ADN not null.

Here is how I'm thinking about this. There are 2 possibilities for a MultiRange. Either it allows a null value or not.

MultiRange(nullAllowed = true) AND IS NULL => IS NULL

MultiRange(nullAllowed = true) AND IS NOT NULL => MultiRange(nullAllowed = false)

MultiRange(nullAllowed = false) AND IS NULL => ALWAYS FALSE

MultiRange(nullAllowed = false) AND IS NOT NULL => MultiRange(nullAllowed = false)

Does this make sense?

Excellence summary. I even added this summary as comments in code.

mbasmanova · 2021-09-28T23:43:59Z

velox/type/Filter.cpp

+      return std::make_unique<MultiRange>(
+          std::move(filters), /*nullAllowed=*/false, this->nanAllowed_);
+    }
+    case FilterKind::kBoolValue:


perhaps, not necessary to list all these values

velox/type/tests/FilterTest.cpp

velox/type/Filter.cpp

mbasmanova

Looks great. Just a few comments.

velox/type/Filter.cpp

velox/type/Filter.h

atanu1991 · 2021-10-01T00:48:19Z

velox/type/Filter.cpp

+      return other->mergeWith(this);
+    case FilterKind::kIsNotNull: {
+      if (this->nullAllowed_) {
+        return this->clone(/*nullAllowed=*/false);


@mbasmanova I dont understand why
MultiRange(nullAllowed=true) AND IS NOT NULL => MultiRange(nullAllowed=false)

A value passes F1 AND F2 only if it passes both F1 and F2. When F2 = IS NOT NULL, any value other than null passes, e.g. null does not pass. However, null passes MultiRange(nullAllowed=true). To exclude null we simply change to MultiRange(nullAllowed=false)

atanu1991 · 2021-10-01T00:53:04Z

velox/type/Filter.h

@@ -1083,8 +1154,26 @@ class BytesValues final : public Filter {
    upper_ = *std::max_element(values_.begin(), values_.end());
  }

-  std::unique_ptr<Filter> clone() const final {
-    return std::make_unique<BytesValues>(*this);
+  BytesValues(const BytesValues& other, bool nullAllowed)


Instead of repeating code (which will need changes twice if we change the logic), we can do something like re-using the above constructor/assigning the value directly. Its also more efficient imho (I guess)?

a12933b#diff-280c77e7a8b1d7cfaeaaed70f8ac11c67e881b4f1682ee099bc326a1b53e9349

I actually tried this style, for example:
BytesValues(const BytesValues& other, bool nullAllowed) {
// call the existing constructor.
}

I am from Java world and this is how usually Java works.

However the compile complains and say bt this way the default base class is not initialized. Do you know how to fix this issue?

Not sure about the compiler complaining. When I did the same as pointed in the diff, it seemed to work.

atanu1991 · 2021-10-01T00:54:59Z

velox/type/Filter.h

@@ -1021,8 +1072,28 @@ class BytesRange final : public AbstractRange {
            !lowerExclusive_ && !upperExclusive_ && !lowerUnbounded_ &&
            !upperUnbounded_ && lower_ == upper_) {}

-  std::unique_ptr<Filter> clone() const final {
-    return std::make_unique<BytesRange>(*this);
+  BytesRange(const BytesRange& other, bool nullAllowed)


Same comment

a12933b#diff-280c77e7a8b1d7cfaeaaed70f8ac11c67e881b4f1682ee099bc326a1b53e9349

atanu1991 · 2021-10-01T00:57:11Z

velox/type/tests/FilterTest.cpp

+  }
+}
+
+TEST(FilterTest, cloneTest) {


Could you explain why all of them are expected to return true for nullallowed?

naming: cloneTest -> clone

Those filters are constructed by setting nullAllowed as false. Then I explicitly call clone(true) to reset the nullAllowed to true, and then expect testIsNull returns true.

atanu1991 · 2021-10-01T01:01:09Z

velox/type/Filter.cpp

+      if (merged.empty()) {
+        return nullOrFalse(bothNullAllowed);
+      } else if (merged.size() == 1) {
+        return merged[0]->clone(bothNullAllowed);


Why not return std::move(merged.front())

Are we cloning to account for bothNullAllowed ?

Yes. This is just ensure that the only filter matches the top level nullAllowed. For MultiRange merge, we will ignore inner filters' nullness.

atanu1991 · 2021-10-01T01:05:12Z

velox/type/Filter.cpp

+            new MultiRange(std::move(merged), bothNullAllowed, bothNanAllowed));
+      }
+    }
+      /*


Doxygen style? https://github.com/facebookincubator/velox/blob/main/CODING_STYLE.md

mbasmanova

Looks great. Just a couple of comments remaining. Please, update the PR description.

mbasmanova · 2021-10-01T01:30:42Z

velox/type/Filter.cpp

+    case FilterKind::kIsNull:
+      return other->mergeWith(this);
+    case FilterKind::kIsNotNull: {
+      if (this->nullAllowed_) {


nit: perhaps, shorten to return this->clone(false);

mbasmanova · 2021-10-01T01:31:45Z

velox/type/tests/FilterTest.cpp

+  }
+}
+
+TEST(FilterTest, cloneTest) {


naming: cloneTest -> clone

amaliujia · 2021-10-01T02:48:02Z

Have updated PR description.

mbasmanova

Looks great. I have one final ask and this will be ready to merge.

mbasmanova · 2021-10-01T07:13:43Z

velox/type/Filter.cpp

+    case FilterKind::kIsNotNull:
+      return this->clone(/*nullAllowed=*/false);
+    default:
+      VELOX_UNREACHABLE();


Looks like the remaining things to add here would be:

kDoubleRange
kFloatRange
kBytesRange
kBytesValues

These can come in a follow-up PR though.

One ask here is to change the order of the 'case' statements to match all other implementations to allow for easier side-by-side comparison, e.g.

case FilterKind::kAlwaysTrue: case FilterKind::kAlwaysFalse: case FilterKind::kIsNull: return other->mergeWith(this); case FilterKind::kIsNotNull: return this->clone(/*nullAllowed=*/false); case FilterKind::kDoubleRange: case FilterKind::kFloatRange: case FilterKind::kBytesRange: case FilterKind::kBytesValues: // TODO Implement VELOX_UNREACHABLE(); case FilterKind::kMultiRange: ... default: VELOX_UNREACHABLE();

It makes sense. Done.

facebook-github-bot · 2021-10-01T07:14:50Z

@mbasmanova has imported this pull request. If you are a Facebook employee, you can view this diff on Phabricator.

facebook-github-bot · 2021-10-01T18:19:17Z

@mbasmanova has imported this pull request. If you are a Facebook employee, you can view this diff on Phabricator.

mbasmanova · 2021-10-01T21:07:29Z

@amaliujia Rui, thank you for the contribution.

…or#299)

amaliujia mentioned this pull request Sep 24, 2021

Implement Filter::mergeWith for MultiRange #154

Closed

facebook-github-bot added the CLA Signed This label is managed by the Facebook bot. Authors need to sign the CLA before a PR can be reviewed. label Sep 24, 2021

mbasmanova requested review from atanu1991 and kgpai September 25, 2021 01:57

mbasmanova requested changes Sep 25, 2021

View reviewed changes

amaliujia mentioned this pull request Sep 27, 2021

Implement mergewith for bytesRange and bytesValue #297

Closed

amaliujia changed the title ~~[WIP] Implement Filter::mergeWith for MultiRange~~ Implement Filter::mergeWith for MultiRange Sep 28, 2021

mbasmanova reviewed Sep 28, 2021

View reviewed changes

amaliujia force-pushed the main branch from bbf7f1d to a3245f8 Compare September 29, 2021 23:16

mbasmanova reviewed Sep 30, 2021

View reviewed changes

velox/type/tests/FilterTest.cpp Outdated Show resolved Hide resolved

velox/type/Filter.cpp Outdated Show resolved Hide resolved

velox/type/Filter.cpp Outdated Show resolved Hide resolved

mbasmanova reviewed Sep 30, 2021

View reviewed changes

velox/type/Filter.cpp Outdated Show resolved Hide resolved

velox/type/Filter.cpp Outdated Show resolved Hide resolved

velox/type/Filter.h Outdated Show resolved Hide resolved

atanu1991 reviewed Oct 1, 2021

View reviewed changes

mbasmanova reviewed Oct 1, 2021

View reviewed changes

mbasmanova approved these changes Oct 1, 2021

View reviewed changes

amaliujia closed this Oct 1, 2021

amaliujia reopened this Oct 1, 2021

Implement Filter::mergeWith for MultiRange. (Rui Wang)

5247f98

amaliujia force-pushed the main branch from d964d1b to 5247f98 Compare October 1, 2021 05:27

mbasmanova approved these changes Oct 1, 2021

View reviewed changes

address comments

648f88d

facebook-github-bot closed this in 61ae37d Oct 1, 2021

rui-mo added a commit to rui-mo/velox that referenced this pull request Mar 17, 2023

[OPPRO-167] Remove two metrics and change decription (facebookincubat…

5e7f452

…or#299)

rui-mo added a commit to rui-mo/velox that referenced this pull request Jun 8, 2023

Fallback murmur3hash on complex types (facebookincubator#299)

e3ec2b9

liujiayi771 pushed a commit to liujiayi771/velox that referenced this pull request Jun 9, 2023

Fallback murmur3hash on complex types (facebookincubator#299)

0016364

PHILO-HE pushed a commit to PHILO-HE/velox that referenced this pull request Jun 27, 2023

Fallback murmur3hash on complex types (facebookincubator#299)

56dab28

zhli1142015 pushed a commit to zhli1142015/velox that referenced this pull request Jul 3, 2023

Fallback murmur3hash on complex types (facebookincubator#299)

addc9d3

Yohahaha pushed a commit to Yohahaha/velox that referenced this pull request Jul 4, 2023

Fallback murmur3hash on complex types (facebookincubator#299)

7e9b9b7

chenxu14 pushed a commit to chenxu14/velox that referenced this pull request Jul 5, 2023

Fallback murmur3hash on complex types (facebookincubator#299)

008ed26

PHILO-HE pushed a commit to PHILO-HE/velox that referenced this pull request Jul 17, 2023

Fallback murmur3hash on complex types (facebookincubator#299)

096ee58

rui-mo added a commit to rui-mo/velox that referenced this pull request Jul 21, 2023

Fallback murmur3hash on complex types (facebookincubator#299)

37f4257

rui-mo added a commit to rui-mo/velox that referenced this pull request Jul 24, 2023

Fallback murmur3hash on complex types (facebookincubator#299)

60eddca

Implement Filter::mergeWith for MultiRange #299

Implement Filter::mergeWith for MultiRange #299

Conversation

amaliujia commented Sep 24, 2021 • edited Loading

facebook-github-bot commented Sep 24, 2021

Action Required

Process

amaliujia commented Sep 24, 2021

facebook-github-bot commented Sep 24, 2021

mbasmanova left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

amaliujia commented Sep 27, 2021

amaliujia commented Sep 27, 2021

mbasmanova left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

amaliujia Sep 29, 2021 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

mbasmanova left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

atanu1991 Oct 1, 2021 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

atanu1991 Oct 1, 2021 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

mbasmanova left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

amaliujia commented Oct 1, 2021

mbasmanova left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

facebook-github-bot commented Oct 1, 2021

facebook-github-bot commented Oct 1, 2021

mbasmanova commented Oct 1, 2021

amaliujia commented Sep 24, 2021 •

edited

Loading

amaliujia Sep 29, 2021 •

edited

Loading

atanu1991 Oct 1, 2021 •

edited

Loading

atanu1991 Oct 1, 2021 •

edited

Loading