mango: fix $beginsWith range

[willholley]

Overview

In the initial implementation of $beginsWith, the range calculation for view indexes mistakenly appends an integer with the size of 8 bits which gets maxed out at FF, rather than building a binary with an extra 3 bytes at the end.

This PR fixes the mango_idx_view:range/5 by correctly appending the U+FFFF code point to create a utf-8 encoded binary. Additionally, the Erlang utf8 binary type ensures the result is a valid utf8 string. If Arg is not a utf8 binary, this will throw a badarg error.

U+FFFF is used instead of U+10FFFF as it's the highest sorting code point according to the collator rules.

We expect Arg strings to be a valid utf8 but, to be safe, mango_selector:norm_ops/1 is enhanced to verify that any argument to $beginsWith is a utf8 string.

Testing recommendations

make mango-test

Related Issues or Pull Requests

Initial implementation: #4810

Checklist

• Code is written and works correctly
• Changes are covered by tests
• Any new configurable parameters are documented in rel/overlay/etc/default.ini
• Documentation changes were made in the src/docs folder
• Documentation changes were backported (separated PR) to affected branches

[pgj]

Please add more unit test cases to cover strings of invalid UTF-8 data.

[willholley]

@pgj I'm trying to think where it would make sense to add unit tests for non-UTF8 data. There are no existing eunit tests for field_ranges or range (perhaps we add some?) and I don't think it's possible to generate non-UTF8 data via the integration tests.

[pgj]

If you implement the pre-check that I suggested then it can be established for mango_idx_view:range/5 that the input will always be a valid UTF-8 input hence no tests are needed. The requirement for the test could be then pushed to the validation phase.

When I experimented with this fix locally, I extended mango_selector:norm_ops/1 as follows:

diff --git a/src/mango/src/mango_selector.erl b/src/mango/src/mango_selector.erl
index 93d3b10ca..24660e963 100644
--- a/src/mango/src/mango_selector.erl
+++ b/src/mango/src/mango_selector.erl
@@ -136,7 +136,10 @@ norm_ops({[{<<"$text">>, Arg}]}) when
 norm_ops({[{<<"$text">>, Arg}]}) ->
     ?MANGO_ERROR({bad_arg, '$text', Arg});
 norm_ops({[{<<"$beginsWith">>, Arg}]} = Cond) when is_binary(Arg) ->
-    Cond;
+    case couch_util:validate_utf8(Arg) of
+        true -> Cond;
+        false -> ?MANGO_ERROR({bad_arg, '$beginsWith', Arg})
+    end;
 % Not technically an operator but we pass it through here
 % so that this function accepts its own output. This exists
 % so that $text can have a field name value which simplifies

With that, the existing definition of match_beginswith_test/0 in the same module could be extended naturally.

[willholley]

@pgj updated as per #4829

[pgj]

<<123>> is the field name here not the prefix. Is this a mistake...?

[willholley]

my mistake - corrected

[nickva]

+1

[nickva]

Full CI tests passed https://ci-couchdb.apache.org/blue/organizations/jenkins/jenkins-cm1%2FFullPlatformMatrix/detail/jenkins-mango-begingswith-fixes-2/1/pipeline but see the minor test fix comment.