Skip to content

Commit

Permalink
mango: add $beginsWith operator (#4810)
Browse files Browse the repository at this point in the history 
Adds a `$beginsWith` operator to selectors, with json and text index
support. This is a compliment / precursor to optimising `$regex`
support as proposed in #4776.

For `json` indexes, a $beginsWith operator translates into a key
range query, as is common practice for _view queries. For example,
to find all rows with a key beginning with "W", we can use a range
`start_key="W", end_key="W\ufff0"`. Given Mango uses compound keys,
this is slightly more complex in practice, but the idea is the same.
As with other range operators (`$gt`, `$gte`, etc), `$beginsWith`
can be used in combination with equality operators and result sorting
but must result in a contiguous key range. That is, a range of
`start_key=[10, "W"], end_key=[10, "W\ufff0", {}]` would be valid,
but `start_key=["W", 10], end_key=["W\ufff0", 10, {}]` would not,
because the second element of the key may result in a non-contiguous
range.

For text indexes, `$beginsWith` translates to a Lucene query on
the specified field of `W*`.

If a non-string operand is provided to `$beginsWith`, the request will
fail with a 400 / `invalid_operator` error.
  • Loading branch information
@willholley
willholley committed Oct 30, 2023
1 parent 682f512 commit 26cfe53
Show file tree
Hide file tree
Showing 6 changed files with 312 additions and 91 deletions.
144 changes: 77 additions & 67 deletions src/docs/src/api/database/find.rst
View file
Original file line number Diff line number Diff line change
Expand Up @@ -200,8 +200,9 @@ A simple selector, inspecting specific fields:
You can create more complex selector expressions by combining operators.
For best performance, it is best to combine 'combination' or
'array logical' operators, such as ``$regex``, with an equality
operators such as ``$eq``, ``$gt``, ``$gte``, ``$lt``, and ``$lte``
'array logical' operators, such as ``$regex``, with an operator
that defines a contiguous range of keys such as ``$eq``,
``$gt``, ``$gte``, ``$lt``, ``$lte``, and ``$beginsWith``
(but not ``$ne``). For more information about creating complex
selector expressions, see :ref:`creating selector expressions
<find/expressions>`.
Expand Down Expand Up @@ -673,68 +674,74 @@ In addition, some 'meta' condition operators are available. Some condition
operators accept any valid JSON content as the argument. Other condition
operators require the argument to be in a specific JSON format.

+---------------+-------------+------------+-----------------------------------+
| Operator type | Operator | Argument | Purpose |
+===============+=============+============+===================================+
| (In)equality | ``$lt`` | Any JSON | The field is less than the |
| | | | argument. |
+---------------+-------------+------------+-----------------------------------+
| | ``$lte`` | Any JSON | The field is less than or equal to|
| | | | the argument. |
+---------------+-------------+------------+-----------------------------------+
| | ``$eq`` | Any JSON | The field is equal to the argument|
+---------------+-------------+------------+-----------------------------------+
| | ``$ne`` | Any JSON | The field is not equal to the |
| | | | argument. |
+---------------+-------------+------------+-----------------------------------+
| | ``$gte`` | Any JSON | The field is greater than or equal|
| | | | to the argument. |
+---------------+-------------+------------+-----------------------------------+
| | ``$gt`` | Any JSON | The field is greater than the |
| | | | to the argument. |
+---------------+-------------+------------+-----------------------------------+
| Object | ``$exists`` | Boolean | Check whether the field exists or |
| | | | not, regardless of its value. |
+---------------+-------------+------------+-----------------------------------+
| | ``$type`` | String | Check the document field's type. |
| | | | Valid values are ``"null"``, |
| | | | ``"boolean"``, ``"number"``, |
| | | | ``"string"``, ``"array"``, and |
| | | | ``"object"``. |
+---------------+-------------+------------+-----------------------------------+
| Array | ``$in`` | Array of | The document field must exist in |
| | | JSON values| the list provided. |
+---------------+-------------+------------+-----------------------------------+
| | ``$nin`` | Array of | The document field not must exist |
| | | JSON values| in the list provided. |
+---------------+-------------+------------+-----------------------------------+
| | ``$size`` | Integer | Special condition to match the |
| | | | length of an array field in a |
| | | | document. Non-array fields cannot |
| | | | match this condition. |
+---------------+-------------+------------+-----------------------------------+
| Miscellaneous | ``$mod`` | [Divisor, | Divisor is a non-zero integer, |
| | | Remainder] | Remainder is any integer. |
| | | | Non-integer values result in a |
| | | | 404. Matches documents where |
| | | | ``field % Divisor == Remainder`` |
| | | | is true, and only when the |
| | | | document field is an integer. |
+---------------+-------------+------------+-----------------------------------+
| | ``$regex`` | String | A regular expression pattern to |
| | | | match against the document field. |
| | | | Only matches when the field is a |
| | | | string value and matches the |
| | | | supplied regular expression. The |
| | | | matching algorithms are based on |
| | | | the Perl Compatible Regular |
| | | | Expression (PCRE) library. For |
| | | | more information about what is |
| | | | implemented, see the see the |
| | | | `Erlang Regular Expression |
| | | | <http://erlang.org/doc |
| | | | /man/re.html>`_. |
+---------------+-------------+------------+-----------------------------------+
+---------------+-----------------+-------------+------------------------------------+
| Operator type | Operator | Argument | Purpose |
+===============+=================+=============+====================================+
| (In)equality | ``$lt`` | Any JSON | The field is less than the |
| | | | argument. |
+---------------+-----------------+-------------+------------------------------------+
| | ``$lte`` | Any JSON | The field is less than or equal to |
| | | | the argument. |
+---------------+-----------------+-------------+------------------------------------+
| | ``$eq`` | Any JSON | The field is equal to the argument |
+---------------+-----------------+-------------+------------------------------------+
| | ``$ne`` | Any JSON | The field is not equal to the |
| | | | argument. |
+---------------+-----------------+-------------+------------------------------------+
| | ``$gte`` | Any JSON | The field is greater than or equal |
| | | | to the argument. |
+---------------+-----------------+-------------+------------------------------------+
| | ``$gt`` | Any JSON | The field is greater than the |
| | | | to the argument. |
+---------------+-----------------+-------------+------------------------------------+
| Object | ``$exists`` | Boolean | Check whether the field exists or |
| | | | not, regardless of its value. |
+---------------+-----------------+-------------+------------------------------------+
| | ``$type`` | String | Check the document field's type. |
| | | | Valid values are ``"null"``, |
| | | | ``"boolean"``, ``"number"``, |
| | | | ``"string"``, ``"array"``, and |
| | | | ``"object"``. |
+---------------+-----------------+-------------+------------------------------------+
| Array | ``$in`` | Array of | The document field must exist in |
| | | JSON values | the list provided. |
+---------------+-----------------+-------------+------------------------------------+
| | ``$nin`` | Array of | The document field not must exist |
| | | JSON values | in the list provided. |
+---------------+-----------------+-------------+------------------------------------+
| | ``$size`` | Integer | Special condition to match the |
| | | | length of an array field in a |
| | | | document. Non-array fields cannot |
| | | | match this condition. |
+---------------+-----------------+-------------+------------------------------------+
| Miscellaneous | ``$mod`` | [Divisor, | Divisor is a non-zero integer, |
| | | Remainder] | Remainder is any integer. |
| | | | Non-integer values result in a |
| | | | 404. Matches documents where |
| | | | ``field % Divisor == Remainder`` |
| | | | is true, and only when the |
| | | | document field is an integer. |
+---------------+-----------------+-------------+------------------------------------+
| | ``$regex`` | String | A regular expression pattern to |
| | | | match against the document field. |
| | | | Only matches when the field is a |
| | | | string value and matches the |
| | | | supplied regular expression. The |
| | | | matching algorithms are based on |
| | | | the Perl Compatible Regular |
| | | | Expression (PCRE) library. For |
| | | | more information about what is |
| | | | implemented, see the see the |
| | | | `Erlang Regular Expression |
| | | | <http://erlang.org/doc |
| | | | /man/re.html>`_. |
+---------------+-----------------+-------------+------------------------------------+
| | ``$beginsWith`` | String | Matches where the document field |
| | | | begins with the specified prefix |
| | | | (case-sensitive). If the document |
| | | | field contains a non-string value, |
| | | | the document is not matched. |
+---------------+-----------------+-------------+------------------------------------+

.. warning::
Regular expressions do not work with indexes, so they should not be used to
Expand All @@ -753,9 +760,12 @@ In general, whenever you have an operator that takes an argument, that argument
can itself be another operator with arguments of its own. This enables us to
build up more complex selector expressions.

However, only equality operators such as ``$eq``, ``$gt``, ``$gte``, ``$lt``,
and ``$lte`` (but not ``$ne``) can be used as the basis of a query. You should
include at least one of these in a selector.
However, only operators that define a contiguous range of values
such as ``$eq``, ``$gt``, ``$gte``, ``$lt``, ``$lte``,
and ``$beginsWith`` (but not ``$ne``) can be used as the basis
of a query that can make efficient use of a ``json`` index. You should
include at least one of these in a selector, or consider using
a ``text`` index if greater flexibility is required.

For example, if you try to perform a query that attempts to match all documents
that have a field called `afieldname` containing a value that begins with the
Expand Down
6 changes: 6 additions & 0 deletions src/mango/src/mango_idx_view.erl
View file
Original file line number Diff line number Diff line change
Expand Up @@ -306,6 +306,8 @@ indexable({[{<<"$gt">>, _}]}) ->
true;
indexable({[{<<"$gte">>, _}]}) ->
true;
indexable({[{<<"$beginsWith">>, _}]}) ->
true;
% This is required to improve index selection for covering indexes.
% Making `$exists` indexable should not cause problems in other cases.
indexable({[{<<"$exists">>, _}]}) ->
Expand Down Expand Up @@ -412,6 +414,10 @@ range(_, _, LCmp, Low, HCmp, High) ->
% operators but its all straight forward once you figure out how
% we're basically just narrowing our logical ranges.

% beginsWith requires both a high and low bound
range({[{<<"$beginsWith">>, Arg}]}, LCmp, Low, HCmp, High) ->
{LCmp0, Low0, HCmp0, High0} = range({[{<<"$gte">>, Arg}]}, LCmp, Low, HCmp, High),
range({[{<<"$lte">>, <<Arg/binary, 16#10FFFF>>}]}, LCmp0, Low0, HCmp0, High0);
range({[{<<"$lt">>, Arg}]}, LCmp, Low, HCmp, High) ->
case range_pos(Low, Arg, High) of
min ->
Expand Down
70 changes: 47 additions & 23 deletions src/mango/src/mango_selector.erl
View file
Original file line number Diff line number Diff line change
Expand Up @@ -135,6 +135,8 @@ norm_ops({[{<<"$text">>, Arg}]}) when
{[{<<"$default">>, {[{<<"$text">>, Arg}]}}]};
norm_ops({[{<<"$text">>, Arg}]}) ->
?MANGO_ERROR({bad_arg, '$text', Arg});
norm_ops({[{<<"$beginsWith">>, Arg}]} = Cond) when is_binary(Arg) ->
Cond;
% Not technically an operator but we pass it through here
% so that this function accepts its own output. This exists
% so that $text can have a field name value which simplifies
Expand Down Expand Up @@ -514,6 +516,11 @@ match({[{<<"$mod">>, [D, R]}]}, Value, _Cmp) when is_integer(Value) ->
Value rem D == R;
match({[{<<"$mod">>, _}]}, _Value, _Cmp) ->
false;
match({[{<<"$beginsWith">>, Prefix}]}, Value, _Cmp) when is_binary(Prefix), is_binary(Value) ->
string:prefix(Value, Prefix) /= nomatch;
% When Value is not a string, do not match
match({[{<<"$beginsWith">>, Prefix}]}, _, _Cmp) when is_binary(Prefix) ->
false;
match({[{<<"$regex">>, Regex}]}, Value, _Cmp) when is_binary(Value) ->
try
match == re:run(Value, Regex, [{capture, none}])
Expand Down Expand Up @@ -652,6 +659,14 @@ fields({[]}) ->
-ifdef(TEST).
-include_lib("eunit/include/eunit.hrl").

-define(TEST_DOC,
{[
{<<"_id">>, <<"foo">>},
{<<"_rev">>, <<"bar">>},
{<<"user_id">>, 11}
]}
).

is_constant_field_basic_test() ->
Selector = normalize({[{<<"A">>, <<"foo">>}]}),
Field = <<"A">>,
Expand Down Expand Up @@ -991,30 +1006,22 @@ has_required_fields_or_nested_or_false_test() ->
Normalized = normalize(Selector),
?assertEqual(false, has_required_fields(Normalized, RequiredFields)).

check_match(Selector) ->
% Call match_int/2 to avoid ERROR for missing metric; this is confusing
% in the middle of test output.
match_int(mango_selector:normalize(Selector), ?TEST_DOC).

%% This test shows the shape match/2 expects for its arguments.
match_demo_test_() ->
Doc =
{[
{<<"_id">>, <<"foo">>},
{<<"_rev">>, <<"bar">>},
{<<"user_id">>, 11}
]},
Check = fun(Selector) ->
% Call match_int/2 to avoid ERROR for missing metric; this is confusing
% in the middle of test output.
match_int(mango_selector:normalize(Selector), Doc)
end,
[
% matching
?_assertEqual(true, Check({[{<<"user_id">>, 11}]})),
?_assertEqual(true, Check({[{<<"_id">>, <<"foo">>}]})),
?_assertEqual(true, Check({[{<<"_id">>, <<"foo">>}, {<<"_rev">>, <<"bar">>}]})),
% non-matching
?_assertEqual(false, Check({[{<<"user_id">>, 1234}]})),
% string 11 doesn't match number 11
?_assertEqual(false, Check({[{<<"user_id">>, <<"11">>}]})),
?_assertEqual(false, Check({[{<<"_id">>, <<"foo">>}, {<<"_rev">>, <<"quux">>}]}))
].
match_demo_test() ->
% matching
?assertEqual(true, check_match({[{<<"user_id">>, 11}]})),
?assertEqual(true, check_match({[{<<"_id">>, <<"foo">>}]})),
?assertEqual(true, check_match({[{<<"_id">>, <<"foo">>}, {<<"_rev">>, <<"bar">>}]})),
% non-matching
?assertEqual(false, check_match({[{<<"user_id">>, 1234}]})),
% string 11 doesn't match number 11
?assertEqual(false, check_match({[{<<"user_id">>, <<"11">>}]})),
?assertEqual(false, check_match({[{<<"_id">>, <<"foo">>}, {<<"_rev">>, <<"quux">>}]})).

fields_of(Selector) ->
fields(test_util:as_selector(Selector)).
Expand Down Expand Up @@ -1054,4 +1061,21 @@ fields_nor_test() ->
},
?assertEqual([<<"field1">>, <<"field2">>], fields_of(Selector2)).

check_beginswith(Field, Prefix) ->
Selector = {[{Field, {[{<<"$beginsWith">>, Prefix}]}}]},
% Call match_int/2 to avoid ERROR for missing metric; this is confusing
% in the middle of test output.
match_int(mango_selector:normalize(Selector), ?TEST_DOC).

match_beginswith_test() ->
% matching
?assertEqual(true, check_beginswith(<<"_id">>, <<"f">>)),
% no match (user_id is not a binary string)
?assertEqual(false, check_beginswith(<<"user_id">>, <<"f">>)),
% invalid (prefix is not a binary string)
?assertThrow(
{mango_error, mango_selector, {invalid_operator, <<"$beginsWith">>}},
check_beginswith(<<"user_id">>, 1)
).

-endif.
11 changes: 11 additions & 0 deletions src/mango/src/mango_selector_text.erl
View file
Original file line number Diff line number Diff line change
Expand Up @@ -142,6 +142,11 @@ convert(Path, {[{<<"$exists">>, ShouldExist}]}) ->
true -> FieldExists;
false -> {op_not, {FieldExists, false}}
end;
convert(Path, {[{<<"$beginsWith">>, Arg}]}) when is_binary(Arg) ->
Prefix = mango_util:lucene_escape_query_value(Arg),
Suffix = <<"*">>,
PrefixSearch = <<Prefix/binary, Suffix/binary>>,
{op_field, {make_field(Path, Arg), PrefixSearch}};
% We're not checking the actual type here, just looking for
% anything that has a possibility of matching by checking
% for the field name. We use the same logic for $exists on
Expand Down Expand Up @@ -821,6 +826,12 @@ convert_nor_test() ->
})
).

convert_beginswith_test() ->
?assertEqual(
{op_field, {[[<<"field">>], <<":">>, <<"string">>], <<"foo*">>}},
convert_selector(#{<<"field">> => #{<<"$beginsWith">> => <<"foo">>}})
).

to_query_test() ->
F = fun(S) -> iolist_to_binary(to_query(S)) end,
Input = {<<"name">>, <<"value">>},
Expand Down

0 comments on commit 26cfe53

Please sign in to comment.