Array support

[bottleofwater]

Hi,

It seems like the $eq and $neq operators currently don't work with arrays. Could this feature be added (maybe in another operators, if needs be)? I feel like it's quite important to be able to correctly select a set of item based on the exact content of an array. Worth nothing that even if $eq can be somewhat emulated ({ field: { $contains: myValues }, fieldLength: { $eq: myValues.length } }), $neq cannot.

On a similar topic, it seems like arrays cannot be easily selected based on their length. For example, the following won't work: { "field.length": { $eq: 3 } } (I guess it's because of #91). In order to be able to do that, one has to manually embed the array length inside the document (hence fieldLength in my first example, instead of field.length).

[VladimirTechMan]

@bottleofwater, while we do not have explicit deep equality operator(s) now in queries, you can, actually, easily add those to your applications, if you need them (I mean, without modifying the internals of LokiJS). The thing here is, that we still lack on properly documenting some of those newer capabilities of the recent releases of LokiJS... Well, at least, let me now introduce you to those new options relevant to your question. Note that all the details below are applicable to the current public version (1.4.1, as I am writing this) – please, check that you use the most recent one (unless you are directly grabbing the code from the top of our master branch at GitHub).

(1) The array length checks are actually supported already. That is done by the new "$size" operator. It can be used on arrays (only on arrays, at the moment) in one of two ways. If you want to query all the documents where a specific array-type field has a specific fixed length, then you can query it like this:

({ array_field: { '$size': length_number } })

Now, if you need to query with other equality operators, then you just put the whole comparison operator query object as the value of the "$size" operator, like in this example:

({ array_field: { '$size': { '$lt': 5 } } })

Here we will select all the documents where the named field contains an array with a length less than 5.

(2) Back to the deep equality comparison testing. We now also have a new universal "$where" operator inside the find queries, which allows you to write small auxiliary test functions and run them as a part of your Loki query. The value of the "$where" operator is a function that takes the value of the specified document field as its only argument and then it returns an explicit true result to indicate that the value passed to it complies with the internal testing logic of the function (otherwise the test is considered as failed). That way, if you have [just for the sake of giving a short example here] a ready deep-equality comparison function that acts on arrays (let's call it areDeepEqualArrays) and you have a fixed sample array to compare against (let's name it sampleArray), then you can put the following as a part of your query in the recent public versions of LokiJS:

({ array_field: { '$where': function (array) {
    return true === areDeepEqualArrays(array, sampleArray);
}}})

I hope that helps. Let me know if you have any questions in relation to the above examples and the usage of the highlighted new query operators.

[bottleofwater]

I see, thanks, I weren't aware of $size and it perfectly suits my needs. However as far as the array comparison goes, unless I'm mistaken, $where probably make use of .where(), right? In such a case, it will probably have an impact over the performances, whereas native comparisons would be able to compute an hash from the arrays in order to quickly find matching data.

[VladimirTechMan]

@bottleofwater, I am glad that the $size operator met your needs. Regarding your concern about the usage of$where`, it does not make things worse in terms of the performance for the case that you asked me about.

More precisely: The implementation of the $where operator does not call the where method of Resultset inside of it. It offers a similar functionality, but the implementations are separate. (Note that, while the where method of results-sets applies the custom test function to the whole document being tested, the $where operator calls the custom function passed as its value against the specified field's value.)

In terms of the performance and optimizations: When using Loki. If you can build a query where the first test operator runs against an indexed field and the operator is one of these: '$eq', '$aeq', '$dteq', '$gt', '$gte', '$lt', or '$lte'. Then Loki will be able to optimize the query thanks to using indexes. That query can still contain the $where operator in it, as one of the subsequent operators, and the initial optimization that I noted will still be applied.

Also, if your query is simple and contains only the $where operator with your test function inside it. Then the performance will be just about the same as in the case of writing an equivalent test function suitable for the where method of Resultset. And that performance of the former use case would not improve, really, if we would re-implement your desired test function as one of embedded operators of Loki – Loki cannot use indexes in those cases now, and will end up just iterating through all the documents to test them. Same way as it does with the $where operator and the custom function that you pass to it.

I hope that will clarify things better. Let me know if some questions or uncertainties still remain on your side.

[bottleofwater]

Thanks a lot, your explanation actually helped me a lot!

I'm not familiar with how Loki works underground, but isn't it hashing values to quickly retrieve them from an hashmap instead of iterating over each element? In such a case, wouldn't a $in operator be able to perform much better than a $where, since the library would be able to compute an array hash (by concatenating the hashes of each value it contains)?

[stale]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.