Possible bug: Date equality

[countermeasure]

If looks to me as if date equality testing in queries might not be working properly.

If we assign

var dateValue = new Date(2015, 8, 14);

then we make a query on a collection

var query = myCollection.find({myDate: {'$eq': dateValue});

records in the collection for which myDate is equal to new Date(2015, 8, 14) don't appear to be returned.

'$gte' and '$lte' also appear to fail for equal dates, although they return all the greater than or less than dates.

If I'm missing something here or I'm mistaken, I apologise, but I think this is a bug.

[techfort]

it may be a bug alright. I'll check it asap, maybe there needs to be a custom compare logic for Date types.

[obeliskos]

Just to be clear about how javascript dates have 'issues', if you do the following :

var dt1 = new Date("9/15/2015");
var dt2 = new Date("9/15/2015");

if (dt1 === dt2) {
    console.log('equal');
}
else {
    console.log('not equal');
}

It will display 'not equal'.

Then there is the time component, if your loki documents have any time component and the times are different, they are 'not equal'.

Then, the issue of serialization... if you save a loki db with any documents with a date object, they will get converted to strings. Your query logic would have to change to handle this.

I have (personal) goal to keep loki internal documents with dates converted to unixms format and I undertake burden of sanitizing input and/or output to do fix up, but realize this may not be ideal for quick caching and querying.

I did find a bug where the javascript date with implicit equality { dt: myDate } interprets the myDate as an object and thinks its setting up a sub expression like { dt: { '$eq': myDate }}. Without a larger strategy in place, i'm not sure fixing that is appropriate yet.

Let me know whether you think full date support (across serialization) is preferred, or whether support in original instance only is acceptable. If it's left to me I would probably add separate date equality operators to resolve with loki ops such as $dteq. If it is a string it can parse and possibly strip time component, otherwise it can strip time component and use getTime() to test equality.

[countermeasure]

@obeliskos, I was wondering if this 'not equal' behaviour for identical date object might have been causing the problem.

I'd be happy if the time component was not stripped before doing the comparison as it is useful to be able to filter based on both date and time of day. In the cases where I want to filter by date alone and ignore time, I'm happy to just use existing operators and set myDate appropriately with a time component of midnight rather than require new operators such as $dteq.

That said, I can imagine that new date operators would also come in very handy, so I suppose it boils down to whether you want to include them. If they existed, I would certainly use them.

Would it be possible/feasible to have the existing operators recognise date objects and call getTime() on them when comparing? This strikes me as a good first step, although I'm a fairly inexperienced programmer, so I can imagine that this approach might have downsides that I don't understand.

I'm not using persistence, so I don't have a view on how this could relate to serialisation. Sorry!

[obeliskos]

If the logic is added to operators then we would need to make sure performance wasn't impacted significantly... if its a big enough global performance hit, we might want user to intelligently choose query path with date specific operator.

[countermeasure]

That sounds reasonable.

[obeliskos]

So I have looked into this and can probably come up with some sort of support for this within a few days or a week.

Performance is affected if comparison is done at the operator level. Our $eq, $gte, $lte operators, as well as our sorting comparison functions, and indexing are affected if the property being tested is an instance of Date.

What I will probably do is provide a parallel set of operators that are 'date safe' such as $dteq, $dtgte, $dtlte. Then, when doing a find, sort, or index I may attempt to autodetect whether dates are involved by looking at only the first element (in the set of documents being compared). If the first document has the relevant property as an instance of Date() I will convert any $eq, $gte, or $lte operators to use the date safe $dteq, $dtgte, $dtlte equivalents. If you need to support nulls for that date and I cannot depend on this 'first object test', you will have to explicity use the new operators.

I will still need to implement that and assess the performance impact but that is the approach I am considering at the moment.

For persons who use loki persistence, when reloading their dates are converted to string (UTC?) so if string comparisons are not valid they will need to reconstruct dates (or just use unixms format to begin with).

I'll probably look to committing initial fix over the weekend.

[countermeasure]

New date-safe operators sound perfectly reasonable if adding logic to the existing operators will result in an unacceptable performance hit.

For what it might (or might not!) be worth, I'd be inclined not to auto-detect dates based on the first document queried. If a field which should contain a date contains a null value sometimes, it sounds like this could result in inconsistent behaviour - sometimes handling dates correctly and other times not - based on the changing value of the field in the first document. A naive user (like me!) could find that confusing and spend a lot of time puzzling over what was going wrong.

[obeliskos]

This should be resolved now but I may not have all the edge cases tested out yet.

A new $dteq operator has been added for exact equality tests of javascript date properties.

For (other) users interested in serializing their database I would encourage them to use a safer format such as unix ms or string encoding to keep logic same for reading those properties across saves.

$lte and $gte should be date compatible now, so new operators should not be needed for them.

I will probably a few more tests over the next day or so and I think @techfort has said that he will build a new version release soon, at which point you can test with your own data.

[obeliskos]

I just updated our Loki Sandbox page so anyone can post code examples which run against the latest/live lokijs.js file. This may be easier or more immediate for some users (like myself) than plunker.

I made a gist showing how the $eq and $dteq operators behave on javascript Date objects which you can run in the sandbox here :
https://rawgit.com/techfort/LokiJS/master/examples/sandbox/LokiSandbox.htm#rawgist=https://gist.githubusercontent.com/obeliskos/ccd97739d4862cde06a0/raw/28adfd1a69f9187a594944b2feb7a329435e1008/$dteq%2520operator%2520test

[countermeasure]

@obeliskos, I like your solution of "if it's not greater than and not less than it must be equal".

It's working for me with no problems. Thanks!

[obeliskos]

Thanks, to summarize for future reference... since javascript has no 'strict' less than or greater than operators, we added the above logic to those 'loose compare' scenarios to handle $lt, $lte, $gt, and $gte operators. For strict equality we have $eq. Our new $dteq operator has loose logic as described above since date equality doesn't work unless javascript date objects are exact same object instance.