BadValue Too many geoNear expressions in 2.3.8 #3767

gs-sramesh · 2017-04-30T09:14:21Z

Issue Description

All geo queries near, withinMiles etc fail with the error:

 Uncaught internal server error. { MongoError: Can't canonicalize query: BadValue Too many geoNear expressions

The same used to work previously under parse version 2.3.0

Steps to reproduce

var query = new Parse.Query("User"); // Or any other clss
var point = new Parse.GeoPoint({
      latitude: location.lat || 0,
      longitude: location.lon || 0
    });
query.withinMiles("location", point, 75);

Expected Results

Should work as it did under version 2.3.0

Actual Outcome

Error on Mongo

Environment Setup

Server
- parse-server version (Be specific! Don't say 'latest'.) : 2.3.8
- Operating System: OSX Sierra
- Hardware:
- Localhost or remote server? (AWS, Heroku, Azure, Digital Ocean, etc): localhost
Database
- MongoDB version: MongoDB shell version: 3.0.12
- Storage engine: [FILL THIS OUT]
- Hardware: [FILL THIS OUT]
- Localhost or remote server? (AWS, mLab, ObjectRocket, Digital Ocean, etc): [FILL THIS OUT]

Logs/Trace

Include all relevant logs. You can turn on additional logging by configuring VERBOSE=1 in your environment.

error: Uncaught internal server error. { MongoError: Can't canonicalize query: BadValue Too many geoNear expressions
    at Function.MongoError.create (/Users/sramesh/Desktop/code/m2e/cloud/parse-backend/parse/node_modules/mongodb-core/lib/error.js:31:11)
    at queryCallback (/Users/sramesh/Desktop/code/m2e/cloud/parse-backend/parse/node_modules/mongodb-core/lib/cursor.js:197:34)
    at /Users/sramesh/Desktop/code/m2e/cloud/parse-backend/parse/node_modules/mongodb-core/lib/connection/pool.js:461:18
    at _combinedTickCallback (internal/process/next_tick.js:67:7)
    at process._tickDomainCallback (internal/process/next_tick.js:122:9)
  name: 'MongoError',
  message: 'Can\'t canonicalize query: BadValue Too many geoNear expressions',
  '$err': 'Can\'t canonicalize query: BadValue Too many geoNear expressions',
  code: 17287 } MongoError: Can't canonicalize query: BadValue Too many geoNear expressions
    at Function.MongoError.create (/Users/sramesh/Desktop/code/m2e/cloud/parse-backend/parse/node_modules/mongodb-core/lib/error.js:31:11)
    at queryCallback (/Users/sramesh/Desktop/code/m2e/cloud/parse-backend/parse/node_modules/mongodb-core/lib/cursor.js:197:34)
    at /Users/sramesh/Desktop/code/m2e/cloud/parse-backend/parse/node_modules/mongodb-core/lib/connection/pool.js:461:18
    at _combinedTickCallback (internal/process/next_tick.js:67:7)
    at process._tickDomainCallback (internal/process/next_tick.js:122:9)
MongoError: Can't canonicalize query: BadValue Too many geoNear expressions
    at Function.MongoError.create (/Users/sramesh/Desktop/code/m2e/cloud/parse-backend/parse/node_modules/mongodb-core/lib/error.js:31:11)
    at queryCallback (/Users/sramesh/Desktop/code/m2e/cloud/parse-backend/parse/node_modules/mongodb-core/lib/cursor.js:197:34)
    at /Users/sramesh/Desktop/code/m2e/cloud/parse-backend/parse/node_modules/mongodb-core/lib/connection/pool.js:461:18
    at _combinedTickCallback (internal/process/next_tick.js:67:7)
    at process._tickDomainCallback (internal/process/next_tick.js:122:9)

The text was updated successfully, but these errors were encountered:

srameshr · 2017-05-01T07:28:16Z

@flovilmart Anything on this?

flovilmart · 2017-05-01T10:49:33Z

Can your provide the logs when running with VERBOSE=1? This will log the full query. We have many tests covering the geoPoints. I'm not sure why your particular query fails

srameshr · 2017-05-01T14:22:57Z

verbose: REQUEST for [GET] /parse/classes/MyClass: {
  "where": {
    "$or": [
      {
        "MyClassDate": {
          "$gte": "2017-05-01T12:39:38.500Z"
        }
      },
      {
        "MyClassDate": null
      }
    ],
    "rLoc": {
      "$nearSphere": {
        "__type": "GeoPoint",
        "latitude": 40,
        "longitude": -30
      }
    }
  },
  "include": "createdBy",
  "keys": "MyClassDate,rCity",
  "limit": 10
} method=GET, url=/parse/classes/MyClass, user-agent=node-XMLHttpRequest, Parse/js1.9.2 (NodeJS 6.4.0), accept=*/*, content-type=text/plain, host=localhost:1337, content-length=471, connection=close, $or=[$gte=2017-05-01T12:39:38.500Z, MyClassDate=null], __type=GeoPoint, latitude=40, longitude=-30, include=createdBy, keys=MyClassDate,rCity, limit=10
error: Error generating response. { MongoError: Can't canonicalize query: BadValue Too many geoNear expressions
    at Function.MongoError.create (/Users/sramesh/Desktop/code/m2e/cloud/parse-backend/parse/node_modules/mongodb-core/lib/error.js:31:11)
    at queryCallback (/Users/sramesh/Desktop/code/m2e/cloud/parse-backend/parse/node_modules/mongodb-core/lib/cursor.js:197:34)
    at /Users/sramesh/Desktop/code/m2e/cloud/parse-backend/parse/node_modules/mongodb-core/lib/connection/pool.js:461:18
    at _combinedTickCallback (internal/process/next_tick.js:67:7)
    at process._tickDomainCallback (internal/process/next_tick.js:122:9)
  name: 'MongoError',
  message: 'Can\'t canonicalize query: BadValue Too many geoNear expressions',
  '$err': 'Can\'t canonicalize query: BadValue Too many geoNear expressions',
  code: 17287 } name=MongoError, message=Can't canonicalize query: BadValue Too many geoNear expressions, $err=Can't canonicalize query: BadValue Too many geoNear expressions, code=17287

@flovilmart

srameshr · 2017-05-04T07:16:40Z

@flovilmart Anything on this?

flovilmart · 2017-05-04T11:20:58Z

I don't believe you can use geonear alongside $or as part of a restriction on mongodb

srameshr · 2017-05-04T11:58:09Z

@flovilmart It used to work all right until i updated my parse server version to 2.3.8

flovilmart · 2017-05-04T12:21:08Z

Can you provide the exact client code that trigger this issue?

srameshr · 2017-05-04T12:36:15Z

Parse.Cloud.run('searchWithLocation', {
            'location': location, // lat and lon
            'date':date // JS date object
          })

@flovilmart

srameshr · 2017-05-05T13:24:53Z

Anyone who updated to 2.3.8 is facing the same?

flovilmart · 2017-05-05T13:53:56Z

And what's the query that's being done in cloud code?

srameshr · 2017-05-05T14:38:41Z

var location = request.params.location,
    date = request.params.date;

    var myClass = Parse.Object.extend("Class"),
    q1 = new Parse.Query(myClass);

    q1.greaterThanOrEqualTo("dated", date); // 6 hours before

    var q2 = new Parse.Query(Class1);
    q2.equalTo('dated', null);


    var query = Parse.Query.or(q1, q2),
    point = new Parse.GeoPoint({latitude: 40.0, longitude: -30.0});
    query.near("rLoc", point);
    query.include("createdBy");
    query.find().then(success, failure);

flovilmart · 2017-05-05T14:59:57Z

The point variable is unused, can you provide the exact code please so I can write a proper test and check against previous versions?

srameshr · 2017-05-05T15:53:31Z

@flovilmart I have edited the previous comment.

flovilmart · 2017-05-06T15:30:36Z

I'll run the tests against 2.3.7 to try to indentify where / if a regression occured

srameshr · 2017-05-08T05:01:52Z

@flovilmart Any updates on this?

flovilmart · 2017-05-08T11:14:31Z

I haven't had a chance to look at it during the week end

flovilmart · 2017-05-10T01:02:37Z

@srameshr I know where it's coming from, actually, it comes from that commit 7319562, from @NotBobTheBuilder, which is detailed enough so you have more information why we don't support it anymore. Yes, it breaks your particular query, but that's for the best I believe.

NotBobTheBuilder · 2017-05-10T01:16:17Z

As a follow up - consider using $geoWithin and $centerSphere instead of $nearSphere. (Bear in mind that results will not be sorted by distance, though)

flovilmart · 2017-05-10T01:50:18Z

thanks @NotBobTheBuilder for the followup!

srameshr · 2017-05-10T07:27:23Z

@flovilmart @NotBobTheBuilder Any workaround for "OR" involving null values?

NotBobTheBuilder · 2017-05-10T07:29:58Z

Could you give an example of an error that's raised from a query with a null value? In the example above, the null shouldn't cause problems

srameshr · 2017-05-10T07:34:01Z

@NotBobTheBuilder Im thinking of using exists.
I have to get all the objects in a collection where the date column can contain an actual date object or a null object. Until now I was doing this

    MainClass = Parse.Object.extend("MainClass"),
    hasDateQuery = new Parse.Query(MainClass);

    hasDateQuery.greaterThanOrEqualTo("date", new Date());

    var noDateQuery = new Parse.Query(MainClass);
    noDateQuery.equalTo('date', null);

    var query = Parse.Query.or(hasDateQuery, noDateQuery),
    point = new Parse.GeoPoint({latitude: 40.0, longitude: -30.0});
    query.withinMiles(point, 100);
    query.find()

This is the error:

Input: {"location":{"__type":"GeoPoint","latitude":12.9096069,"longitude":77.6404125}}
  Result: "Updated" functionName=lastKnownLocation, __type=GeoPoint, latitude=12.9096069, longitude=77.6404125, user=k6C6Nmo0ag
error: Error generating response. { MongoError: Can't canonicalize query: BadValue Too many geoNear expressions
    at Function.MongoError.create (node_modules/mongodb-core/lib/error.js:31:11)
    at queryCallback (node_modules/mongodb-core/lib/cursor.js:197:34)
    at node_modules/mongodb-core/lib/connection/pool.js:461:18
    at _combinedTickCallback (internal/process/next_tick.js:67:7)
    at process._tickDomainCallback (internal/process/next_tick.js:122:9)
  name: 'MongoError',
  message: 'Can\'t canonicalize query: BadValue Too many geoNear expressions',
  '$err': 'Can\'t canonicalize query: BadValue Too many geoNear expressions',
  code: 17287 } name=MongoError, message=Can't canonicalize query: BadValue Too many geoNear expressions, $err=Can't canonicalize query: BadValue Too many geoNear expressions, code=17287

NotBobTheBuilder · 2017-05-10T07:59:27Z

It looks like the error is occuring because withinMiles uses $nearSphere under the hood, not $geoWithin. (location in code). The problem isn't the null in this case, so using $exists instead of null might not help.

I'm not familiar enough with Parse to know if there's a way to use $geoWithin and not $nearSphere, but if there is, you should use that.

Otherwise, the options I see:

(1). Using geoWithin could be an improvement to be made in the client SDK
(2). We could add a special case to avoid moving the $nearSphere or $near operators
(3). You could try using logic like {$not: {$lt: date}}, since null values will be returned in that query.

It's hacky but (3) is probably your best short term fix. I'm not sure how you express $not in Parse, I thought initially it would be doesNotMatchQuery but I'm not certain.

Longer term, I will look at implementing (2) given others are likely to encounter this problem too, though I think (1) would be the ideal solution.

Mongo has a hard limit on 1 $near operation per query. Restructuring to avoid SERVER-13732 should not invalidate a query by creating multiple $near operations. Additionally, queries with multiple $ors are now recursively handled, whereas before, ops at the top level would only have been pushed one level deeper. parse-community#3767

NotBobTheBuilder · 2017-05-10T08:43:06Z

I filed a PR for 2. @srameshr if you have a chance to try it, could you try out the changes in #3798 and confirm that they fix this problem for you?

NotBobTheBuilder · 2017-05-10T08:43:21Z

(Lets move further discussion to that PR.)

srameshr · 2017-05-10T08:44:50Z

@NotBobTheBuilder I will do that and let you know ASAP.

srameshr · 2017-05-10T12:22:38Z

@NotBobTheBuilder That commit fixed it! Thanks! Hope its merged soon and available on 2.3.9

Mongo has a hard limit on 1 $near operation per query. Restructuring to avoid SERVER-13732 should not invalidate a query by creating multiple $near operations. Additionally, queries with multiple $ors are now recursively handled, whereas before, ops at the top level would only have been pushed one level deeper. #3767

flovilmart self-assigned this May 6, 2017

flovilmart closed this as completed May 10, 2017

flovilmart removed their assignment May 10, 2017

flovilmart added the won't fix label May 10, 2017

NotBobTheBuilder mentioned this issue May 10, 2017

Avoid multiple $nears in one query #3798

Merged

TomWFox added wontfix and removed won't fix labels Mar 29, 2019

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

BadValue Too many geoNear expressions in 2.3.8 #3767

BadValue Too many geoNear expressions in 2.3.8 #3767

gs-sramesh commented Apr 30, 2017 •

edited

Loading

srameshr commented May 1, 2017

flovilmart commented May 1, 2017

srameshr commented May 1, 2017

srameshr commented May 4, 2017

flovilmart commented May 4, 2017

srameshr commented May 4, 2017

flovilmart commented May 4, 2017

srameshr commented May 4, 2017

srameshr commented May 5, 2017

flovilmart commented May 5, 2017

srameshr commented May 5, 2017 •

edited

Loading

flovilmart commented May 5, 2017

srameshr commented May 5, 2017

flovilmart commented May 6, 2017

srameshr commented May 8, 2017

flovilmart commented May 8, 2017

flovilmart commented May 10, 2017

NotBobTheBuilder commented May 10, 2017

flovilmart commented May 10, 2017

srameshr commented May 10, 2017

NotBobTheBuilder commented May 10, 2017

srameshr commented May 10, 2017 •

edited

Loading

NotBobTheBuilder commented May 10, 2017

NotBobTheBuilder commented May 10, 2017

NotBobTheBuilder commented May 10, 2017

srameshr commented May 10, 2017

srameshr commented May 10, 2017

BadValue Too many geoNear expressions in 2.3.8 #3767

BadValue Too many geoNear expressions in 2.3.8 #3767

Comments

gs-sramesh commented Apr 30, 2017 • edited Loading

Issue Description

Steps to reproduce

Expected Results

Actual Outcome

Environment Setup

Logs/Trace

srameshr commented May 1, 2017

flovilmart commented May 1, 2017

srameshr commented May 1, 2017

srameshr commented May 4, 2017

flovilmart commented May 4, 2017

srameshr commented May 4, 2017

flovilmart commented May 4, 2017

srameshr commented May 4, 2017

srameshr commented May 5, 2017

flovilmart commented May 5, 2017

srameshr commented May 5, 2017 • edited Loading

flovilmart commented May 5, 2017

srameshr commented May 5, 2017

flovilmart commented May 6, 2017

srameshr commented May 8, 2017

flovilmart commented May 8, 2017

flovilmart commented May 10, 2017

NotBobTheBuilder commented May 10, 2017

flovilmart commented May 10, 2017

srameshr commented May 10, 2017

NotBobTheBuilder commented May 10, 2017

srameshr commented May 10, 2017 • edited Loading

NotBobTheBuilder commented May 10, 2017

NotBobTheBuilder commented May 10, 2017

NotBobTheBuilder commented May 10, 2017

srameshr commented May 10, 2017

srameshr commented May 10, 2017

gs-sramesh commented Apr 30, 2017 •

edited

Loading

srameshr commented May 5, 2017 •

edited

Loading

srameshr commented May 10, 2017 •

edited

Loading