_geo_distance sort: Support for many-to-many geo distance sort

[bobrik]

What we'd like to see:

Ability to specify many geo points in geo distance sort, like this:

{
    "sort": [
        {
            "_geo_distance": {
                "geo_points.point": [
                    {
                        "lat": 59.959946,
                        "lon": 30.313819
                    },
                    {
                        "lat": 59.979788,
                        "lon": 30.304513
                    }
                ]
            }
        }
    ]
}

Use-case: each user has several points of interest and wants to find other users who are close to these points. For example, I have work, home, and favourite breakfast cafe. I'd like to find people who hang out near any of those places. Right now I should fire as many queries, as many points of interest I have, and then I need to merge results outside of elasticsearch. Having ability to use single query for this.

Looks like it could be quite trivial change, an extra loop to iterate requested points. I might be wrong about this.

I understand, that it could be very cpu intensive for querying with many points, but the point here is to use small amount of points.

[bobrik]

Any feedback about this?

[brwe]

Have you considered using the function_score query?
You can configure it so that people returned by a query are sorted by their closest distance to any number of defined points. Is this what you want?

Here is how the query would roughly look like:

"query": {
      "function_score": {
         "query": {
            put here which kinds of people you look for and also maybe put distance filters to get rid of people that are too far anyway
         },
         "functions": [
            {
               "gauss": {
                   "name of geo location field for person": {
                        "origin": "geo point of favorite cafe 1",
                        "scale": ".."
                  }
               }
            },
            {
               "gauss": {
                  "name of geo location field for person": {
                      "origin": "geo point of favorite cafe 2",
                      "scale": "..."
                   }
               }
            },
            ...put even more places...
         ],
         .. use the score of nearest place...
         "score_mode": "max"
      }
   }

[bobrik]

@brwe I think It could work too, but _geo_distance would be easier to debug, because it tells you the distance. I also think _geo_distance should work faster in my case.

I tried function_score with query like this:

{
  "fields": [
    "geo_points"
  ],
  "size": 10000,
  "filter": {
    "and": [
      {
        "or": [
          {
            "geo_distance": {
              "geo_points.point": {
                "lat": 59.953478,
                "lon": 30.315557
              },
              "distance": "1.2km"
            }
          },
          {
            "geo_distance": {
              "geo_points.point": {
                "lat": 60.002,
                "lon": 30.298391
              },
              "distance": "1.2km"
            }
          }
        ]
      },
      {
        "terms": {
          "user_id": [
            51933602,
            63087823,
            45214178
          ]
        }
      }
    ]
  },
  "query": {
    "function_score": {
      "boost_mode": "replace",
      "score_mode": "max",
      "functions": [
        {
          "gauss": {
            "geo_points.point": {
              "origin": {
                "lat": 59.953478,
                "lon": 30.315557
              },
              "scale": "1.2km"
            }
          }
        },
        {
          "gauss": {
            "geo_points.point": {
              "origin": {
                "lat": 60.002,
                "lon": 30.298391
              },
              "scale": "1.2km"
            }
          }
        }
      ]
    }
  }
}

Notice that I filtered only 3 users, because their score for some reason is 0. Response looks like this:

{
  "total": 3,
  "max_score": 0,
  "hits": [
    {
      "_index": "female",
      "_type": "users",
      "_id": "45214178",
      "_score": 0,
      "fields": {
        "geo_points": [
          {
            "point": "59.957,30.303"
          },
          {
            "point": "0.004,0.004"
          },
          {
            "point": "59.948,30.276"
          },
          {
            "point": "59.944,30.272"
          }
        ]
      }
    },
    {
      "_index": "female",
      "_type": "users",
      "_id": "63087823",
      "_score": 0,
      "fields": {
        "geo_points": [
          {
            "point": "59.956,30.318"
          },
          {
            "point": "0,0"
          }
        ]
      }
    },
    {
      "_index": "female",
      "_type": "users",
      "_id": "51933602",
      "_score": 0,
      "fields": {
        "geo_points": [
          {
            "point": "59.956,30.314"
          },
          {
            "point": "0,-0.004"
          }
        ]
      }
    }
  ]
}

There are points far away from requested, but there are closer points in there same objects too. I thought that best match should be used. Am I missing something?

[brwe]

No, you did not miss something, I did. I completely misunderstood your request. function_score unfortunately does not work for fields with multiple values. Hope you did not waste too much time with it.

[imotov]

Perhaps you can try implementing it as script-based sort that will calculated distances to all points of interest and then return the shortest one.

[bobrik]

@brwe maybe function_sort should fire an exception if it called against field with multiple values?

@imotov yep, that should work, but scripting is much slower than native support. But I'll try.

[bobrik]

@imotov Looks like I cannot iterate array of objects in elasticsearch. I've found this thread: https://groups.google.com/forum/#!topic/elasticsearch/8Z2KwuPlyas

My mapping looks like this:

{
  "users" : {
    "properties" : {
      "geo_points" : {
        "properties" : {
          "point" : {
            "type" : "geo_point"
          }
        }
      }
    }
  }
}

and script looks like this:

current_to_any_geo_distance = 100;
if (!doc["geo_points"].empty && !current_geo.empty) {
    foreach (point : doc["geo_points"].values) {
        distance = point["point"].arcDistanceInKm(current_geo.lat, current_geo.lon);
        if (distance < current_to_any_geo_distance) {
            current_to_any_geo_distance = distance;
        }
    }
}

and elasticsearch complains like this:

{
  "took": 44,
  "timed_out": false,
  "_shards": {
    "total": 2,
    "successful": 1,
    "failed": 1,
    "failures": [
      {
        "index": "female",
        "shard": 3,
        "status": 500,
        "reason": "RemoteTransportException[[search04][inet[/192.168.1.91:9300]][search/phase/query]]; nested: QueryPhaseExecutionException[[female][3]: query[ConstantScore(cache(_type:users))],from[0],size[20],sort[<custom:\"_script\": org.elasticsearch.index.fielddata.fieldcomparator.DoubleScriptDataComparator$InnerSource@1dd5b162>!]: Query Failed [Failed to execute main query]]; nested: CompileException[[Error: No field found for [geo_points] in mapping with types [users]]\n[Near : {... order = OLDER_ORDER; ....}]\n             ^\n[Line: 1, Column: 1]]; nested: ElasticSearchIllegalArgumentException[No field found for [geo_points] in mapping with types [users]]; "
      }
    ]
  },
  "hits": {
    "total": 0,
    "max_score": null,
    "hits": []
  }
}

Is that kind of iteration really supported in elasticsearch? It looks like bug to me, but I couldn't find an existing issue.

cc @dadoonet

[imotov]

@bobrik - such iteration is going to work only on source. A doc field can only be obtain by the actual field name - geo_points.point in your case. doc["geo_points"] simply doesn't exist in the index. Here is an example that demonstrates the idea - https://github.com/imotov/elasticsearch-test-scripts/blob/master/multi_geo_search.sh.

[bobrik]

@imotov looks like I forgot to submit my solution yesterday :)

I looked at source code and found that I can actually iterate doc["geo_points.point"].values, but it will be an array of objects with lat and lon properties, not geo_points, therefore I cannot compute distance.

Looks like I can compute distance in pretty hardcore way:

import org.elasticsearch.common.geo.GeoDistance;
import org.elasticsearch.common.unit.DistanceUnit;

foreach (point : doc["geo_points.point"].values) {
    distance = GeoDistance.ARC.calculate(point.lat, point.lon, current_geo.lat, current_geo.lon, DistanceUnit.KILOMETERS);
}

I'm not sure if this is friendly. It's not in the docs for sure :)

[bobrik]

Nice, thanks!