Null geopoint should not throw an exception

[MattFriedmanAfilias]

Refers to: #5390 and https://gist.github.com/hkorte/9936192

While upgrading to 1.2.1 we have run into this issue.

We use model objects that we serialize into JSON. One of the fields is a geopoint. Sometimes we don't have the lat/lon yet, so we were leaving the field null until we do have the data, then we update the document.

I can understand that a location that is missing lat or missing lon would be invalid; this warrants an exception. But null or empty is not invalid, nor is it reason to throw an exception.

Since, according to the Gist above there's is no longer a way to leave the geopoint null you are forcing me to write a custom JSON serializer to exclude the field when the geo data is not yet available. This is considerably more work/maintenance for myself and everyone else who encounters this. I don't see a good reason for this.

I suggest that Elasticsearch should handle null and empty gracefully by ignoring/skipping the field instead of throwing an exception. Null and empty aren't invalid values. They simply mean there's no value for this field at this time.

---

Update: Later on after writing this I realized I can use @JsonInclude(JsonInclude.Include.NON_NULL) to exclude null fields from my JSON. However, I still think my above points are valid.

Thanks, Matt

[colings86]

I have tested this on both master and 1.x branches and both exhibit the same behaviour:

• Trying to index a document with "location": null does not error correctly indexes the document.
• Trying to index a document with "location":{} errors with:

{
   "error": "MapperParsingException[failed to parse]; nested: ElasticsearchParseException[field [lat] missing]; ",
   "status": 400
}

I think this is ok behaviour as we gracefully deal with the null case and interpret it as 'there is no data yet' which I think is as a user expects. The empty object case feels to me to be an error case and should throw as error as it does now still I think its correctly interpreted as 'there is some data to index but it is malformed'.

@clintongormley thoughts?

[clintongormley]

@colings86 i agree

[clintongormley]

It looks like there was a bug in null handling in geopoints in 1.1.0 which has subsequently been fixed, so I'm going to close this issue.

[goooooouwa]

Hi,
I'm just wandering how ES will treat document with "location": null when sorting by distance?
It returns a giant number as sort value in my case( I'm using ES 1.3.2). I'm wandering if this is desired.

// example of sort by distance on document with "location": null
// search request body

{
  "query": {
    "match_all": {}
  },
    "sort" : [
    {
      "_geo_distance" : {
        "location" : "-70, 40",
        "order" : "asc",
        "unit" : "km",
        "mode" : "min",
        "distance_type" : "sloppy_arc"
      }
    }
  ]
}

// response
{
  "_index": ...,
    "_type": ...,
    "_id": "1",
    "_score": null,
    "_source": {
      "id": 1,
      "name": ...,
    },  
    "sort": [
      1.7976931348623157e+308
      ]   
}

[clintongormley]

Hi @goooooouwa - you can control whether a missing value is sorted first or last: http://www.elasticsearch.org/guide/en/elasticsearch/reference/current/search-request-sort.html#_missing_values

[goooooouwa]

Hi @clintongormley , thanks for reply.
I'm not sure if I use "missing value" correctly, but when I search like this:

// example of sort by distance on document with "location": null and setting "missing" as "_last"
// search request body

{
  "query": {
    "match_all": {}
  },
    "sort" : [
    {
      "_geo_distance" : {
        "missing": "_last",
        "location" : "-70, 40",
        "order" : "asc",
        "unit" : "km",
        "mode" : "min",
        "distance_type" : "sloppy_arc"
      }
    }
  ]
}

I get the following error

{
   "error": "SearchPhaseExecutionException[Failed to execute phase [query], all shards failed; shardFailures {[fpzi7BmbQGC1HZBaNQK8iQ][constr_base-companies][3]: SearchParseException[[constr_base-companies][3]: query[ConstantScore(*:*)],from[-1],size[-1]: Parse Failure [Failed to parse source [{\n  \"query\": {\n    \"match_all\": {}\n  },\n    \"sort\" : [\n    {\n      \"_geo_distance\" : {\n          \"missing\": \"_last\",\n        \"lat_lon\" : \"-40,70\",\n        \"order\" : \"asc\",\n        \"unit\" : \"km\",\n        \"mode\" : \"min\",\n        \"distance_type\" : \"sloppy_arc\"\n      }\n    }\n  ]\n}\n]]]; nested: ElasticsearchIllegalArgumentException[the character '_' is not a valid geohash character]; }{[fpzi7BmbQGC1HZBaNQK8iQ][constr_base-companies][4]: SearchParseException[[constr_base-companies][4]: query[ConstantScore(*:*)],from[-1],size[-1]: Parse Failure [Failed to parse source [{\n  \"query\": {\n    \"match_all\": {}\n  },\n    \"sort\" : [\n    {\n      \"_geo_distance\" : {\n          \"missing\": \"_last\",\n        \"lat_lon\" : \"-40,70\",\n        \"order\" : \"asc\",\n        \"unit\" : \"km\",\n        \"mode\" : \"min\",\n        \"distance_type\" : \"sloppy_arc\"\n      }\n    }\n  ]\n}\n]]]; nested: ElasticsearchIllegalArgumentException[the character '_' is not a valid geohash character]; }{[fpzi7BmbQGC1HZBaNQK8iQ][constr_base-companies][1]: SearchParseException[[constr_base-companies][1]: query[ConstantScore(*:*)],from[-1],size[-1]: Parse Failure [Failed to parse source [{\n  \"query\": {\n    \"match_all\": {}\n  },\n    \"sort\" : [\n    {\n      \"_geo_distance\" : {\n          \"missing\": \"_last\",\n        \"lat_lon\" : \"-40,70\",\n        \"order\" : \"asc\",\n        \"unit\" : \"km\",\n        \"mode\" : \"min\",\n        \"distance_type\" : \"sloppy_arc\"\n      }\n    }\n  ]\n}\n]]]; nested: ElasticsearchIllegalArgumentException[the character '_' is not a valid geohash character]; }{[fpzi7BmbQGC1HZBaNQK8iQ][constr_base-companies][2]: SearchParseException[[constr_base-companies][2]: query[ConstantScore(*:*)],from[-1],size[-1]: Parse Failure [Failed to parse source [{\n  \"query\": {\n    \"match_all\": {}\n  },\n    \"sort\" : [\n    {\n      \"_geo_distance\" : {\n          \"missing\": \"_last\",\n        \"lat_lon\" : \"-40,70\",\n        \"order\" : \"asc\",\n        \"unit\" : \"km\",\n        \"mode\" : \"min\",\n        \"distance_type\" : \"sloppy_arc\"\n      }\n    }\n  ]\n}\n]]]; nested: ElasticsearchIllegalArgumentException[the character '_' is not a valid geohash character]; }{[fpzi7BmbQGC1HZBaNQK8iQ][constr_base-companies][0]: SearchParseException[[constr_base-companies][0]: query[ConstantScore(*:*)],from[-1],size[-1]: Parse Failure [Failed to parse source [{\n  \"query\": {\n    \"match_all\": {}\n  },\n    \"sort\" : [\n    {\n      \"_geo_distance\" : {\n          \"missing\": \"_last\",\n        \"lat_lon\" : \"-40,70\",\n        \"order\" : \"asc\",\n        \"unit\" : \"km\",\n        \"mode\" : \"min\",\n        \"distance_type\" : \"sloppy_arc\"\n      }\n    }\n  ]\n}\n]]]; nested: ElasticsearchIllegalArgumentException[the character '_' is not a valid geohash character]; }]",
   "status": 400
}

basically it says: ElasticsearchIllegalArgumentException[the character '_' is not a valid geohash character]

any idea?( I'm using ES 1.3.2)

Thanks

[clintongormley]

@goooooouwa looks like it doesn't support missing. Please open a new issue for that