Print more details with JsonParseException

[melissachang]

I am using dynamic field mappings to index a BigQuery table. Some data is missing in the BigQuery table, which is common and supported (in BigQuery). I see this in my Elasticsearch logs:

elasticsearch_1    | [2018-07-03T21:58:56,122][WARN ][r.suppressed             ] path: /_bulk, params: {}
elasticsearch_1    | com.fasterxml.jackson.core.JsonParseException: Non-standard token 'NaN': enable JsonParser.Feature.ALLOW_NON_NUMERIC_NUMBERS to allow
elasticsearch_1    |  at [Source: org.elasticsearch.transport.netty4.ByteBufStreamInput@5c758f6f; line: 1, column: 1616]
elasticsearch_1    | 	at com.fasterxml.jackson.core.JsonParser._constructError(JsonParser.java:1702) ~[jackson-core-2.8.10.jar:2.8.10]
elasticsearch_1    | 	at com.fasterxml.jackson.core.base.ParserMinimalBase._reportError(ParserMinimalBase.java:558) ~[jackson-core-2.8.10.jar:2.8.10]
elasticsearch_1    | 	at com.fasterxml.jackson.core.json.UTF8StreamJsonParser._handleUnexpectedValue(UTF8StreamJsonParser.java:2667) ~[jackson-core-2.8.10.jar:2.8.10]
elasticsearch_1    | 	at com.fasterxml.jackson.core.json.UTF8StreamJsonParser.nextToken(UTF8StreamJsonParser.java:832) ~[jackson-core-2.8.10.jar:2.8.10]
elasticsearch_1    | 	at com.fasterxml.jackson.core.JsonGenerator.copyCurrentStructure(JsonGenerator.java:1813) ~[jackson-core-2.8.10.jar:2.8.10]
elasticsearch_1    | 	at org.elasticsearch.common.xcontent.json.JsonXContentGenerator.copyCurrentStructure(JsonXContentGenerator.java:407) ~[elasticsearch-6.2.2.jar:6.2.2]
elasticsearch_1    | 	at org.elasticsearch.common.xcontent.XContentBuilder.copyCurrentStructure(XContentBuilder.java:1021) ~[elasticsearch-6.2.2.jar:6.2.2]
elasticsearch_1    | 	at org.elasticsearch.action.update.UpdateRequest.fromXContent(UpdateRequest.java:741) ~[elasticsearch-6.2.2.jar:6.2.2]
elasticsearch_1    | 	at org.elasticsearch.action.bulk.BulkRequest.add(BulkRequest.java:433) ~[elasticsearch-6.2.2.jar:6.2.2]
elasticsearch_1    | 	at org.elasticsearch.rest.action.document.RestBulkAction.prepareRequest(RestBulkAction.java:94) ~[elasticsearch-6.2.2.jar:6.2.2]
elasticsearch_1    | 	at org.elasticsearch.rest.BaseRestHandler.handleRequest(BaseRestHandler.java:80) ~[elasticsearch-6.2.2.jar:6.2.2]
elasticsearch_1    | 	at org.elasticsearch.rest.RestController.dispatchRequest(RestController.java:240) [elasticsearch-6.2.2.jar:6.2.2]
elasticsearch_1    | 	at org.elasticsearch.rest.RestController.tryAllHandlers(RestController.java:336) [elasticsearch-6.2.2.jar:6.2.2]
elasticsearch_1    | 	at org.elasticsearch.rest.RestController.dispatchRequest(RestController.java:174) [elasticsearch-6.2.2.jar:6.2.2]
elasticsearch_1    | 	at org.elasticsearch.http.netty4.Netty4HttpServerTransport.dispatchRequest(Netty4HttpServerTransport.java:500) [transport-netty4-6.2.2.jar:6.2.2]
elasticsearch_1    | 	at org.elasticsearch.http.netty4.Netty4HttpRequestHandler.channelRead0(Netty4HttpRequestHandler.java:80) [transport-netty4-6.2.2.jar:6.2.2]

I see that the invalid token is NaN. It would be nice to know the relevant document id and field, so I know where to look in the BigQuery table.

(Maybe this exception is due to #12366)

[dadoonet]

Normally you could get that information back from the BulkResponse itself.

[danielmitterdorfer]

If we store the following in a file called bulk_request:

{ "index" : { "_index" : "test", "_type" : "_doc", "_id" : "1" } }
{ "field1" : 2 }
{ "index" : { "_index" : "test", "_type" : "_doc", "_id" : "2" } }
{ "field1" : NaN }

and send it via the bulk API:

curl -s -H "Content-Type: application/x-ndjson" -XPOST localhost:9200/_bulk\?pretty --data-binary "@bulk_request"

we get:

{
  "took" : 961,
  "errors" : true,
  "items" : [
    {
      "index" : {
        "_index" : "test",
        "_type" : "_doc",
        "_id" : "1",
        "_version" : 1,
        "result" : "created",
        "_shards" : {
          "total" : 2,
          "successful" : 1,
          "failed" : 0
        },
        "_seq_no" : 0,
        "_primary_term" : 1,
        "status" : 201
      }
    },
    {
      "index" : {
        "_index" : "test",
        "_type" : "_doc",
        "_id" : "2",
        "status" : 400,
        "error" : {
          "type" : "mapper_parsing_exception",
          "reason" : "failed to parse",
          "caused_by" : {
            "type" : "json_parse_exception",
            "reason" : "Non-standard token 'NaN': enable JsonParser.Feature.ALLOW_NON_NUMERIC_NUMBERS to allow\n at [Source: org.elasticsearch.common.bytes.BytesReference$MarkSupportingStreamInputWrapper@451469ff; line: 1, column: 17]"
          }
        }
      }
    }
  ]
}

So, in case the id is provided in the bulk request, it is contained in the response as well. I am not sure whether it is possible to also include the field name in the error message, hence marking it as "Discuss".

[elasticmachine]

Pinging @elastic/es-core-infra

[danielmitterdorfer]

We discussed it in FixIt Friday: parse exceptions can occur in different circumstances and thus it might not even possible to provide a field name. Also, the id as well as line and column are provided which point to the exact place where the parse exception has occurred.