Index fails with MapperParsingException when dynamic_templates provided

[jasonpolites]

I'm not sure if the title of this is correct, because I'm not sure what the cause of this is, but here's what I'm seeing:

The following sequence of commands will fail:

curl -XDELETE "http://localhost:9200/test/"

curl -XPUT "http://localhost:9200/test/"

curl -XPUT http://localhost:9200/test/test/_mapping -d '
{
  "test": {
    "dynamic_templates": [
      {
        "all_fields": {
          "match": "*",
          "mapping": {
              "type": "string"
          }
        }
      }
    ],  
    "properties": {
      "meta": {
        "type": "object",
        "properties": {
          "timestamp": {
            "store": "yes",
            "type": "date"
          },
          "foo": {
            "type": "object",
            "properties": {
              "bar": {
                  "index": "not_analyzed",
                  "store": "yes",
                  "type": "string"
              }
            }
          }
        }
      }
    }
  }
}'


curl -XPUT http://localhost:9200/test/test/1 -d '
{
  "meta": [{
    "timestamp": 1234567890,
    "foo": {
      "bar" : "Hello World"
    },
    "fruits" : {
      "canned": {
        "orchard" : ["Apples", "Oranges", "Peaches"]
      }
    }
  }]
}'

The failure is

{
    "error": "MapperParsingException[Failed to parse [meta.fruits]]; nested: ElasticSearchIllegalArgumentException[unknown property [canned]]; ",
    "status": 400
}

It's not clear from the documentation why this doesn't work, or how one would assign a default type to unexpected field values if this mapping approach is wrong.

The issue seems to have been introduced from this commit:

6243f8e

[jasonpolites]

FYI PUTting the document shown above with the "meta" element as a single object (not an array) works

curl -XPUT http://localhost:9200/test/test/1 -d '
{
  "meta": {
    "timestamp": 1234567890,
    "foo": {
      "bar" : "Hello World"
    },
    "fruits" : {
      "canned": {
        "orchard" : ["Apples", "Oranges", "Peaches"]
      }
    }
  }
}'

{
    "ok": true,
    "_index": "test",
    "_type": "test",
    "_id": "1",
    "_version": 1
}

[jasonpolites]

Strike that last comment.. the non-array variant works in 0.19.12 but still fails in 0.20.xx.

The original case fails in 0.19.12 also, but with a different error:

{
    "error": "MapperParsingException[object mapping for [test] with array for [meta] tried to parse as array, but got EOF, is there a mismatch in types for the same field?]",
    "status": 400
}

[kimchy]

The reason this happens is that the fruits element ends up being forced to be a string type because of the dynamic template mapping that matches on *.... .

Btw, no need to explicitly store fields, since the _source document is always stored.

[jasonpolites]

OK thanks. The "store" option is there because I copy/pasted from the "real" mapping which has source storage disabled.

I won't pollute the issue tracker with "forum" style questions, but perhaps you could advise as to how I create a mapping that defaults any unknown value to string. I run into problems with auto-inferring types because my documents are not consistent (outside my control).

I have created a forum thread here:

https://groups.google.com/forum/?fromgroups=#!searchin/elasticsearch/Jason$20MapperParsingException/elasticsearch/kxBC3xZfeh0/-pirsB1eVjQJ

Thanks!

[kimchy]

you can set in the root object mapping date_detection to false, and numeric_detection set to false, and it won't try and auto detect dates/numerics. That should work out I think.

[jasonpolites]

oh awesome.. thanks

[jasonpolites]

No dice :( the following fails:

curl -XDELETE "http://localhost:9200/test/"

curl -XPUT "http://localhost:9200/test/"

curl -XPUT http://localhost:9200/test/test/_mapping -d '
{
  "test": {
    "date_detection" : false,
    "numeric_detection": false,
    "properties": {
      "meta": {
        "type": "object",
        "properties": {
          "timestamp": {
            "store": "yes",
            "type": "date"
          },
          "foo": {
            "type": "object",
            "properties": {
              "bar": {
                  "index": "not_analyzed",
                  "type": "string"
              }
            }
          }
        }
      }
    }
  }
}'

curl -XPUT http://localhost:9200/test/test/1 -d '
{
  "meta": [{
    "timestamp": 1234567890,
    "test": 123,
    "foo": {
      "bar" : "Hello World"
    },
    "fruits" : {
      "canned": {
        "orchard" : ["Apples", "Oranges", "Peaches"]
      }
    }
  }]
}'

curl -XPUT http://localhost:9200/test/test/1 -d '
{
  "meta": [{
    "timestamp": 1234567890,
    "test": "abc",
    "foo": {
      "bar" : "Hello World"
    },
    "fruits" : {
      "canned": {
        "orchard" : ["Apples", "Oranges", "Peaches"]
      }
    }
  }]
}'    

[jasonpolites]

Also setting specific types on the dynamic_mapping doesn't seem to work for me:

curl -XDELETE "http://localhost:9200/test/"

curl -XPUT "http://localhost:9200/test/"

curl -XPUT http://localhost:9200/test/test/_mapping -d '
{
  "test": {
    "date_detection" : false,
    "numeric_detection": false,
    "dynamic_templates": [
      {
        "all_fields": {
          "match": "*",
          "match_mapping_type" : "integer",
          "mapping": {
              "type": "string"
          }
        }
      }
    ]
  }
}'

curl -XPUT http://localhost:9200/test/test/1 -d '
{
  "test": 123,
}'

curl -XPUT http://localhost:9200/test/test/2 -d '
{
  "test": "abc",
}'    

Results in:

java.lang.NumberFormatException: For input string: "abc"
    at java.lang.NumberFormatException.forInputString(NumberFormatException.java:48)
    at java.lang.Long.parseLong(Long.java:410)
    at java.lang.Long.parseLong(Long.java:468)
    at org.elasticsearch.common.xcontent.support.AbstractXContentParser.longValue(AbstractXContentParser.java:72)
    at org.elasticsearch.index.mapper.core.LongFieldMapper.innerParseCreateField(LongFieldMapper.java:284)
    at org.elasticsearch.index.mapper.core.NumberFieldMapper.parseCreateField(NumberFieldMapper.java:182)
    at org.elasticsearch.index.mapper.core.AbstractFieldMapper.parse(AbstractFieldMapper.java:307)
    ... 11 more

[jasonpolites]

This last comment may be a separate issue, I have opened a new issue to isolate this case:

#2904

The original problem remains however. That is, how to create a mapping that deals with variable types in documents. More specifically, "default to string where not mapped". This does not seem to be possible.

[jasonpolites]

Any workarounds for this? The only solution outside of ES I can see is to manually traverse each document switching non-string values to strings.. which is possible but a little painful.

[clintongormley]

@jasonpolites the original issue seems to have been fixed already.

the issue in your last example works if you use this instead:

 "match_mapping_type" : "integer",