New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Index fails with MapperParsingException when dynamic_templates provided #2898

Closed
jasonpolites opened this Issue Apr 15, 2013 · 11 comments

Comments

Projects
None yet
3 participants
@jasonpolites

jasonpolites commented Apr 15, 2013

I'm not sure if the title of this is correct, because I'm not sure what the cause of this is, but here's what I'm seeing:

The following sequence of commands will fail:

curl -XDELETE "http://localhost:9200/test/"

curl -XPUT "http://localhost:9200/test/"

curl -XPUT http://localhost:9200/test/test/_mapping -d '
{
  "test": {
    "dynamic_templates": [
      {
        "all_fields": {
          "match": "*",
          "mapping": {
              "type": "string"
          }
        }
      }
    ],  
    "properties": {
      "meta": {
        "type": "object",
        "properties": {
          "timestamp": {
            "store": "yes",
            "type": "date"
          },
          "foo": {
            "type": "object",
            "properties": {
              "bar": {
                  "index": "not_analyzed",
                  "store": "yes",
                  "type": "string"
              }
            }
          }
        }
      }
    }
  }
}'


curl -XPUT http://localhost:9200/test/test/1 -d '
{
  "meta": [{
    "timestamp": 1234567890,
    "foo": {
      "bar" : "Hello World"
    },
    "fruits" : {
      "canned": {
        "orchard" : ["Apples", "Oranges", "Peaches"]
      }
    }
  }]
}'

The failure is

{
    "error": "MapperParsingException[Failed to parse [meta.fruits]]; nested: ElasticSearchIllegalArgumentException[unknown property [canned]]; ",
    "status": 400
}

It's not clear from the documentation why this doesn't work, or how one would assign a default type to unexpected field values if this mapping approach is wrong.

The issue seems to have been introduced from this commit:

6243f8e

@jasonpolites

This comment has been minimized.

Show comment
Hide comment
@jasonpolites

jasonpolites Apr 16, 2013

FYI PUTting the document shown above with the "meta" element as a single object (not an array) works

curl -XPUT http://localhost:9200/test/test/1 -d '
{
  "meta": {
    "timestamp": 1234567890,
    "foo": {
      "bar" : "Hello World"
    },
    "fruits" : {
      "canned": {
        "orchard" : ["Apples", "Oranges", "Peaches"]
      }
    }
  }
}'

{
    "ok": true,
    "_index": "test",
    "_type": "test",
    "_id": "1",
    "_version": 1
}

jasonpolites commented Apr 16, 2013

FYI PUTting the document shown above with the "meta" element as a single object (not an array) works

curl -XPUT http://localhost:9200/test/test/1 -d '
{
  "meta": {
    "timestamp": 1234567890,
    "foo": {
      "bar" : "Hello World"
    },
    "fruits" : {
      "canned": {
        "orchard" : ["Apples", "Oranges", "Peaches"]
      }
    }
  }
}'

{
    "ok": true,
    "_index": "test",
    "_type": "test",
    "_id": "1",
    "_version": 1
}
@jasonpolites

This comment has been minimized.

Show comment
Hide comment
@jasonpolites

jasonpolites Apr 16, 2013

Strike that last comment.. the non-array variant works in 0.19.12 but still fails in 0.20.xx.

The original case fails in 0.19.12 also, but with a different error:

{
    "error": "MapperParsingException[object mapping for [test] with array for [meta] tried to parse as array, but got EOF, is there a mismatch in types for the same field?]",
    "status": 400
}

jasonpolites commented Apr 16, 2013

Strike that last comment.. the non-array variant works in 0.19.12 but still fails in 0.20.xx.

The original case fails in 0.19.12 also, but with a different error:

{
    "error": "MapperParsingException[object mapping for [test] with array for [meta] tried to parse as array, but got EOF, is there a mismatch in types for the same field?]",
    "status": 400
}
@kimchy

This comment has been minimized.

Show comment
Hide comment
@kimchy

kimchy Apr 16, 2013

Member

The reason this happens is that the fruits element ends up being forced to be a string type because of the dynamic template mapping that matches on *.... .

Btw, no need to explicitly store fields, since the _source document is always stored.

Member

kimchy commented Apr 16, 2013

The reason this happens is that the fruits element ends up being forced to be a string type because of the dynamic template mapping that matches on *.... .

Btw, no need to explicitly store fields, since the _source document is always stored.

@jasonpolites

This comment has been minimized.

Show comment
Hide comment
@jasonpolites

jasonpolites Apr 16, 2013

OK thanks. The "store" option is there because I copy/pasted from the "real" mapping which has source storage disabled.

I won't pollute the issue tracker with "forum" style questions, but perhaps you could advise as to how I create a mapping that defaults any unknown value to string. I run into problems with auto-inferring types because my documents are not consistent (outside my control).

I have created a forum thread here:

https://groups.google.com/forum/?fromgroups=#!searchin/elasticsearch/Jason$20MapperParsingException/elasticsearch/kxBC3xZfeh0/-pirsB1eVjQJ

Thanks!

jasonpolites commented Apr 16, 2013

OK thanks. The "store" option is there because I copy/pasted from the "real" mapping which has source storage disabled.

I won't pollute the issue tracker with "forum" style questions, but perhaps you could advise as to how I create a mapping that defaults any unknown value to string. I run into problems with auto-inferring types because my documents are not consistent (outside my control).

I have created a forum thread here:

https://groups.google.com/forum/?fromgroups=#!searchin/elasticsearch/Jason$20MapperParsingException/elasticsearch/kxBC3xZfeh0/-pirsB1eVjQJ

Thanks!

@kimchy

This comment has been minimized.

Show comment
Hide comment
@kimchy

kimchy Apr 16, 2013

Member

you can set in the root object mapping date_detection to false, and numeric_detection set to false, and it won't try and auto detect dates/numerics. That should work out I think.

Member

kimchy commented Apr 16, 2013

you can set in the root object mapping date_detection to false, and numeric_detection set to false, and it won't try and auto detect dates/numerics. That should work out I think.

@jasonpolites

This comment has been minimized.

Show comment
Hide comment
@jasonpolites

jasonpolites Apr 16, 2013

oh awesome.. thanks

jasonpolites commented Apr 16, 2013

oh awesome.. thanks

@jasonpolites

This comment has been minimized.

Show comment
Hide comment
@jasonpolites

jasonpolites Apr 16, 2013

No dice :( the following fails:

curl -XDELETE "http://localhost:9200/test/"

curl -XPUT "http://localhost:9200/test/"

curl -XPUT http://localhost:9200/test/test/_mapping -d '
{
  "test": {
    "date_detection" : false,
    "numeric_detection": false,
    "properties": {
      "meta": {
        "type": "object",
        "properties": {
          "timestamp": {
            "store": "yes",
            "type": "date"
          },
          "foo": {
            "type": "object",
            "properties": {
              "bar": {
                  "index": "not_analyzed",
                  "type": "string"
              }
            }
          }
        }
      }
    }
  }
}'

curl -XPUT http://localhost:9200/test/test/1 -d '
{
  "meta": [{
    "timestamp": 1234567890,
    "test": 123,
    "foo": {
      "bar" : "Hello World"
    },
    "fruits" : {
      "canned": {
        "orchard" : ["Apples", "Oranges", "Peaches"]
      }
    }
  }]
}'

curl -XPUT http://localhost:9200/test/test/1 -d '
{
  "meta": [{
    "timestamp": 1234567890,
    "test": "abc",
    "foo": {
      "bar" : "Hello World"
    },
    "fruits" : {
      "canned": {
        "orchard" : ["Apples", "Oranges", "Peaches"]
      }
    }
  }]
}'    

jasonpolites commented Apr 16, 2013

No dice :( the following fails:

curl -XDELETE "http://localhost:9200/test/"

curl -XPUT "http://localhost:9200/test/"

curl -XPUT http://localhost:9200/test/test/_mapping -d '
{
  "test": {
    "date_detection" : false,
    "numeric_detection": false,
    "properties": {
      "meta": {
        "type": "object",
        "properties": {
          "timestamp": {
            "store": "yes",
            "type": "date"
          },
          "foo": {
            "type": "object",
            "properties": {
              "bar": {
                  "index": "not_analyzed",
                  "type": "string"
              }
            }
          }
        }
      }
    }
  }
}'

curl -XPUT http://localhost:9200/test/test/1 -d '
{
  "meta": [{
    "timestamp": 1234567890,
    "test": 123,
    "foo": {
      "bar" : "Hello World"
    },
    "fruits" : {
      "canned": {
        "orchard" : ["Apples", "Oranges", "Peaches"]
      }
    }
  }]
}'

curl -XPUT http://localhost:9200/test/test/1 -d '
{
  "meta": [{
    "timestamp": 1234567890,
    "test": "abc",
    "foo": {
      "bar" : "Hello World"
    },
    "fruits" : {
      "canned": {
        "orchard" : ["Apples", "Oranges", "Peaches"]
      }
    }
  }]
}'    
@jasonpolites

This comment has been minimized.

Show comment
Hide comment
@jasonpolites

jasonpolites Apr 16, 2013

Also setting specific types on the dynamic_mapping doesn't seem to work for me:

curl -XDELETE "http://localhost:9200/test/"

curl -XPUT "http://localhost:9200/test/"

curl -XPUT http://localhost:9200/test/test/_mapping -d '
{
  "test": {
    "date_detection" : false,
    "numeric_detection": false,
    "dynamic_templates": [
      {
        "all_fields": {
          "match": "*",
          "match_mapping_type" : "integer",
          "mapping": {
              "type": "string"
          }
        }
      }
    ]
  }
}'

curl -XPUT http://localhost:9200/test/test/1 -d '
{
  "test": 123,
}'

curl -XPUT http://localhost:9200/test/test/2 -d '
{
  "test": "abc",
}'    

Results in:

java.lang.NumberFormatException: For input string: "abc"
    at java.lang.NumberFormatException.forInputString(NumberFormatException.java:48)
    at java.lang.Long.parseLong(Long.java:410)
    at java.lang.Long.parseLong(Long.java:468)
    at org.elasticsearch.common.xcontent.support.AbstractXContentParser.longValue(AbstractXContentParser.java:72)
    at org.elasticsearch.index.mapper.core.LongFieldMapper.innerParseCreateField(LongFieldMapper.java:284)
    at org.elasticsearch.index.mapper.core.NumberFieldMapper.parseCreateField(NumberFieldMapper.java:182)
    at org.elasticsearch.index.mapper.core.AbstractFieldMapper.parse(AbstractFieldMapper.java:307)
    ... 11 more

jasonpolites commented Apr 16, 2013

Also setting specific types on the dynamic_mapping doesn't seem to work for me:

curl -XDELETE "http://localhost:9200/test/"

curl -XPUT "http://localhost:9200/test/"

curl -XPUT http://localhost:9200/test/test/_mapping -d '
{
  "test": {
    "date_detection" : false,
    "numeric_detection": false,
    "dynamic_templates": [
      {
        "all_fields": {
          "match": "*",
          "match_mapping_type" : "integer",
          "mapping": {
              "type": "string"
          }
        }
      }
    ]
  }
}'

curl -XPUT http://localhost:9200/test/test/1 -d '
{
  "test": 123,
}'

curl -XPUT http://localhost:9200/test/test/2 -d '
{
  "test": "abc",
}'    

Results in:

java.lang.NumberFormatException: For input string: "abc"
    at java.lang.NumberFormatException.forInputString(NumberFormatException.java:48)
    at java.lang.Long.parseLong(Long.java:410)
    at java.lang.Long.parseLong(Long.java:468)
    at org.elasticsearch.common.xcontent.support.AbstractXContentParser.longValue(AbstractXContentParser.java:72)
    at org.elasticsearch.index.mapper.core.LongFieldMapper.innerParseCreateField(LongFieldMapper.java:284)
    at org.elasticsearch.index.mapper.core.NumberFieldMapper.parseCreateField(NumberFieldMapper.java:182)
    at org.elasticsearch.index.mapper.core.AbstractFieldMapper.parse(AbstractFieldMapper.java:307)
    ... 11 more
@jasonpolites

This comment has been minimized.

Show comment
Hide comment
@jasonpolites

jasonpolites Apr 16, 2013

This last comment may be a separate issue, I have opened a new issue to isolate this case:

#2904

The original problem remains however. That is, how to create a mapping that deals with variable types in documents. More specifically, "default to string where not mapped". This does not seem to be possible.

jasonpolites commented Apr 16, 2013

This last comment may be a separate issue, I have opened a new issue to isolate this case:

#2904

The original problem remains however. That is, how to create a mapping that deals with variable types in documents. More specifically, "default to string where not mapped". This does not seem to be possible.

@jasonpolites

This comment has been minimized.

Show comment
Hide comment
@jasonpolites

jasonpolites Apr 17, 2013

Any workarounds for this? The only solution outside of ES I can see is to manually traverse each document switching non-string values to strings.. which is possible but a little painful.

jasonpolites commented Apr 17, 2013

Any workarounds for this? The only solution outside of ES I can see is to manually traverse each document switching non-string values to strings.. which is possible but a little painful.

@clintongormley

This comment has been minimized.

Show comment
Hide comment
@clintongormley

clintongormley Jul 23, 2014

Member

@jasonpolites the original issue seems to have been fixed already.

the issue in your last example works if you use this instead:

 "match_mapping_type" : "integer",
Member

clintongormley commented Jul 23, 2014

@jasonpolites the original issue seems to have been fixed already.

the issue in your last example works if you use this instead:

 "match_mapping_type" : "integer",
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment