Mapping: MapperParsingException when create default mapping with 'include_in_all' nested

[JeffreyZZ]

One noticeable issue found with ES 1.2.0 during our deployment is that it threw exception when created default mappings with ‘include_in_all’ nested under it (doesn’t matter it’s set to true/false). For example, the following index creation command returns error when against ES 1.2.0 but it works well against ES 1.1.1

PUT test
{
"mappings": {
"default": {
"include_in_all": true
}
}
}

Result
{
"error": "MapperParsingException[mapping [default]]; nested: MapperParsingException[Root type mapping not empty after parsing! Remaining fields: [include_in_all : true]]; ",
"status": 400
}

Is this a regression with ES 1.2.0?

[clintongormley]

HI @JeffreyZZ

That mapping is incorrect. include_in_all needs to be specified under a field, not at the top level. In previous versions it was just ignored, but in 1.2 it now tells you that it is incorrect.

You're probably looking for:

PUT /test 
{
  "mappings": {
    "_default_": {
      "_all": {
        "enabled": false
      }
    }
  }
}

[JeffreyZZ]

Make sense. Thanks @clintongormley !

[skurfuerst]

Are you sure? According to http://www.elasticsearch.org/guide/en/elasticsearch/reference/current/mapping-root-object-type.html, it says:

The root object mapping is an object type mapping that maps the root object (the type itself). On top
of all the different mappings that can be set using the object type mapping, it allows for additional,
type level mapping definitions.

and in http://www.elasticsearch.org/guide/en/elasticsearch/reference/current/mapping-object-type.html, it says:

include_in_all can be set on the object type level. When set, it propagates down to all the inner
mappings defined within the object that do no explicitly set it.

Greets, Sebastian

[clintongormley]

@skurfuerst

You're absolutely right - this is a regression. /cc @brwe

[brwe]

Yes, opened pull request #6353

[JeffreyZZ]

@brwe @brwe For ES 1.2.0, is there any workaround to this issue or any impact of this issue?

[brwe]

You could create a dynamic template for the types (example below). "include_in_all" will then not be set in the root object but still be applied to all fields and objects.

As for the impact, if "include_in_all" is already set in the root type and you upgrade to 1.2, a MapperParsingException will be thrown on startup. The next time that the type mapping is updated, the "include_at_all" setting will be removed.

Unfortunately, I found that a side effect seems to be that it will not only be removed from the root object, but also from fields. I'll have to look into this further.

"include_in_all" via "dynamic_template":

PUT test
{
   "mappings": {
      "_default_": {
         "dynamic_templates": [
            {
               "include_all": {
                  "match": "*",
                  "mapping": {
                     "include_in_all": "true"
                  }
               }
            }
         ]
      }
   }
}
POST test/cat/1
{
    "text": "text"
}
GET test/_mapping

Result should be

{
   "test": {
      "mappings": {
         "_default_": {
            "dynamic_templates": [
               {
                  "include_all": {
                     "mapping": {
                        "include_in_all": "true"
                     },
                     "match": "*"
                  }
               }
            ],
            "properties": {}
         },
         "cat": {
            "dynamic_templates": [
               {
                  "include_all": {
                     "mapping": {
                        "include_in_all": "true"
                     },
                     "match": "*"
                  }
               }
            ],
            "properties": {
               "text": {
                  "type": "string",
                  "include_in_all": true
               }
            }
         }
      }
   }
}

[JeffreyZZ]

@brwe @clintongormley

Seems this regression also has impact on the index template if the template defines 'include_in_all' under the type. It allows you to create the template but you're NOT able to create the index of the template in ES 1.2.0. There is the repro steps :

[Step 1] create the following index template against ES 1.2.0 cluster
PUT /_template/template_logs
{
"order": 0,
"template": "logs*",
"settings": {
"index.analysis.analyzer.keyword_analyzer.tokenizer": "keyword"
},
"mappings": {
"LogMessage": {
"include_in_all": false,
"properties": {
"activityId": {
"analyzer": "keyword_analyzer",
"type": "string"
}
}
}
}
}

[Step 2] Try to index this the following document
PUT /logs-monday/LogMessage/1
{
"activityId": "fcfdab46-730a-4918-b5bd-1da9448608b1"
}

[Result]
{
"error": "RemoteTransportException[[ES-TEST-2-master][inet[/10.1.0.38:9300]][indices/create]]; nested: MapperParsingException[mapping [LogMessage]]; nested: MapperParsingException[Root type mapping not empty after parsing! Remaining fields: [include_in_all : false]]; ",
"status": 400
}

[Expected]
Index 'logs-monday' is created and 1 document is indexed.

Try the above steps against ES 1.1.1, it works. Any workaround?

[brwe]

There is unfortunately no workaround.

[JeffreyZZ]

@brwe Thanks for reply! BTW, any ETA for ES 1.2.1 release?

[brwe]

We released yesterday: http://www.elasticsearch.org/blog/elasticsearch-1-2-1-released/
Sorry for the late reply.

[JeffreyZZ]

@brwe Excellent, thank you!

[clintongormley]

Your mapping is invalid. I'm assuming the intent of the above is to create the type pseudo_doc in any new index, which has the field content?

In this case, the config/mappings/_default/pseudo_doc.json file should look like this:

{
  "properties": {
    "content": {
      "dynamic": false,
      "properties": {
        "author_id": {
          "type": "string"
        },
        "author_name": {
          "type": "string"
        },
        "content": {
          "type": "string"
        },
        "title": {
          "type": "string"
        },
        "urls": {
          "type": "string"
        },
        "destination_url": {
          "type": "string"
        },
        "publisher": {
          "type": "string"
        }
      }
    }
  }
}

Also, I really recommend doing something like this with index templates, rather than with config files. It is much easier to manage such things via the API instead of via static config files.