Inconsistent results between template-applied Index Alias and manually-applied Index Alias filters

[cvializ]

I am using Logstash, ElasticSearch 1.1.1 and Kibana to allow groups of users to view logs for their machines. I'd like to use an Index Template to automatically add the Index Alias Filter to the Logstash indices when they are created.

However, the template-added alias filters return inconsistent results, as you can see by running my Gist recreation https://gist.github.com/cvializ/0a494579e19e645ecd4a

A manually created index alias filter returns the correct results as expected. I believe this to be a bug in Elasticsearch.

[clintongormley]

Replicated in master. It looks like the filter is created before the mapping, and isn't updated after the mapping is changed.

The explain API output for a query via alias created via the template is:

ConstantScore(+cache(_type:example) +MatchNoDocsFilter) doesn't match id 0

While for the alias created after the mapping is in place is:

ConstantScore(+cache(_type:example) +cache(host:computer1 host:computer2 host:computer3)), 

This has nothing to do with the filter caching. Slightly simplified recreation:

PUT /accesscontrol/group/admin
{
    "name" : "admin",
    "hosts" : ["computer1","computer2","computer3"]
}

PUT /_template/admin_group
{
    "template" : "logstash-*",
    "aliases" : {        
        "template-admin-{index}" : {
            "filter" : {
                "terms" : {
                    "host" : {
                        "index" : "accesscontrol",
                        "type" : "group",
                        "id" : "admin",
                        "path" : "hosts"
                    }
                }
            }
        }
    }
}

POST /logstash-2014.05.09/example/1
{
    "message":"my sample data",
    "@version":"1",
    "@timestamp":"2014-05-09T16:25:45.613Z",
    "type":"example",
    "host":"computer1"
}

POST /_aliases
{
    "actions" : [
        { 
            "add" : { 
                "index" : "logstash-2014.05.09",
                "alias" : "admin-logstash-2014.05.09",
                "filter": {
                    "terms" : {
                        "host" : {
                            "index" : "accesscontrol",
                            "type" : "group",
                            "id" : "admin",
                            "path" : "hosts"
                        }
                    }
                }
            }
        }
    ]
}

# returns the document
GET /admin-logstash-2014.05.09/_search

# matches no docs
GET /template-admin-logstash-2014.05.09/_search


GET /admin-logstash-2014.05.09/example/1/_explain
{"query": {"match_all": {}}}

GET /template-admin-logstash-2014.05.09/example/1/_explain
{"query": {"match_all": {}}}

[javanna]

I confirm @clintongormley's diagnosis. This happens specifically with terms filter lookup since the filter gets parsed upon index creation, before the dynamic mapping gets created.

Same reproduces with ordinary aliases too if created before the first document gets indexed (just creating an empty index before creating the alias):

PUT /accesscontrol/group/admin
{
    "name" : "admin",
    "hosts" : ["computer1","computer2","computer3"]
}

PUT /logstash-2014.05.09

POST /_aliases
{
    "actions" : [
        { 
            "add" : { 
                "index" : "logstash-2014.05.09",
                "alias" : "admin-logstash-2014.05.09",
                "filter": {
                    "terms" : {
                        "host" : {
                            "index" : "accesscontrol",
                            "type" : "group",
                            "id" : "admin",
                            "path" : "hosts"
                        }
                    }
                }
            }
        }
    ]
}

POST /logstash-2014.05.09/example/1
{
    "message":"my sample data",
    "@version":"1",
    "@timestamp":"2014-05-09T16:25:45.613Z",
    "type":"example",
    "host":"computer1"
}

# matches no docs
GET /admin-logstash-2014.05.09/_search

A work around till we find a proper way to solve it is to manually create the index specifying its mappings before indexing the first document. This way the filter gets created after the mapping is parsed:

PUT /accesscontrol/group/admin
{
    "name" : "admin",
    "hosts" : ["computer1","computer2","computer3"]
}

PUT /_template/admin_group
{
    "template" : "logstash-*",
    "aliases" : {        
        "template-admin-{index}" : {
            "filter" : {
                "terms" : {
                    "host" : {
                        "index" : "accesscontrol",
                        "type" : "group",
                        "id" : "admin",
                        "path" : "hosts"
                    }
                }
            }
        }
    }
}


PUT /logstash-2014.05.09
{
  "mappings": {
    "example" : {
      "properties": {
        "host" : {
          "type" : "string"
        }  
      } 
    }
  }
}

POST /logstash-2014.05.09/example/1
{
    "message":"my sample data",
    "@version":"1",
    "@timestamp":"2014-05-09T16:25:45.613Z",
    "type":"example",
    "host":"computer1"
}

GET /template-admin-logstash-2014.05.09/_search

[cvializ]

Thank you, that helps a lot!

[javanna]

Glad to hear @cvializ !

One thing I missed is that you can also add the mappings to your template, that would make things nicely work without needing to manually create the index upfront:

PUT /accesscontrol/group/admin
{
    "name" : "admin",
    "hosts" : ["computer1","computer2","computer3"]
}


PUT /_template/admin_group
{
    "template" : "logstash-*",
    "aliases" : {        
        "template-admin-{index}" : {
            "filter" : {
                "terms" : {
                    "host" : {
                        "index" : "accesscontrol",
                        "type" : "group",
                        "id" : "admin",
                        "path" : "hosts"
                    }
                }
            }
        }
    },
    "mappings": {
      "example" : {
        "properties": {
          "host" : {
            "type" : "string"
          }  
        } 
      }
    }
}


POST /logstash-2014.05.09/example/1
{
    "message":"my sample data",
    "@version":"1",
    "@timestamp":"2014-05-09T16:25:45.613Z",
    "type":"example",
    "host":"computer1"
}


GET /template-admin-logstash-2014.05.09/_search

[clintongormley]

Fixed by #6664

[loren]

I'm confused how this got fixed with #6664 and marked closed. Without @javanna 's workaround, @clintongormley 's slightly simplified recreation fails in 1.4.0 with

{
   "error": "ElasticsearchIllegalArgumentException[failed to parse filter for alias [template-admin-logstash-2014.05.09]]; nested: QueryParsingException[[logstash-2014.05.09] Strict field resolution and no field mapping can be found for the field with name [host]]; ",
   "status": 400
}

If host is mapped explicitly in the logstash mapping, why can't the admin_group template make use of it?

[clintongormley]

@loren not following exactly. In my example, the host field is not mapped explicitly in the logstash mapping. The way to fix this is to add the host field into the template, eg:

DELETE _all

PUT /accesscontrol/group/admin
{
    "name" : "admin",
    "hosts" : ["computer1","computer2","computer3"]
}

PUT /_template/admin_group
{
    "template" : "logstash-*",
    "mappings": {
      "example": {
        "properties": {
          "host": {
            "type": "string",
            "index": "not_analyzed"
          }
        }
      }
    }, 
    "aliases" : {        
        "template-admin-{index}" : {
            "filter" : {
                "terms" : {
                    "host" : {
                        "index" : "accesscontrol",
                        "type" : "group",
                        "id" : "admin",
                        "path" : "hosts"
                    }
                }
            }
        }
    }
}

POST /logstash-2014.05.09/example/1
{
    "message":"my sample data",
    "@version":"1",
    "@timestamp":"2014-05-09T16:25:45.613Z",
    "type":"example",
    "host":"computer1"
}