Failure to create Filtered Alias on empty index with template mappings

[andersosthus]

Hi,

ES 1.4.x gives the error [repro] Strict field resolution and no field mapping can be found for the field with name [testfield] when I try to create an alias on an empty Index that has template mappings.
This works fine in ES 1.3.x.

I see in the guide (http://www.elasticsearch.org/guide/en/elasticsearch/reference/current/indices-aliases.html#filtered) that in 1.4.0.beta1 there was added a clause stating that the field has to exist in the mapping, but it does not mention anything about mapping in templates - so I'm not sure if this is a documentation bug or and ES bug.

Repro case:

curl -XPUT 'http://localhost:9200/_template/repro' -d '{
  "template": "repro",
  "settings": {
    "index.number_of_shards": 1,
    "index.number_of_replicas": 0
  },
  "mappings": {
    "_default_": {
      "properties": {
        "testfield": {
          "type": "string",
          "index": "not_analyzed"
        }
      }
    }
  }
}'

curl -XPOST 'http://localhost:9200/repro'

curl -XPOST  'http://localhost:9200/_aliases' -d '{
   "actions": [
      {
         "add": {
            "index": "repro",
            "alias": "testalias",
            "filter": {
               "term": {
                  "testfield": "filtervalue"
               }
            }
         }
      }
   ]
}'

[shivangshah]

+1. I have the exact same problem. I have more details here:
http://stackoverflow.com/questions/28977610/elastic-search-filtered-aliases-with-unmapped-fields

[javanna]

Templates per se are fine, this happens because the _default_ type doesn't hold actual mappings, but mappings that will get applied whenever a new type is created. A new type is created either when you issue a put mapping or when you index the first document under a new type.

There are a couple of workarounds for this: index a dummy document that contains the field that you want to filter on before you create the alias, or replace the _default_ mapping in the index template with an explicit type (or add the explicit mapping that contains the field definition to the create index call).

This is unfortunate, but I tend not to see it as a bug. Thoughts?

[andersosthus]

Hi,

Yeah, I did that workaround when I encountered this issue, and it's no problem, at least not for us. But the documentation should then maybe clarify this explicitly, especially since this is a change in behaviour from 1.3.x, since I think a lot of people, including myself, thought that the _default mapping in templates worked like regular mappings :)

[shivangshah]

I would like to add here that even if _default is not considered an actual mapping and is applied when a document is indexed/a new mapping is explicitly put, it would still make sense for a filtered alias to also be able to look up the same mapping from _default, especially when in most cases you create a filtered alias when you already know your data (just like you put mappings in _default because you already know your data and thus some default mapping). From my perspective, they both kind of go hand in hand. For both filtered alias & _default, you know your mappings (and filter in case of filtered alias) and that's why you want to define them BEFORE indexing any data. The workaround is tested and is working, but it'd be good to have an actual fix and/or change in documentation.

[javanna]

@shivangshah it's subtle, but the _default_ mapping is just a fallback mechanism to avoid repetitions throughout different types. When the concrete mapping gets created (e.g. when creating the index), you could have a field with same name but a different date type, which takes precedence over the default one. That affects how the filter gets parsed so that it properly works depending on the data type. That is why we have to wait till the concrete mapping is defined and we cannot parse the filter depending on the default mapping, so we make sure that the filter is properly parsed. The workaround here is to use the concrete type(s) rather than the _default_ one. Makes sense?

[shivangshah]

@javanna Actually yes. I see your point now. A pre-defined filtered alias would be a problem if the concrete mapping of a specific type is different than the default.
So now here's another question. Assume that people who know their data and want "default pre-defined filtered aliases" based off of the default mappings, is that a feasible feature request? Would it make sense to have something like this. The usecase should be quite common for people using one common index for multiple tenants and want predefined filtered aliases based on default mapping of a specific field (in most cases, the "tenantId" field or some other "id" field that uniquely identifies tenant specific data).

[javanna]

I am marking this issue for discussion. I do see your point @shivangshah , I am just not sure how we can get there. The feature you are requesting turns around the _default_ mapping as we would need to lock down the mappings for those fields and make sure that the parsed filter is always consistent with the actual mapping for the field. That would mean that instead of a fallback mechanism we would need to be able to force certain mappings for all of the types.

[andersosthus]

@javanna Btw, what changed in 1.4 that changed this, since it worked in 1.3 ?

[javanna]

Good question @andersosthus ! Nothing changed around the default mappings. We made alias filters parsing stricter, have a look at #6664 . The reason for it is that you really want to make sure that the the alias filter gets properly parsed based on the mapped data type.

[andersosthus]

Ok, thanks @javanna. That explains why this "worked" in 1.3 :)

[shivangshah]

@javanna : How about make it more of a configurable flag just like many other configurations handled by the clients ?

[javanna]

I don't think this can be just a flag since it's a completely different feature compared to the current behaviour...

[clintongormley]

I am marking this issue for discussion. I do see your point @shivangshah , I am just not sure how we can get there. The feature you are requesting turns around the default mapping as we would need to lock down the mappings for those fields and make sure that the parsed filter is always consistent with the actual mapping for the field. That would mean that instead of a fallback mechanism we would need to be able to force certain mappings for all of the types.

Given the work happening in #8871 to have fields of the same name in different types always have the same mapping, perhaps this is something we can support after all?

[javanna]

yea, good point @clintongormley that makes sense. Or if we go for parsing filters at search time instead, we could turn off strict query parsing I believe: #10135 (comment) .

[javanna]

we now have an issue for parsing filters at search time, see #10485.

[clintongormley]

Alias filters are now parsed at search time, so this works again