Added support for aliases to index templates

[javanna]

This PR finalizes the work that has been done in #2739 towards adding support for aliases to index templates.

Aliases can now be specified when creating an index template as follows:

curl -XPUT localhost:9200/_template/template_1 -d '
{
    "template" : "te*",
    "settings" : {
        "number_of_shards" : 1
    },
    "aliases" : {
        "alias1" : {},
        "alias2" : {
            "filter" : {
                "term" : {"user" : "kimchy" }
            },
            "routing" : "kimchy"
        },
        "{index}-alias" : {}
    }
}
'

The {index} placeholder within the alias name will be replaced with the actual index name that the template gets applied to during index creation.

Closes #1825

[kimchy]

I like it!. I think we miss one use case, where the index name is logging-2012.01.01, and we want to create an alias classed alas-2012.01.01. I can't come up with a simple way to design it, maybe allow to define a regular expression from the index name to extract? Something like: {index:xxx}?

Also, are we good with {...} as the replacement chars? Should it be ${..}?

[clintongormley]

Also, are we good with {...} as the replacement chars? Should it be ${..}?

We already use "type": "{dynamic_type}" in dynamic templates, so I'm fine with {} being reserved.

I think we miss one use case, where the index name is logging-2012.01.01, and we want to create an alias classed alas-2012.01.01. I can't come up with a simple way to design it, maybe allow to define a regular expression from the index name to extract? Something like: {index:xxx}?

I'm not understanding this use case. Typically alias names are kinda hard coded, eg last_3_months, rather than being a direct rename of an index. (and i'm also struggling to see how one would specify this syntax with the current layout).

I suppose you could have something like alias-{year} so index-2014-01 -> alias-2014. Perhaps:

{
  "aliases": {
      "alias-{\1}": {
           "regex": "index-(\d+).+"
      }
  }
}

So the presence of the regex key would trigger a regex replace on the alias name. Not convinced that this is worth doing.

The other thing I'm missing is the ability to automatically remove older indices from aliases, eg the last_3_months use case. Easier to do via cron though, so probably not needed.

[uboness]

Also, are we good with {...} as the replacement chars? Should it be ${..}?

With query templates now in place, how about we move all our placeholders to mustache syntax (ie. {{placeholder}})

[kimchy]

@clintongormley actually, most of the cases for aliases in LS was not around "last-3-months", since Kibana solves it easily thought the main interaction that people have with it. What people were after was using aliases as a way to filter data, in which case, using the same alias created that follows the naming rules as the index name, but with a filter.

[jordansissel]

@kimchy +1 to the use case you describe; kibana indeed solves the "last-N-months" problem.

Templated aliases helps logstash (or any time-oriented indexing name) users get filtered aliases. In the past, we've given some hand-wavey answer "Well just set up a cron job to create aliases every day!" and that answer was lame. This feature is sweeeet.

[jordansissel]

I think we miss one use case, where the index name is logging-2012.01.01, and we want to create an alias classed alas-2012.01.01

I agree some users might ask for this, but I'm not totally sure it's needed. The difference between alas-2012.01.01 and alas-logstash-2012.01.01 isn't much, though maybe some users would like to strip the "logstash-" part as you propose. I don't have any data to support what users might want; I will ask!

[torrancew]

In a previous life, my department was subject to regular audits of the garden variety (SOX, PCI, SOC, etc). As such, I often find myself wondering if monthly/yearly aliases would have provided more operational effectiveness during audit season. I begin to imagine a wonderful fantasy land where I simply run (or even cron) a custom script that queries the desired subset of logs via an alias, and outputs a report that is in the format the auditor desires. It's great - people are cheering, audits practically run themselves, and everything is magical.

Then, I wake up. I realize that every audit I went through during that time ended in precisely the same way - the auditor staring over my shoulder as I show them the log visualization tool's interface with the query I promise him or her is accurate for their question. Eventually the auditor says, "Thank you, now please e-mail me the full screenshot of this". I tried many times, but never succeeded, in getting them to accept other, more easily automatable formats.

As such, I'm on the fence about the "last-N-months" problem - I see a theoretical value to it, but I myself have never worked in an environment where that theoretical value would be realized or preferred to what Kibana provides. Likewise, if you're already writing a custom script for such a report, it would not be much more effort to automate the generation of the index list (though I don't know if you'd risk overflowing the max URI length at some point).

All in all, I'm +1 to the feature in general (adding filtered aliases via mapping is something I've been longing for for a while now), and have no real strong opinion on the "last-N-months/last-year" use case - especially when I consider the amount of resources such a query would be likely to consume.

[javanna]

Thanks for your comments guys, seems like we are on the right track here. Currently the alias name needs to contain the whole index name, and it seems to me that adding the ability to use only a part of it wouldn't buy much compared to how it would complicate the syntax.

I lean towards keeping things simple for now, unless we find a magic syntax other than regexes :)

[jordansissel]

unless we find a magic syntax other than regexes

If we find such a magic syntax, I would love to purge regexes from my life entirely ;)

[kimchy]

code looks great!

[stevencdavis]

I think we miss one use case, where the index name is logging-2012.01.01, and we want to create an alias classed alas-2012.01.01

Our use case is similar to this -- we have versioned indices named per month (e.g. projectA-2014.01__v1, projectA-2014.02__v1), and we create aliases to make it easy to query a whole year or all of our documents for a particular project (e.g. aliases projectA-2014, projectA) and also to allow us to use the aliases as a means of updating index versions without downtime (i.e. by allowing the application level to use a particular alias while swapping out projectA-...__v1 for projectA..._v2 -- I believe this is a best-practice). This allows us to logically group indices and to keep our data and processing versioned with zero downtime. Also, last-N-months wouldn't work for us because we have many projects (e.g. projectA, projectB, etc.), and we create aliases for each of these.

Currently, we have a cron job that checks periodically to see if any new indices have been created that need additional aliases, but obviously this is a bit hacky. Am I missing an obvious alternative method, or is this really not a common approach?

[javanna]

Hi @stevencdavis, it is a common approach, the problem here is finding the right syntax to be able to use partial indices names as part of alias names. We'll get this PR merged as it seems to be a good start and give these other requirements some more thoughts, we can always improve this later on.

[javanna]

Merged, thanks a lot @jbrook !

[chetandhembre]

Hey.. Is there any changes in this after this PR?
I have per day base index from logstash like test-2016-08-29 / test-2016-08-30. I want to create aliase like test. I am doing this because I do not want to use logstash created index name in my application code.
above PR helps you to add index name i aliase but not partial name of it.
Is there way to do it? or do I have alternative approach to my problem?
Thanks you.