Alias to an alias

[asm89]

We are potentially interested in exposing parts of our Elasticsearch cluster to the Internet. The idea is to do authentication in Varnish before we hit the actual backend. With the authentication done we do want to limit the accessible documents on a per customer basis.

With Filtered Aliases it is possible to create an alias for an existing index and add additional filters to each request. It is possible to create such an index alias for every customer in our system. When a request comes in we use Varnish to rewrite the original request URL to include the customer id returned by the authentication backend. Simplified example:

Client
   | /jobs/123
   v
Varnish
   | /{customerId}/jobs/123
   v
Elasticsearch

In our current setup we already use aliases in order to do live reindexing. We have an alias production_jobs pointing to an index production_jobs_1368699241. We can swap the alias to another index without noticeable downtime for our users. It would be great if the setup described above could also leverage this pattern, but now by pointing the alias for a customer to the production index alias. In that case we would only have to update the alias as we do now, instead of doing it for every customer specific alias.

[sumangurung]

I have a very similar use case. I have index aliases set up for zero downtime reindexing. And for a particular search, I search across a couple of indexes(via their aliases). It would be ideal if I could create an alias which points to a group of indexes.

At the moment, the choices are:

1. maintain the list of aliases that I want to search against.
2. create an alias which points to the indexes. And update the alias to use the new index name when we perform live reindexing.

This would be a nice feature addition.

[nik9000]

I'd use it as well. This is my setup:

              foo
  +------------+------------+
  |                         |
foo_a                     foo_b
  |                         |
foo_a_1368699241          foo_b_1368699241

Where foo, foo_a, and foo_b are aliases. Right now, foo points to foo_a_138699241 and foo_b_136899241 so it can include both. It'd be more convenient if foo count point to foo_a and foo_b. The swaps can still be atomic because because you can swap both in the same call but there would be fewer moving parts to juggle with aliases to aliases.

[zwrss]

I'd love to see that feature. That would allow me to handle reindexing while not bothering about all the aliases to that index but one "main" alias that all others should point to. Definitely a nice feature to have.

[bhaskarvk]

I would love to see this feature too.

[RomanShestakov]

+1 would be very useful feature

[jaytaylor]

+1 this would be very valuable.

[clintongormley]

@javanna any thoughts on this?

[nik9000]

If no once gets a chance to look at this in the next few days I can pick it
up. Its something I've wanted for a while but not enough to build it....

On Fri, Jul 25, 2014 at 4:50 AM, Clinton Gormley notifications@github.com
wrote:

@javanna https://github.com/javanna any thoughts on this?

—
Reply to this email directly or view it on GitHub
#3138 (comment)
.

[clintongormley]

I don't believe anybody is working on it. it's still in the discussion phase though, so perhaps let us mull over it a bit before starting? I don't want you to do work that we can't use because it interferes with something we're not aware of yet.

[nik9000]

Sure! I'm not going to have time for the next few days anyway.

On Fri, Jul 25, 2014 at 8:16 AM, Clinton Gormley notifications@github.com
wrote:

I don't believe anybody is working on it. it's still in the discussion
phase though, so perhaps let us mull over it a bit before starting? I don't
want you to do work that we can't use because it interferes with something
we're not aware of yet.

—
Reply to this email directly or view it on GitHub
#3138 (comment)
.

[javanna]

I like the idea too, we need to find a good way to resolve aliases to indices though if we allow for it, as aliases might point to other aliases...

[clintongormley]

... and to ensure that we don't loop: foo -> bar -> foo -> ∞

[nik9000]

Yeah - loops should cause 400 errors when you attempt to build them. Maybe
500 errors if you are able to sneak them in and try to use them. Certainly
not spin forever.....

On Fri, Jul 25, 2014 at 8:50 AM, Clinton Gormley notifications@github.com
wrote:

... and to ensure that we don't loop: foo -> bar -> foo -> ∞

—
Reply to this email directly or view it on GitHub
#3138 (comment)
.

[stevemer]

Has it been decided yet whether this is being implemented?

[nik9000]

I have bigger fish to fry at the moment. If I were to implement it the
first thing is do is look at how hard loop detection would be. Like how
hard inserting it into the right spots would be.
On Aug 6, 2014 10:49 PM, "Steve Merritt" notifications@github.com wrote:

Has it been decided yet whether this is being implemented?

—
Reply to this email directly or view it on GitHub
#3138 (comment)
.

[perryn]

Another vote for this feature.

(I'm looking at combining mutli-tenancy via filtered aliases with index-per-timeframe and I'd rather not have to manage a combinatorial explosion of aliases)

[jaytaylor]

FWIW, my use case was similar. This would be a very useful feature imo!

On Mon, Jan 12, 2015 at 9:54 PM, Perryn Fowler notifications@github.com
wrote:

Another vote for this feature.

(I'm looking at combining mutli-tenancy via filtered aliases with
index-per-timeframe and I'd rather not have to manage a combinatorial
explosion of aliases)

—
Reply to this email directly or view it on GitHub
#3138 (comment)
.

[ghost]

+1

[jannemann]

+1

[Osiriz]

+1

[kubz]

+1

[bradvido]

I have many use cases for this as well, especially since aliases can be filtered -- it would be nice to "nest" those filtered aliases instead of repeating the same common filters in each alias.

For those not aware, you can specify an alias when you create an alias, but it probably isn't working as you'd expect (just adds the alias to the "root" index) as discussed here: #10106

[lewchuk]

+1 for aliases of aliases, my usecase is multiple streams of time based data, I want to window alias each stream individually and have a combined alias for all my streams.

[j0hnsmith]

+1 for aliases of aliases

[mattaylor]

+1

[nik9000]

So I took a look at this a while ago and its not a simple thing to implement which is why no one's done it.

[pcerioli]

+1 I would love to see this feature added as well.

[clintongormley]

We've talked about this a lot internally and nobody is particularly keen on it. For a start, there is nowhere to store an alias which points to other aliases (an alias is stored with the index it points to). Then you have the problems of cycles. Plus there is a practical limit to the number of aliases you can use, as they are stored in the cluster state.

I think we are unlikely to implement this feature.

[colings86]

Agreed, closing this issue

[yinrong]

+1 need this

[sleaze]

I did need this, but since they said they'll never implement it I found another data store instead to solve my problem and it turned out just fine.

Thanks anyways ES team!

[bjorn-ali-goransson]

@clintongormley + gang, the crucial part of alias-of-aliases isn't to make the whole monolithic aliases feature work recursively. The most attractive USP of the aliases feature is the Index Alias and Zero Downtime part.

But if you want the filtering or routing feature for your indices, you're busted on the zero-downtime feature. This is clearly suboptimal.

Surely, it cannot be so complex to make the mere index rewriting recursive, so we should maybe consider splitting them up physically and logically? Aliases do not imply that there is any special functionality - an alias is just an alias: another name for the index.

The other stuff (as I'm not very familiar with it) may be called dynamic/pluggable index filtering or routing, or something else.

[clintongormley]

But if you want the filtering or routing feature for your indices, you're busted on the zero-downtime feature. This is clearly suboptimal.

Why? You can remove multiple aliases from one index (incl filtering and routing) and add them to another in a single atomic step.

[bjorn-ali-goransson]

Ok then. 👍

2016-04-18 11:36 GMT+02:00 Clinton Gormley notifications@github.com:

But if you want the filtering or routing feature for your indices, you're
busted on the zero-downtime feature. This is clearly suboptimal.

Why? You can remove multiple aliases from one index (incl filtering and
routing) and add them to another in a single atomic step.

—
You are receiving this because you commented.
Reply to this email directly or view it on GitHub
#3138 (comment)

[shayts7]

I wonder if this is the time to revive this idea?

I have a use case coming from kibana perspective: I have a lot of visualizations, sometimes I want to switch the visualizations to my "staging env".
Alias could be a great way to do it, but since I'm already using aliases in my elasticsearch index layer (I have "latest" alias that points to a new index that I create every day) I cannot do it :-(

[a1exus]

+1

[mikijov]

+1

I run a large Elastic platform, 1.5TB of RAM worth of clusters, so you can believe me when I tell you that I have many many indices. To keep the platform healthy we need to start automating some index management tasks and we need either alias-to-alias or index-renaming to allow us to transparently manage indices without disrupting users. Basically our problem is that we need two layers, one for users to manage aliased to group related indices together, and one for operators to give us abstraction to manage indices (reindex, reshard, etc) without disrupting the user.

[mapiram]

+1

[slanich]

Ended up here looking for the same thing. I'm not sure why I didn't think of this sooner, but here was my problem:
Actual index name: prod.xenial-logs
Deployed service, trying to read with hard-coded stage name: prod-blue.xenial-logs

Developers are working to fix that, but a great interim fix, was just setting a new alias, pointing to the actual prod indices:
POST /_aliases
{
"actions": [
{
"add": {
"index": "prod.xenial-logs*",
"alias": "prod-blue.xenial-logs"
}
}
]
}

Since ILM is creating a new index every ~4 hours (rapid ingestion), this will allow our service to read in real time, despite using a stage name prefixing the index. Hope this helps someone!

[kaykumar]

+1 alias to alias

[hauntingEcho]

+1 on this. Alias-to-alias mapping makes it a lot easier to present a consistent API to users while updating mappings, when many filters are used. For example:

• index customers-v1.0 holds the actual data
• alias customers points to customers-v1.0
• alias customers_na filters customers to customers based in North America
• alias customers_eu filters customers to customers based in Europe
• { more filtering aliases... }

When we need to update e.g. the customer's street name from a keyword to text, this cannot be done in-place on customers-v1.0, so we have to create customers-v1.1 and use the re-indexing API to move the data over. We then need to hunt down all existing filter aliases that point to customers-v1.0 and update them to point to customers-v1.1. With the option to use aliases-of-aliases, this means we only have one alias to update.

[freakynit]

+1 we need aliaseption

[yinrong]

@elastic-crm @elastic-jira @elastic-data-integration @elastic-stack-testing @elastic-Mak-Ito would you please reopen this issue?

[jlarroque]

+1 to this, is truly needed

[yinrong]

I did need this, but since they said they'll never implement it I found another data store instead to solve my problem and it turned out just fine.

Thanks anyways ES team!

@sleaze what's the alternative? I would like to check it out in my next project.