Repository navigation
Description
Original comment by @clintongormley:
In Index Lifecycle Management we have come across a problem with rollover which we hope to solve with a change to aliases.
Problem 1: Rollover requires an alias
Rollovers creates a new index, then changes the alias from pointing to the old index to point to the new index.
The alias name is the only index-specific value that is needed by the ILM policy today, because of the rollover action. If we manage to remove this requirement, then the same policy could be applied to several different index patterns/templates. The rollover action in ILM needs a way to inspect the index to know which alias should be swapped. (eg you shouldn't swap an alias like all-logs which is intended for reading).
Problem 2: Coordinating multiple settings
We could add a setting like index.lifecycle.rollover_alias to know which alias to use, but we also need to add the alias to the first index to bootstrap the process. It would be more convenient to have a single way of specifying the alias.
Problem 3: A multi-index alias cannot be used for writing
The rollover alias cannot be applied to more than one index, otherwise it cannot be used for indexing.
Beats or Logstash need to be able to bootstrap ILM and indices by:
- Putting the ILM policy if it doesn't exist
- Putting the index template if it doesn't exist
- Creating the initial index (eg
mysql-logs-2018.01.01-00001) with the appropriate alias:mysql-logs-write)
Imagine a thousand beats doing this near midnight: you could create two indices with different dates, both of which have the mysql-logs-write alias, so causing indexing to fail. Ideally, we need a way to stop index creation if the alias is already in use.
Proposal
- We add a
typeormodeparameter to aliases, which can be set toread_onlyorread_write. read_onlyaliases can point to multiple indices.read_writealiases can only point to a single index, and an exception will be thrown at index creation time or alias update time if the user tries to assign aread_writealias to more than one index.- When rollover is called on an index (instead of an alias), then all
read_writealiases will be switched to the new index, butread_onlyaliases will not be changed. - For BWC, any aliases which currently point to multiple indices will automatically become
read_only, but single-index aliases will remain unspecified. We add a deprecation warning to say that thetypewill become mandatory in 7.0
Activity
elasticmachine commentedon Feb 10, 2018
Original comment by @MorrieAtElastic:
Please make sure that the exception associated with mapping multiple indexes to a read-write alias is clearly stated, because this is precisely the kind of error that is likely to be made.
Other than that I can see no problems with implementing read-write aliases.
elasticmachine commentedon Feb 10, 2018
Original comment by @pickypg:
What happens for users that don't need to create / set any new aliases between now and then, thus they never see the deprecation? What happens in 7.0? They are automatically set to
read_write?elasticmachine commentedon Feb 11, 2018
Original comment by @s1monw:
I don't understand that, you are switching all aliases or only the ones that point to this specific index? What if there are more than one write alias that point to it?
Also can you apply write ops to an alias with 1 index that is
read_only? I think this is very very confusing if that is the case.I also wonder what the implications are that we are going from an
O(1)pattern to aO(n)pattern. I don't think we should do such a change without considering the implications (I am not sure what they are).if we do this are we sure we won't need any specific parameters in the future and is that worth the trouble?
FWIW I really like specifying the alias from the index via a setting wich can be a one-off this is less ambiguous IMO. The proposal here has too many
what-ifparts...elasticmachine commentedon Feb 12, 2018
Original comment by @clintongormley:
Yes, we'd switch all write aliases to the new index. I agree that I'm not crazy about this. I'd prefer to have a
rolloveralias type of which there can only be one. The problem is that rollover is x-pack and so it seems weird to introduce such a concept into core.No, we'd enforce that a
read_onlyalias would not accept writes.Agreed.
This was the original idea. The reasons for not liking it:
For the second point, beats could always try to create
{index}-0001as opposed to{index}-{YYYY.MM.dd}-0001, which would reduce the race condition to "highly unlikely".elasticmachine commentedon Feb 12, 2018
Original comment by @bleskes:
How does having read-write alias solve the race condition if multiple beats use a daily pattern and disagree on the date? one of them will try to create an index on it's date and fail because there's already another index which has that alias. That index has the wrong date from the perspective of that specific beat so at this point, we either get stuck or accept the existing index as a target. However, that index will then receive data from a beat with a different idea of what day it is. This mean an index named
2018-02-01can have data with2018-01-31. At this point what's the added value of that date over a{index}-001naming scheme?elasticmachine commentedon Feb 12, 2018
Original comment by @clintongormley:
The beat would use the existing alias. The dates on the indices are a convenience, rather than a requirement. It just makes it easy to see when they were created which is useful when it comes to eg snapshot management etc. But as @colings86 said, we should rather expose the index creation date in more APIs.
elasticmachine commentedon Feb 13, 2018
Original comment by @s1monw:
I mean we can make a write alias creation a side-effect of the index creation and skip it if the alias already exists. I wonder if we should do this by default at some point. If you create an index we give it some pattern name starting with
-00000and just create an alias with the specified name. This index creation would then set the setting to associate the index with the alias. It's all part of the create index call or some template?elasticmachine commentedon Feb 13, 2018
Original comment by @clintongormley:
I like the idea, but there are some issues with it, eg the alias may be
apache-logs-write. We would want to create the indexapache-logs-000001rather thanapache-logs-write-000001.There have been requests to add the ability to infer the index name from an alias,
eg LINK REDACTED
What about something like this?
Or with a regex to match more widely:
or if the user wanted the date in there as well:
elasticmachine commentedon Feb 13, 2018
Original comment by @s1monw:
I am ok with this proposal! (I still hate regexes)
elasticmachine commentedon Feb 13, 2018
Original comment by @bleskes:
@clintongormley I want to make sure I understand the entire flow:
apache-logs-0001.alias_patternsclause that adds aliases if the pattern matches. That section will add a*-writealias if the index name ends with0001. (PS - I don't understand how the alias name is derived from"/(.*-logs)-write/").apache-logs-0002. This one is configured by the template, but thealias-patternsis ignored and the alias isn't added.Last regarding your date example - I think that still has the issue of a race condition when beats are not aligned on dates? now we have two date indices ending with 0001 and both have an alias with a
-writesuffix?elasticmachine commentedon Feb 13, 2018
Original comment by @clintongormley:
@bleskes No.
A user indexes into
apache-logs-write. The index doesn't exist. The template matches the alias againstalias_patternsand uses it to create indexapache-logs-000001and adds theapache-logs-writealias to it.When rollover is called, it create
apache-logs-000002and uses theindex.lifecycle.rollover_aliassetting to choose which alias to rollover.No, because the beats never create the index. They just index directly into the alias.
elasticmachine commentedon Feb 13, 2018
Original comment by @colings86:
I'm confused, I thought one of the goals of doing this was to avoid us needed to have a
index.lifecycle.rollover_aliassetting?elasticmachine commentedon Feb 13, 2018
Original comment by @clintongormley:
It was, but see LINK REDACTED - I think it is not worth it given all the complications.
This solution still requires you to configure things in two places, but it solves both issues.
elasticmachine commentedon Feb 23, 2018
Original comment by @clintongormley:
After discussion with @talevy and @colings86, we've decided to simplify the
alias_patternsproposal as follows:(we can always revisit patterns later on if we need to)
match_aliastakes preference overindex_patternsmatch_alias, all templates with matchingindex_patternswill be checked and applied inordermatch_aliasis not required to have anindex_patternssection, but if it does, then the resolved index name must match one of the patterns.elasticmachine commentedon Mar 25, 2018
Original comment by @bleskes:
I spend some more days thinking about this and I'm still worried that the solution we came up with is too "messy":
match_aliassections in all template and then will try to resolve it as an index name. This is trappy, so we should an extra parameter to theCreateIndexRequestcalledindexOrAliasor some other name (all I can think of sucks). I don't think it's OK to hijack an explicit create index request in a way that will result in an index with another name being created.apache-logs-00001(as it doesn't exist due to the rollover) with theapache-logs-writeadded to it. This will cause all indexing to stop as we now have a write alias pointing at two indices. This is maybe fixable by requiring theindexOrAliasparameter to be also checked against existing aliases - but that's IMO another messiness in the API.I still believe that the "read/write" aliases is the cleanest solution we had so far but I agree it's a big change and we should all be comfortable with it if we go down that path.
Instead, I would like to suggest another solution. This one is also "messy" but at least it is way more contained, enforced by the master, and we can easily remove it when it's not needed. We can add a parameter to the create index api called "ensureNoExisitngAlias=apache-logs-write" . The master will check, under the cluster state thread that there is no existing alias with that name before creating the index. This will allow the beats to allows start by creating the
apache-logs-000001index (which will succeed or fail with anResourceAlreadyExistsExceptiondue to an existing index/alias) and then starting to index.elasticmachine commentedon Mar 28, 2018
Original comment by @clintongormley:
@bleskes and I had a discussion where we came up a proposal which seems better than what has been discussed previously, based on an idea that @s1monw mentioned previously:
What if we move away from
logs-readandlogs-writeand instead just have thelogsalias, and one index pointed to by thelogsalias can be marked asis_write_index: true? If not specified, then the first index the alias is assigned to would default totrue:would return:
Indexing into
barwould write to thefoo1index. Searching againstbarwould search bothfoo1andfoo2indices.Beats can create the alias and assign a write index atomically with:
This would fail if either index
logs-00001exists or if thelogsalias exists and another index is marked asis_write_index: true.The
logsalias (orlogs-read/logs-write) would no longer be mentioned in the index template (although other aliases could be specified). The rollover API would create the new index and, once the index is green, would add the rollover alias to the new index, and mark the new index asis_write_index: true.Besides rollover, the other way to change which index is the write index would be with the update-aliases API:
ILM, Rollover, and Aliases
ILM still needs to look at the
index.lifecycle.rollover_aliassetting to know which alias to rollover. We could move this out of the template and make it part of the create index call from Beats:and the rollover API, when called by ILM, could transfer this setting too:
talevy commentedon Apr 26, 2018
@clintongormley I've opened #30195
I'd like to discuss it once more over before proceeding. For example, I do not see the immediate need for the
set_write_indexalias action. atomic swaps can be done with twoaddactions. Downside to this is that ordering matters, and one needs to know the previous write index. Rollover is fortunate in that it knows these two things, so it can swap with twoaddactions.clintongormley commentedon Apr 27, 2018
@talevy makes sense - I agree that we don't need it (at least for now)
bleskes commentedon Apr 30, 2018
@talevy ++ , though it does depend on where we end up with the alias structures. If we end up going somewhere else than where things look to be heading now, we may need an atomic command (i.e., we need to dynamic visibility and/or late validation to make it work without a command)
talevy commentedon Jul 30, 2018
Closed by tasks in meta issue: #31959