Skip to content

Alias types: read_only vs read_write #30061

Closed

Description

@elasticmachine

Original comment by @clintongormley:

In Index Lifecycle Management we have come across a problem with rollover which we hope to solve with a change to aliases.

Problem 1: Rollover requires an alias

Rollovers creates a new index, then changes the alias from pointing to the old index to point to the new index.

The alias name is the only index-specific value that is needed by the ILM policy today, because of the rollover action. If we manage to remove this requirement, then the same policy could be applied to several different index patterns/templates. The rollover action in ILM needs a way to inspect the index to know which alias should be swapped. (eg you shouldn't swap an alias like all-logs which is intended for reading).

Problem 2: Coordinating multiple settings

We could add a setting like index.lifecycle.rollover_alias to know which alias to use, but we also need to add the alias to the first index to bootstrap the process. It would be more convenient to have a single way of specifying the alias.

Problem 3: A multi-index alias cannot be used for writing

The rollover alias cannot be applied to more than one index, otherwise it cannot be used for indexing.

Beats or Logstash need to be able to bootstrap ILM and indices by:

  • Putting the ILM policy if it doesn't exist
  • Putting the index template if it doesn't exist
  • Creating the initial index (eg mysql-logs-2018.01.01-00001) with the appropriate alias: mysql-logs-write)

Imagine a thousand beats doing this near midnight: you could create two indices with different dates, both of which have the mysql-logs-write alias, so causing indexing to fail. Ideally, we need a way to stop index creation if the alias is already in use.

Proposal

  • We add a type or mode parameter to aliases, which can be set to read_only or read_write.
  • read_only aliases can point to multiple indices.
  • read_write aliases can only point to a single index, and an exception will be thrown at index creation time or alias update time if the user tries to assign a read_write alias to more than one index.
  • When rollover is called on an index (instead of an alias), then all read_write aliases will be switched to the new index, but read_only aliases will not be changed.
  • For BWC, any aliases which currently point to multiple indices will automatically become read_only, but single-index aliases will remain unspecified. We add a deprecation warning to say that the type will become mandatory in 7.0

Activity

  1. elasticmachine commented on Feb 10, 2018

    @elasticmachine
    CollaboratorAuthor

    Original comment by @MorrieAtElastic:

    Please make sure that the exception associated with mapping multiple indexes to a read-write alias is clearly stated, because this is precisely the kind of error that is likely to be made.

    Other than that I can see no problems with implementing read-write aliases.

  2. elasticmachine commented on Feb 10, 2018

    @elasticmachine
    CollaboratorAuthor

    Original comment by @pickypg:

    We add a deprecation warning to say that the type will become mandatory in 7.0

    What happens for users that don't need to create / set any new aliases between now and then, thus they never see the deprecation? What happens in 7.0? They are automatically set to read_write?

  3. elasticmachine commented on Feb 11, 2018

    @elasticmachine
    CollaboratorAuthor

    Original comment by @s1monw:

    When rollover is called on an index (instead of an alias), then all read_write aliases will be switched to the new index, but read_only aliases will not be changed.

    I don't understand that, you are switching all aliases or only the ones that point to this specific index? What if there are more than one write alias that point to it?

    Also can you apply write ops to an alias with 1 index that is read_only? I think this is very very confusing if that is the case.

    I also wonder what the implications are that we are going from an O(1) pattern to a O(n) pattern. I don't think we should do such a change without considering the implications (I am not sure what they are).

    In Index Lifecycle Management we have come across a problem with rollover which we hope to solve with a change to aliases.

    if we do this are we sure we won't need any specific parameters in the future and is that worth the trouble?

    FWIW I really like specifying the alias from the index via a setting wich can be a one-off this is less ambiguous IMO. The proposal here has too many what-if parts...

  4. elasticmachine commented on Feb 12, 2018

    @elasticmachine
    CollaboratorAuthor

    Original comment by @clintongormley:

    I don't understand that, you are switching all aliases or only the ones that point to this specific index? What if there are more than one write alias that point to it?

    Yes, we'd switch all write aliases to the new index. I agree that I'm not crazy about this. I'd prefer to have a rollover alias type of which there can only be one. The problem is that rollover is x-pack and so it seems weird to introduce such a concept into core.

    Also can you apply write ops to an alias with 1 index that is read_only? I think this is very very confusing if that is the case.

    No, we'd enforce that a read_only alias would not accept writes.

    I also wonder what the implications are that we are going from an O(1) pattern to a O(n) pattern. I don't think we should do such a change without considering the implications (I am not sure what they are).

    Agreed.

    FWIW I really like specifying the alias from the index via a setting wich can be a one-off this is less ambiguous IMO. The proposal here has too many what-if parts...

    This was the original idea. The reasons for not liking it:

    • you need to specify the settings AND add the alias, as opposed to doing both in a single place
    • it doesn't solve the need for atomic index+alias creation from beats

    For the second point, beats could always try to create {index}-0001 as opposed to {index}-{YYYY.MM.dd}-0001, which would reduce the race condition to "highly unlikely".

  5. elasticmachine commented on Feb 12, 2018

    @elasticmachine
    CollaboratorAuthor

    Original comment by @bleskes:

    For the second point, beats could always try to create {index}-0001 as opposed to {index}-{YYYY.MM.dd}-0001, which would reduce the race condition to "highly unlikely"

    How does having read-write alias solve the race condition if multiple beats use a daily pattern and disagree on the date? one of them will try to create an index on it's date and fail because there's already another index which has that alias. That index has the wrong date from the perspective of that specific beat so at this point, we either get stuck or accept the existing index as a target. However, that index will then receive data from a beat with a different idea of what day it is. This mean an index named 2018-02-01 can have data with 2018-01-31. At this point what's the added value of that date over a {index}-001 naming scheme?

  6. elasticmachine commented on Feb 12, 2018

    @elasticmachine
    CollaboratorAuthor

    Original comment by @clintongormley:

    That index has the wrong date from the perspective of that specific beat so at this point, we either get stuck or accept the existing index as a target. However, that index will then receive data from a beat with a different idea of what day it is. This mean an index named 2018-02-01 can have data with 2018-01-31. At this point what's the added value of that date over a {index}-001 naming scheme?

    The beat would use the existing alias. The dates on the indices are a convenience, rather than a requirement. It just makes it easy to see when they were created which is useful when it comes to eg snapshot management etc. But as @colings86 said, we should rather expose the index creation date in more APIs.

  7. elasticmachine commented on Feb 13, 2018

    @elasticmachine
    CollaboratorAuthor

    Original comment by @s1monw:

    This was the original idea. The reasons for not liking it:

    you need to specify the settings AND add the alias, as opposed to doing both in a single place
    it doesn't solve the need for atomic index+alias creation from beats

    I mean we can make a write alias creation a side-effect of the index creation and skip it if the alias already exists. I wonder if we should do this by default at some point. If you create an index we give it some pattern name starting with -00000 and just create an alias with the specified name. This index creation would then set the setting to associate the index with the alias. It's all part of the create index call or some template?

  8. elasticmachine commented on Feb 13, 2018

    @elasticmachine
    CollaboratorAuthor

    Original comment by @clintongormley:

    I like the idea, but there are some issues with it, eg the alias may be apache-logs-write. We would want to create the index apache-logs-000001 rather than apache-logs-write-000001.

    There have been requests to add the ability to infer the index name from an alias,
    eg LINK REDACTED

    What about something like this?

    PUT _template/logs
    {
      "alias_patterns": {
        "apache-logs-write": "apache-logs-000001"
      },
      "aliases": {
        "apache-logs-read": {}
      }
    }
    

    Or with a regex to match more widely:

    PUT _template/logs
    {
      "alias_patterns": {
        "/(.*-logs)-write/": "\\1-000001"
      },
      "aliases": {
        "{index}-read": {}
      }
    }
    

    or if the user wanted the date in there as well:

    PUT _template/logs
    {
      "alias_patterns": {
        "/(.*-logs)-write/": "<\\1-{now/d}-000001>"
      },
      "aliases": {
        "{index}-read": {}
      }
    }
    
  9. elasticmachine commented on Feb 13, 2018

    @elasticmachine
    CollaboratorAuthor

    Original comment by @s1monw:

    I am ok with this proposal! (I still hate regexes)

  10. elasticmachine commented on Feb 13, 2018

    @elasticmachine
    CollaboratorAuthor

    Original comment by @bleskes:

    @clintongormley I want to make sure I understand the entire flow:

    1. A user tries to create an index named apache-logs-0001.
    2. If the index exists, it fails. If it does we match template and you can add a new alias_patterns clause that adds aliases if the pattern matches. That section will add a *-write alias if the index name ends with 0001. (PS - I don't understand how the alias name is derived from "/(.*-logs)-write/").
    3. When a rollover is called, it creates a new index named apache-logs-0002. This one is configured by the template, but the alias-patterns is ignored and the alias isn't added.
    4. The rollover waits for the new index to be allocated and then transfer aliases. How does it know which one to move? I guess we configure another pattern on the lifecycle / API call?

    Last regarding your date example - I think that still has the issue of a race condition when beats are not aligned on dates? now we have two date indices ending with 0001 and both have an alias with a -write suffix?

  11. elasticmachine commented on Feb 13, 2018

    @elasticmachine
    CollaboratorAuthor

    Original comment by @clintongormley:

    @bleskes No.

    • A user indexes into apache-logs-write. The index doesn't exist. The template matches the alias against alias_patterns and uses it to create index apache-logs-000001 and adds the apache-logs-write alias to it.

    • When rollover is called, it create apache-logs-000002 and uses the index.lifecycle.rollover_alias setting to choose which alias to rollover.

    Last regarding your date example - I think that still has the issue of a race condition when beats are not aligned on dates? now we have two date indices ending with 0001 and both have an alias with a -write suffix?

    No, because the beats never create the index. They just index directly into the alias.

  12. elasticmachine commented on Feb 13, 2018

    @elasticmachine
    CollaboratorAuthor

    Original comment by @colings86:

    When rollover is called, it create apache-logs-000002 and uses the index.lifecycle.rollover_alias setting to choose which alias to rollover.

    I'm confused, I thought one of the goals of doing this was to avoid us needed to have a index.lifecycle.rollover_alias setting?

  13. elasticmachine commented on Feb 13, 2018

    @elasticmachine
    CollaboratorAuthor

    Original comment by @clintongormley:

    It was, but see LINK REDACTED - I think it is not worth it given all the complications.

    This solution still requires you to configure things in two places, but it solves both issues.

  14. elasticmachine commented on Feb 23, 2018

    @elasticmachine
    CollaboratorAuthor

    Original comment by @clintongormley:

    After discussion with @talevy and @colings86, we've decided to simplify the alias_patterns proposal as follows:

    1. Initially, don't accept patterns, just literal 1-to-1 mappings, eg:
    PUT _template/logs
    {
      "match_alias": {
        "apache-logs-write": "apache-logs-000001"
      },
      "aliases": {
        "apache-logs-read": {}
      }
    }
    

    (we can always revisit patterns later on if we need to)

    1. match_alias takes preference over index_patterns
    2. Once we have resolved the real index name from the match_alias, all templates with matching index_patterns will be checked and applied in order
    3. A template which uses match_alias is not required to have an index_patterns section, but if it does, then the resolved index name must match one of the patterns.
  15. elasticmachine commented on Mar 25, 2018

    @elasticmachine
    CollaboratorAuthor

    Original comment by @bleskes:

    I spend some more days thinking about this and I'm still worried that the solution we came up with is too "messy":

    1. It adds complexity to a place which is already complex - i.e., templates.
    2. When an indexing requests comes and the string in the url doesn't resolve to an index or an alias, we translate it internally to a create index request where the string is set in the index name. Based on the solution, the master will then resolve the incoming index name parameter first against match_alias sections in all template and then will try to resolve it as an index name. This is trappy, so we should an extra parameter to the CreateIndexRequest called indexOrAlias or some other name (all I can think of sucks). I don't think it's OK to hijack an explicit create index request in a way that will result in an index with another name being created.
    3. As stated, there is no guarantee that the index being created will actually end up having the alias that was used to create it. We can of course add that to the logic of "alias matched" templates - they should automatically add the alias to the create index.
    4. There are still annoying esoteric distributed race conditions which can result in Bad Things. For example - an esoteric combination of indexing and rollover calls can cause a situation where a node which is behind on it's cluster state processing and didn't learn of the alias can translate an incoming indexing op to a create index request to the master. The master will go through the proposed flow and will create a new index named apache-logs-00001 (as it doesn't exist due to the rollover) with the apache-logs-write added to it. This will cause all indexing to stop as we now have a write alias pointing at two indices. This is maybe fixable by requiring the indexOrAlias parameter to be also checked against existing aliases - but that's IMO another messiness in the API.

    I still believe that the "read/write" aliases is the cleanest solution we had so far but I agree it's a big change and we should all be comfortable with it if we go down that path.

    Instead, I would like to suggest another solution. This one is also "messy" but at least it is way more contained, enforced by the master, and we can easily remove it when it's not needed. We can add a parameter to the create index api called "ensureNoExisitngAlias=apache-logs-write" . The master will check, under the cluster state thread that there is no existing alias with that name before creating the index. This will allow the beats to allows start by creating the apache-logs-000001 index (which will succeed or fail with an ResourceAlreadyExistsException due to an existing index/alias) and then starting to index.

  16. elasticmachine commented on Mar 28, 2018

    @elasticmachine
    CollaboratorAuthor

    Original comment by @clintongormley:

    @bleskes and I had a discussion where we came up a proposal which seems better than what has been discussed previously, based on an idea that @s1monw mentioned previously:

    What if we move away from logs-read and logs-write and instead just have the logs alias, and one index pointed to by the logs alias can be marked as is_write_index: true? If not specified, then the first index the alias is assigned to would default to true:

    PUT foo1
    {
      "aliases": {
        "bar": {}
      }
    }
    
    PUT foo2
    {
      "aliases": {
        "bar": {}
      }
    }
    
    GET _alias/bar
    

    would return:

    {
      "foo1": {
        "aliases": {
          "bar": {
            "is_write_index": true
          }
        }
      },
      "foo2": {
        "aliases": {
          "bar": {
            "is_write_index": false
          }
        }
      }
    }
    

    Indexing into bar would write to the foo1 index. Searching against bar would search both foo1 and foo2 indices.

    Beats can create the alias and assign a write index atomically with:

    PUT logs-00001
    {
      "aliases": {
        "logs": {
          "is_write_index": true
        }
      }
    }
    

    This would fail if either index logs-00001 exists or if the logs alias exists and another index is marked as is_write_index: true.

    The logs alias (or logs-read/logs-write) would no longer be mentioned in the index template (although other aliases could be specified). The rollover API would create the new index and, once the index is green, would add the rollover alias to the new index, and mark the new index as is_write_index: true.

    Besides rollover, the other way to change which index is the write index would be with the update-aliases API:

    POST _aliases
    {
      "actions": [
        {
          "set_write_index": {
            "index": "foo2",
            "alias": "bar"
          }
        }
      ]
    }
    

    ILM, Rollover, and Aliases

    ILM still needs to look at the index.lifecycle.rollover_alias setting to know which alias to rollover. We could move this out of the template and make it part of the create index call from Beats:

    PUT logs-00001
    {
      "aliases": {
        "logs": {
          "is_write_index": true
        }
      },
      "settings": {
        "index.lifecycle.rollover_alias": "logs"
      }
    }
    

    and the rollover API, when called by ILM, could transfer this setting too:

    POST /logs/_rollover
    {
      "conditions": {
        "max_size": "50gb"
      },
      "settings": {
        "index.lifecycle.rollover_alias": "logs"
      }
    }
    
  17. talevy commented on Apr 26, 2018

    @talevy
    Contributor

    @clintongormley I've opened #30195

    I'd like to discuss it once more over before proceeding. For example, I do not see the immediate need for the set_write_index alias action. atomic swaps can be done with two add actions. Downside to this is that ordering matters, and one needs to know the previous write index. Rollover is fortunate in that it knows these two things, so it can swap with two add actions.

  18. clintongormley commented on Apr 27, 2018

    @clintongormley
    Contributor

    @talevy makes sense - I agree that we don't need it (at least for now)

  19. bleskes commented on Apr 30, 2018

    @bleskes
    Contributor

    @talevy ++ , though it does depend on where we end up with the alias structures. If we end up going somewhere else than where things look to be heading now, we may need an atomic command (i.e., we need to dynamic visibility and/or late validation to make it work without a command)

  20. talevy commented on Jul 30, 2018

    @talevy
    Contributor

    Closed by tasks in meta issue: #31959

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    :Data Management/ILM+SLMDO NOT USE. Use ":StorageEngine/ILM" or ":Distributed Coordination/SLM" instead.>enhancement

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions