[RFC] Index Management

[elfisher]

The purpose of this issue is to capture feedback and comments regarding the project's request for comments.

[negbie]

Something similar to Graylogs Index Management would be great. http://docs.graylog.org/en/3.0/pages/configuration/index_model.html

[fenneh]

Kibana 6.6 has a similiar feature - https://www.elastic.co/guide/en/kibana/current/creating-index-lifecycle-policies.html

I think the RFC covers off quite a bit of the functionality of this and what is offered by curator.

As for additional things there may be some settings you may want to do;

1. Combine indicies by re-indexing. As an example, if a user has created 30 indicies which are all 4gb in size they may be better off re-indexing all of them into a single one.
2. Altering of some index settings on the fly? E.g. increase refesh_interval when under load, mapping.total_fields.limit increases when the limit has been hit.
3. Re-indexing to other clusters? Keeping non production environments up to date with data for testing purposes etc

[gaby]

How about creating a Kibana Plugin for Curator? Something were we can define rules/actions to take for an index pattern based on intervals or cron.

I'm currently using Curator running in a Docker container to manage my indexes.

Source: https://github.com/deviantony/docker-elk/tree/master/extensions/curator

[elfisher]

@gabrielcalderon thanks for the feedback. This project aims to address many similar use cases, but rather than running as a separate agent, it would be built into Elasticsearch. We plan to support cron expressions to trigger state changes and actions as well as a variety of other conditions. The plan is to eventually have all of this configurable through Kibana. Is there are particular condition or action you think we are missing?

[gaby]

@elfisher The actions/conditions state in the RFC pretty much cover the conditions I currently use with curator. The only thing I didn't see was a condition based on name or is that cover by a different field?

[elfisher]

@gabrielcalderon I'd like to dig into the use case a bit more. Our thinking was you could apply these policies to index templates or individual indexes. Then they would run for whichever indexes they are assigned to. Are you thinking of something more? Thanks!

[CarlMeadows]

I got feedback from a conversation today to add remote reindex as supported operation to move indexes to another cluster. To add to your list for consideration.

[elfisher]

@dbbaughe, @CarlMeadows feedback is interesting. We could potentially support remote reindex as an action.

[marklaczynski]

I think this looks very promising.

[dbbaughe]

Closing this as ISM has already released. Any new feature requests should go into their own github issues.