Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Dynamic Index Setting to Exclude Index from Snapshots #76899

Open
BenB196 opened this issue Aug 24, 2021 · 1 comment
Open

Dynamic Index Setting to Exclude Index from Snapshots #76899

BenB196 opened this issue Aug 24, 2021 · 1 comment
Labels
:Data Management/ILM+SLM Index and Snapshot lifecycle management >enhancement Team:Data Management Meta label for data/management team

Comments

@BenB196
Copy link

BenB196 commented Aug 24, 2021

Feature Request

It would be nice if there was a dynamic index setting that could be set on an index to have it be excluded from all snapshots.

Use Case

I have a cluster which has an SLM policy that snapshots all indices (*).

I'm currently migrating some data from a different cluster to this one via remote reindex.

This data is a bit old so as soon as the remove reindex is complete it gets put through ILM to be made a searchable snapshot.

Index Life:

  1. (content node/hot phase)
    • remote reindex
  2. (warm phase)
    • force merge to 1
    • default compression to best compression
  3. (cold phase)
    • snapshot for searchable snapshot

What I noticed during this process and what I'd like to stop/prevent, is that the SLM policy takes multiple snapshots of the index while on the hot and warm phases. These snapshots often result in some rather large deltas as the data is frequently changing during this time. I'd like to be able to exclude the index from snapshots during this time so that these delta snapshots aren't generated (as they provide no real value).


This setting would be kind of like setting the refresh_interval to -1, but for snapshots, in the sense that it stops a relatively normal cluster action/operation from happening on the specific index.

Bonus

I'd be nice if I could just leave this setting active, then once it hits the cold phase with searchable snapshot it removes the setting automatically so that it doesn't have to be a manual process

@BenB196 BenB196 added >enhancement needs:triage Requires assignment of a team area label labels Aug 24, 2021
@not-napoleon not-napoleon added :Data Management/ILM+SLM Index and Snapshot lifecycle management and removed needs:triage Requires assignment of a team area label labels Sep 1, 2021
@elasticmachine elasticmachine added the Team:Data Management Meta label for data/management team label Sep 1, 2021
@elasticmachine
Copy link
Collaborator

Pinging @elastic/es-data-management (Team:Data Management)

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
:Data Management/ILM+SLM Index and Snapshot lifecycle management >enhancement Team:Data Management Meta label for data/management team
Projects
None yet
Development

No branches or pull requests

3 participants