New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Feature Request: Snapshot/Backup #24

Closed
untergeek opened this Issue Jan 23, 2014 · 11 comments

Comments

Projects
None yet
6 participants
@untergeek
Copy link
Member

untergeek commented Jan 23, 2014

With snapshots being part of the new features of ES 1.0 it makes sense to have Curator be able to capture snapshots and save to a designated target (S3, for example). Bonus points if we get it to delete the old index after a successful snapshot/backup.

@electrical

This comment has been minimized.

Copy link

electrical commented Jan 23, 2014

Would be nice to initiate a snapshot and remove it then indeed.

@DanielRedOak

This comment has been minimized.

Copy link

DanielRedOak commented Jan 29, 2014

I would say that the functionality that would best align with Curator would be to allow for removal of old snapshots within whatever repository is used (configured as an outside process) at a specified threshold. Since repository setup is a separate task, this functionality would work with all current and future repository providers. To accomplish this, you would need time based snapshot names, but combining this with the trimming of curator in one script would be awesome! Perhaps curator for local indices + snapshots for backup and/or archiving.

@imperialwicket

This comment has been minimized.

Copy link

imperialwicket commented Feb 18, 2014

It would be great to support restore (maybe even directly from S3) as well.

@jordansissel

This comment has been minimized.

Copy link
Contributor

jordansissel commented Apr 14, 2014

@imperialwicket I'm not sure how restore would necessarily fit in. Curator feels like a program that acts as a maintenance worker for your Elasticsearch needs. Daily maintenance, for example, doesn't feel like something you'd want to do "restore" process. In my mind, it certainly feels weird to propose doing a 'restore' via daily cronjob.

Maybe a separate tool or workflow to help doing restores, that is, if the restore API is insufficient?

Of course, I might be totally misunderstanding your request. Can you help me understand it more? :)

@imperialwicket

This comment has been minimized.

Copy link

imperialwicket commented Apr 14, 2014

@jordansissel fair point; the fact that curator manages snapshots, closing, and deleting seems to qualify it as a candidate for restore capability to me. I completely understand if the concept for curator is 'routine' index maintenance and not general index maintenance, and we can leave restoration for a separate project.

However, I was under the impression that restore could fit into the 'custom tasks' mentioned in #44. That may circumvent this discussion entirely.

@untergeek

This comment has been minimized.

Copy link
Member

untergeek commented Apr 14, 2014

Curator is there to make repetitive practices easier, especially with regards to cron jobs. If it's really a daily occurrence for people, then it makes sense to add it. It does make some sense to have a mode to make it easier to "open" closed indices, or potentially to "restore" archived snapshots, but the latter is tricky to implement cleanly with command-line options, while the former is so simple you probably don't need curator to do it. I mean, how often are indices getting restored? In my experience it's 1 out of 1000 or less. When that arrives, you have to decide, "am I going to restore to a different index name?" among other things. Trying to provide all of the command-line flags for that is a yak shave I'd rather avoid, to be honest.

Regardless, snapshots to filesystem and S3 will be the first goals I tackle on that front.

@victorcoder

This comment has been minimized.

Copy link

victorcoder commented May 28, 2014

+1

@untergeek

This comment has been minimized.

Copy link
Member

untergeek commented May 28, 2014

See #82
It's almost here!

@untergeek

This comment has been minimized.

Copy link
Member

untergeek commented May 28, 2014

And for those who want restore, for now I suggest looking into the Kopf elasticsearch plugin's snapshot tab.

@victorcoder

This comment has been minimized.

Copy link

victorcoder commented May 29, 2014

@untergeek Great news!

@untergeek

This comment has been minimized.

Copy link
Member

untergeek commented Jun 11, 2014

#82 is merged...

@untergeek untergeek closed this Jun 11, 2014

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment