Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Metadata about snapshots destroyed when Consul is destroyed #36

Closed
headw01 opened this issue May 25, 2016 · 6 comments
Closed

Metadata about snapshots destroyed when Consul is destroyed #36

headw01 opened this issue May 25, 2016 · 6 comments
Labels

Comments

@headw01
Copy link

headw01 commented May 25, 2016

When I bring an entire environment up (and down and up), should the database snapshot from the previous time (which is stored in Manta) be used?

It seems that the Consul state isn't maintained anywhere persistently, so the next time the environment comes up, it starts fresh.

If I scale the mysql service after the environment is up, it does get the snapshot from Manta...

First MySQL service startup (after many other "docker-compose up -d")

2016-05-25 16:15:57,893 DEBUG manage.py [pre_start]  pre_start
2016-05-25 16:15:57,895 DEBUG manage.py [pre_start]    has_snapshot
2016-05-25 16:15:58,114 DEBUG manage.py [pre_start]    has_snapshot: None
2016-05-25 16:15:58,114 DEBUG manage.py [pre_start]    initialize_db

prompt> docker-compose scale mysql=2
(slave sees snapshot)

2016-05-25 16:31:44,959 DEBUG manage.py [pre_start]  pre_start
2016-05-25 16:31:44,960 DEBUG manage.py [pre_start]    has_snapshot
2016-05-25 16:31:45,073 DEBUG manage.py [pre_start]    has_snapshot: mysql-backup-2016-05-25T16-16-11Z
2016-05-25 16:31:45,074 DEBUG manage.py [pre_start]    get_snapshot
2016-05-25 16:31:45,960 DEBUG manage.py [pre_start]    get_snapshot: None
@tgross
Copy link
Contributor

tgross commented May 25, 2016

It seems that the Consul state isn't maintained anywhere persistently, so the next time the environment comes up, it starts fresh.

We write a key to Consul to let the preStart know where to find the snapshot. If Consul's data was blown away then yeah, this won't work. You want to keep Consul running.

@headw01
Copy link
Author

headw01 commented May 25, 2016

Is there a way to have the Consul instance write it's key/value pairs to a persistent storage area?

Could the manage.py script look into Manta to find the latest backup if Consul doesn't know?

@tgross
Copy link
Contributor

tgross commented May 26, 2016

Is there a way to have the Consul instance write it's key/value pairs to a persistent storage area?

That would be a function of how and where you deploy Consul. In an environment with attached network volumes (ex. NFS or soon-to-be-released Joyent/RFD26-style volumes) I imagine you could have your Consul containers use that. A better approach is to have a Consul raft for HA and then the failure of individual nodes won't delete data.

Could the manage.py script look into Manta to find the latest backup if Consul doesn't know?

Sure, in theory, but none of our locks can work without Consul. But maybe I'm not understanding the problem you're running into. If you blow away all the metadata for orchestration then none of your service discovery works. I think the right way to fix that is to not blow it away and make sure it's HA.

@headw01
Copy link
Author

headw01 commented Jun 1, 2016

Sorry for the late reply, I was on vacation...

My concern is that if (for whatever reason) all of the Consul instances in HA somehow die, none of the metadata will be available when the environment is repaired. The databases will come backup fresh and clean, and all of the data previously stored is "lost". The last backups would stil exist, but the DB servers would not go and fetch them to restore.

Are there examples for Consul using a network volume to store the metadata persistently (e.g., into Manta)?

@tgross
Copy link
Contributor

tgross commented Jun 6, 2016

Are there examples for Consul using a network volume to store the metadata persistently (e.g., into Manta)?

Manta isn't a block store so it wouldn't be suitable except for backups. You're probably looking for NFS or soon-to-be-released Joyent/RFD26-style volumes. Alternately, you could add ContainerPilot to your Consul instances and give it a postStop handler to snapshot the data to Manta.

@tgross tgross added the usage label Jun 6, 2016
@tgross tgross changed the title Database snapshots persisted to Manta, but not used on 'docker-compose up -d' Metadata about snapshots destroyed when Consul is destroyed Jun 6, 2016
@tgross
Copy link
Contributor

tgross commented Jun 15, 2016

Ok, so this is really a "how do I backup Consul?" question at this point, so if there aren't any objections I'm going to close it. Feel free to reopen if there's more to discuss @headw01

@tgross tgross closed this as completed Jun 15, 2016
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Projects
None yet
Development

No branches or pull requests

2 participants