Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Server shutdown failing (snapshot timeout) #101

Open
logwet opened this issue Aug 7, 2019 · 1 comment

Comments

@logwet
Copy link
Contributor

commented Aug 7, 2019

I'm completely unable to shut my server down. Here's the latest section of the Gamocosm log:

019 Aug 7 (06:16:03 PDT) app/workers/wait_for_snapshotting_server_worker.rb:35:in `perform'
Digital Ocean took too long to snapshot server. Aborting
2019 Aug 7 (06:16:03 PDT) app/workers/wait_for_snapshotting_server_worker.rb:32:in `perform'
Still waiting for Digital Ocean server to snapshot, tried 128 times
2019 Aug 7 (06:15:04 PDT) app/workers/wait_for_snapshotting_server_worker.rb:32:in `perform'
Still waiting for Digital Ocean server to snapshot, tried 120 times
2019 Aug 7 (06:14:03 PDT) app/workers/wait_for_snapshotting_server_worker.rb:32:in `perform'
Still waiting for Digital Ocean server to snapshot, tried 112 times
2019 Aug 7 (06:13:11 PDT) app/workers/wait_for_snapshotting_server_worker.rb:32:in `perform'
Still waiting for Digital Ocean server to snapshot, tried 104 times
2019 Aug 7 (06:12:20 PDT) app/workers/wait_for_snapshotting_server_worker.rb:32:in `perform'
Still waiting for Digital Ocean server to snapshot, tried 96 times
2019 Aug 7 (06:11:14 PDT) app/workers/wait_for_snapshotting_server_worker.rb:32:in `perform'
Still waiting for Digital Ocean server to snapshot, tried 88 times
2019 Aug 7 (06:10:23 PDT) app/workers/wait_for_snapshotting_server_worker.rb:32:in `perform'
Still waiting for Digital Ocean server to snapshot, tried 80 times
2019 Aug 7 (06:09:20 PDT) app/workers/wait_for_snapshotting_server_worker.rb:32:in `perform'
Still waiting for Digital Ocean server to snapshot, tried 72 times
2019 Aug 7 (06:08:28 PDT) app/workers/wait_for_snapshotting_server_worker.rb:32:in `perform'
Still waiting for Digital Ocean server to snapshot, tried 64 times

I took a look at the DigitalOcean control panel and I have 4 snapshots on there, coinciding with the automated shutdown and the three manual shutdowns I tried after that. So I'm pretty sure that Gamocosm is doing every thing up until snapshotting correctly, and that DO has no issues snapshotting the server, but the issue happens while Gamocosm is waiting for DO to snapshot and times out. I have no way of stopping the server which is pretty undesirable because it's one of the more expensive plans :(

This issue only arose after I bumped the server plan from s-2vcpu-2gb to s-2vcpu-4gb which comes with 80gb of disk storage over the 2gb plan. I think (just a guess) DO takes longer to snapshot higher disk space droplets which may be why I haven't experienced his before. A quick look at the server worker being referenced in the logs leads me to believe line 34

if times >= 128
server.log('Digital Ocean took too long to snapshot server. Aborting')
server.reset_state
return
end
is the issue here because every time the process has failed it stops after roughly 120 attempts. A solution may be to dynamically change this timeout value depending on the disk space of the plan slug the user has selected or bump it to a higher value (256?)

logwet added a commit to logwet/Gamocosm that referenced this issue Aug 7, 2019

@Raekye

This comment has been minimized.

Copy link
Member

commented Aug 7, 2019

I've merged the PR and updated the server. I'm busy working right now but if you're still having issues later I can take a closer look

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
2 participants
You can’t perform that action at this time.