Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Digitalocean builder timeout #3853

Closed
jsaletan opened this Issue Sep 2, 2016 · 12 comments

Comments

Projects
None yet
5 participants
@jsaletan
Copy link

jsaletan commented Sep 2, 2016

I'm using packer 0.10.1 on Ubuntu 16.04 x86_64. I'm able to create a snapshot using the digitalocean builder, but the build times out and returns an error. No artifact ID is returned for the created snapshot.

==> digitalocean: Error waiting for snapshot to complete: Timeout while waiting to for droplet to become 'active'.

Example template and debug log:
https://gist.github.com/jsaletan/b2bc8bd266d163de07e86ea0f222d3fd

Build with:
packer build -var 'do_api_token=<API_TOKEN>' build.json

@rickard-von-essen

This comment has been minimized.

Copy link
Collaborator

rickard-von-essen commented Sep 3, 2016

Try to increase the state timeout, see
https://www.packer.io/docs/builders/digitalocean.html#state_timeout

On Sep 2, 2016 23:34, "jsaletan" notifications@github.com wrote:

I'm using packer 0.10.1 on Ubuntu 16.04 x86_64. I'm able to create a
snapshot using the digitalocean builder, but the build times out and
returns an error. No artifact ID is returned for the created snapshot.

==> digitalocean: Error waiting for snapshot to complete: Timeout while
waiting to for droplet to become 'active'.

Example template and debug log:
https://gist.github.com/jsaletan/b2bc8bd266d163de07e86ea0f222d3fd

Build with:
packer build -var 'do_api_token=<API_TOKEN>' build.json


You are receiving this because you are subscribed to this thread.
Reply to this email directly, view it on GitHub
#3853, or mute the thread
https://github.com/notifications/unsubscribe-auth/AAiCg5J6b3eTT5Q3X9_Fy0Y3MVcTGXw6ks5qmJZggaJpZM4J0GuB
.

@jsaletan

This comment has been minimized.

Copy link
Author

jsaletan commented Sep 3, 2016

I tried increasing state_timeout from the default of 6m to 15m with the same result. I really don't think it's a problem with slow image creation, as it seems to complete in under a minute. Both the Digital Ocean web site and the API show successful image creation, and I've been able to create a droplet from an image right away. It would be nice if packer's logging exposed the API requests it was making, as that might pinpoint the issue.

@rickard-von-essen

This comment has been minimized.

Copy link
Collaborator

rickard-von-essen commented Sep 3, 2016

It would be great if you could check in the web ui what is the state of the snapshot or the cli. Just to verify that the error really is in Packer.

@jsaletan

This comment has been minimized.

Copy link
Author

jsaletan commented Sep 3, 2016

That's what I meant to convey in my last comment. Both the web UI and the API show successful image (snapshot) creation within 90 seconds of starting the build. The droplet from which the snapshot was created has a state of "off" according to the API, and the UI confirms that it is not powered on. Snapshots don't appear to have state. They simply exist or they don't. Once the snapshot has been created, packer begins checking the droplet's state according to the log. It seems packer expects it to be "active", but instead it is "off". I just tried manually powering on the new droplet before packer timed out and that caused the build to complete successfully. So it seems that packer expects the droplet to be active before it will destroy it and complete the build, but it is never made active.

@rickard-von-essen

This comment has been minimized.

Copy link
Collaborator

rickard-von-essen commented Sep 3, 2016

Thank you for your analysis. Seems like packer misses the active state.

@csainty

This comment has been minimized.

Copy link

csainty commented Sep 5, 2016

This is biting me too.
I think there have been changes on the DigitalOcean side.

Previously they would restart a droplet after the snapshot was finished - https://www.digitalocean.com/community/questions/why-does-powered-off-droplet-get-started-after-snapshot

Packer seems to use this fact to know when the snapshot is done so it can clean up.

DigitalOcean recently enabled "live snapshots" so you do not need to power-down first - https://twitter.com/digitalocean/status/751140236298706947

My hypothesis is that they turned off the auto-restart feature now it isn't needed to get around the lack of online snapshots.

As mentioned a workaround at the moment is to manually start the droplet once you see the snapshot is complete in the UI then Packer continues.

@rickard-von-essen

This comment has been minimized.

Copy link
Collaborator

rickard-von-essen commented Sep 5, 2016

Thanks for that insight. Then it should be fairly easily reproducible.

@csainty

This comment has been minimized.

Copy link

csainty commented Sep 5, 2016

I don't have time right now to make a PR of it, but I think I can see what needs doing.

Take the Action returned from starting the snapshot - https://github.com/mitchellh/packer/blob/master/builder/digitalocean/step_snapshot.go#L23

Add a new waitForActionState like the ones already defined in wait.go
But poll the Action status api instead - https://github.com/digitalocean/godo/blob/master/action.go#L19

rickard-von-essen added a commit to rickard-von-essen/packer that referenced this issue Sep 8, 2016

@rickard-von-essen

This comment has been minimized.

Copy link
Collaborator

rickard-von-essen commented Sep 8, 2016

@csainty I blindley followed your instructions and it seems to fix the issue. Can you please review the changes in #3868. Maybe there are redundant waits or steps now.?

rickard-von-essen added a commit that referenced this issue Sep 8, 2016

Merge pull request #3868 from rickard-von-essen/di-snapshot-fix
builders/digitalocean: fixes timeout waiting for snapshot #3853
@kapaggar

This comment has been minimized.

Copy link

kapaggar commented Sep 26, 2016

@csainty Your hypothesis was correct. manually starting packer instance pushes progress in waiting prompt of packer.

@rickard-von-essen Thanks for the fix. waiting for merge to come in packer release.
right now I am on Packer v0.10.2 and this bug is there.

@wmhilton

This comment has been minimized.

Copy link

wmhilton commented Oct 11, 2016

I love it when I google my error ("Timeout while waiting to for droplet to become 'active'") and find the exact Github issue where my error gets fixed. Hahaha! (does happy it-wasn't-my-fault dance)

@jsaletan

This comment has been minimized.

Copy link
Author

jsaletan commented Oct 26, 2016

Confirmed working in 0.11.0. Thanks!

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
You can’t perform that action at this time.