When setting up the host, there is a delay before go-machine-service is ready. If user adds host before it's ready, the vm will not get created and immediately go into an "Active" state.

[deniseschannon]

Version: v0.11.1

When adding a Digital Ocean machine, it sometimes just never gets started. Other times, it works.

For the time that never gets started, the UI eventually shows my host to look like this:

When I check the API in the hosts section, I can't find this host to check any UI status, so not sure why it's showing up in my UI.

@cjellick looked into this. The bug is most likely related to the fact that after "Host Setup", there is a delay before Machine Service comes online. Therefore, if you try and create machine before Machine Service is online it will go to this state.

[vincent99]

This screen shows a combination of /v1/machines and /v1/hosts, so you're seeing that the machine is active (which should mean the DigitalOcean VM is created) but the agent is never checking in, so the host record we expect to eventually get created isn't.

[deniseschannon]

The DigitalOcean VM was never created. In the past when this has happened before, I've seen it just sit in "Creating" state, but this was the first time that it went to an "Active" state.

[vincent99]

Can you find the corresponding /v1/machines entry? What state is it in?

[deniseschannon]

State="Active"

[vincent99]

Ok, so this is an issue in Machine and/or the Rancher integration with it, the UI is showing what it's told. Ping @cjellick @will-chan

[cjellick]

Looking. If you didn't delete the machine, let me know the name and/or id so I can look it up in the api and logs

[cjellick]

I looked at this one a little bit over the weekend. Seems like there could be a legitimate concurrency issue. Something to keep an eye on if nothing else.

It could also be that the machine got into a funky state and that the error handling changes could alleviate this.

[vincent99]

This happened to me twice today against a new 10acre with the :beta tag.. Anything you want to look at @cjellick?

[cjellick]

Will touch base with @will-chan on it today. My hunch is that it is related to the update we did to push more detailed status changes.

[cjellick]

@vincent99 Did you have multiple machine create requests going at the same time or just a single one when this happened?

[vincent99]

They were separated by several minutes so it should've been a single request at a time

[deniseschannon]

Updated the main issue with this comment:

@cjellick looked into this. The bug is most likely related to the fact that after "Host Setup", there is a delay before Machine Service comes online. Therefore, if you try and create machine before Machine Service is online it will go to this state.

[deniseschannon]

@cjellick @will-chan Any update on when this might be fixed? Just ran into it again today.

If anything, is there a way to throw an error message if we try to add a host before go-machine-service is ready?

Alternatively, could we show the "host setup" page on the first time we launch Rancher? This would hopefully allow go-machine-service time to start before we start adding hosts.

[deniseschannon]

Minimally, this should be in an error state and not an active state.

[vincent99]

I tried that initially but both @ibuildthecloud and I dislike setup questions being the first thing you see...

[cjellick]

This seems to be happening more lately. Are more people using the feature or has something change?
Either way, seems like something we need to fix.

[cjellick]

Possible fix: adda a validation filter to the machine API that errors out if the appropriate external handler is not active for physical host.
We could also expose that as a link or action on physical host so that vince could block the API if it is not active. Possibly, we could dynamically remove POST as a resource method on machine if an external handler is not configured.
What say you @vincent99 and @ibuildthecloud?

[vincent99]

Knowing that it's not active is useful, but doesn't really help much.. roughly the first thing the user has to do is add a host, so if it's not active yet all I can really do is show them

Can we fix the reason it takes a while to startup in the first place? And have machines that are created that fail go to error instead of active?

[cjellick]

fixed by @hibooboo2. ping him with follow up Qs :-)

[deniseschannon]

Unable to reproduce with v0.59.1 so it seems to be fixed. If I see it again, I'll re-open.