Nomad didn't deregister services in Consul

[adrianlop]

Hi there,

I have Consul 0.6.3 and Nomad 0.2.3 agents running as a system service.
When I rebooted some machines, the services running on these machines reallocated to other available machines. That worked like a charm.
The problem was that the services were still registered (but not running, I double-checked that) on the old machines, and in the new ones.
I checked Nomad logs in the old machines and there's no reference about the allocID (even after restarting the Nomad agent too).
So my guess is that when the reboot happened, Nomad cluster saw that the machines were unreachable, so it reallocated the services, but when they came up again, the Nomad agents didn't send a query to Consul to deregister the services.

Can you please help? is this fix in Nomad 0.3 maybe?

[diptanu]

@adrianlop Can you share the Nomad client logs of the machines where the services were supposed to be not registered?

[adrianlop]

Ok, I'll rerun the reboots. do you prefer them on DEBUG?

[diptanu]

Yes, please share the debug logs.

Sent from my iPhone

On Feb 17, 2016, at 9:21 AM, Adrián López notifications@github.com wrote:

Ok, I'll rerun the reboots. do you prefer them on DEBUG?

—
Reply to this email directly or view it on GitHub.

[adrianlop]

now that you released v0.3, I'm going to upgrade the whole cluster first, then I'll try to reproduce this again. thanks!

[adrianlop]

hi again,
after upgrading to Nomad 0.3, I rebooted several servers and everything went OK.

I think what happened is:

• We are affected by a Docker bug (see Sometimes a docker container gets stuck so that it can't be stopped or removed moby/moby#10589 and Docker 1.9.1 hanging at build step "Setting up ca-certificates-java" moby/moby#18180) which causes zombie processes that cannot be killed (so you can't finish stopping the container neither).
• Nomad tries to stop and deregister the allocation from Nomad so it can be balanced to another node, and also deregister service in Consul
• Since the bug is about a reckless process that cannot be killed, Nomad doesn't finish deregistering it in Consul because it couldn't stop it, right???

So I'm going to close the issue for now, if I reboot machines some other times, and there are no zombie Dockers, then I'll reopen and send you debug logs.

Thanks @diptanu !

[github-actions]

I'm going to lock this issue because it has been closed for 120 days ⏳. This helps our maintainers find and focus on the active issues.
If you have found a problem that seems similar to this, please open a new issue and complete the issue template so we can capture all the details necessary to investigate further.