Emergency shutdown race condition?

[olivierlambert]

From @pacohope on Disqus:

It got everything suspended, but the system didn't shutdown. (I think I didn't wait long enough. That will be obvious after you read the rest). So I logged in to XenServer as root via ssh. I did a vm-list and saw that all the VMs were suspended. I shutdown the XenServer and did my stuff.
When I started it back up, everything came up normally. Great. Within a few minutes of coming up, the entire system suspended all VMs and shutdown again. Somehow, in the XO VM, it does not know that the suspend-and-shutdown has happened. As soon as I start that VM, it suspends all the VMs and shuts down the dom0. So I'm going to delete that XO VM and install another XO VM. That's pretty easy and quick. I haven't lost any work. But this was REALLY annoying to troubleshoot. I'm not sure how XO records "I have been asked to suspend-and-shutdown". I suspect there is a race condition where my XO VM suspended before it recorded the fact that it had fulfilled my request to suspend-and-shutdown. I don't know how to repair the XO VM, so I won't. I'll just install a new copy.

[julien-f]

Could you summarize the issue?

Current algorithm:

1. iterate over all running VMs and suspend them
2. disable the host
3. shutdown the host

If I understand correctly I think it might be due to xo-lib automatically retrying a failed call on connection errors.
This is not the first time this behaviour has caused issues, I just pushed an update to disable it, hopefully it will fix more issues that it will create.

[olivierlambert]

Yes, it's related to shutdown a host with XOA running on it. It means:

• cutting the connection between xo-server and xo-web
• when XOA is back online, the browser reconnect to xo-server and re-send the order
• ... and it triggers this issue!

Your fix will solve this :)

[olivierlambert]

Fixed by vatesfr/xo-lib@2f29408