Deleting a host causes Rancher to retry purge requests forever

[sheng-liang]

Rancher Version: v1.1.0-dev2-rc5

Docker Version: 1.10.3

OS and where are the hosts located? (cloud, bare metal, etc): Ubuntu 14.04, DigitalOcean

Setup Details: (single node rancher vs. HA rancher, internal DB vs. external DB) single node Rancher

Environment Type: (Cattle/Kubernetes/Swarm/Mesos) Cattle

Steps to Reproduce: Delete hosts in Cattle environment directly from DO (not from Rancher), Rancher shows hosts in reconnect state. Then deactivate the hosts in Rancher, and delete the host. Later on I also deleted the stack that was running.

Results: Rancher retries purge requests forever

Expected: Rancher should give up after a while

[alena1108]

Debugged the issue; volume.purge gets stuck on volumestoragepoolmap.remove process. volumestoragepoolmap.remove triggers agent event that times out forever in this case. It happens because host's agent doesn't get removed along with the host removal. So while host is in removed state, the agent is still being active. That causes this bug.

Seems like state transition from reconnecting to removed is missing for agent.remove process, enabling should fix the problem.

[soumyalj]

Tested on master with the above steps. Did not see volumes stuck in purge state.