Duffy can't return node when $target node is unreachable

[arrfab]

By default, when someone requests a node, duffy does a mysql query to find which nodes are available to be "given" to projects (https://github.com/CentOS/duffy/blob/master/duffy/api_v1/views.py#L62)
problem with that query is that it will always try the same node first, because of the "order by used_count"

So when that node is unreachable (the case we had today), duffy can't contextualize it (https://github.com/CentOS/duffy/blob/master/duffy/api_v1/views.py#L76) and so answer "Failed to allocate nodes" (https://github.com/CentOS/duffy/blob/master/duffy/api_v1/views.py#L90)

We should enhance (https://github.com/CentOS/duffy/blob/master/duffy/models/nodes.py#L56) to be sure that if we can't reach a node, it's just skipped and be marked to be reinstalled, so that it's removed from "Ready" status

So probably switching that line ? https://github.com/CentOS/duffy/blob/master/duffy/models/nodes.py#L72

Let's discuss this

[kbsingh]

at the very first attempt, it should mark the node as 'Failed' and move along - the node must not remain in the db as 'Ready'.

[bstinsonmhk]

We might want to put the host back to 'Active' instead. The common case is that we had a flake with a Seamicro machine and a single reinstall will fix it.

If the machine goes back to Active and something is more seriously wrong, the install workers will mark it failed.

[arrfab]

Yes, +1 on my side : putting it to "Active" so that a reinstall would be tested