A solution to the problem of racing with the CI to claim a device that will make tests fail

[a1291762]

Because of the way our CI grabs devices, we currently need a bit of a dance to Claim a device in order to fix issues that are breaking test runs.

1. Remove it from the pool
2. Wait for CI to Return it
3. Claim it
4. Put it back in the pool
5. ...
6. Return it

The reason for this is that unless there are no pending jobs, the CI will be trying to find machines to get and moving devices out of the pool is the only way to ensure the CI does not race you to grab them.

It would simplify things if we could "Claim Next" such that when the CI releases the device, it is automatically given to us (pre-empting any racing). It would remove the need for steps 3 and 4.

1. Click Claim Next
2. Wait for CI to Return it (becomes ours)
3. ...
4. Return it

Alternative

The main issue with the above is the extra state that’s needed… A possibly simpler solution would be a “Take Over” button that does the same as a Return/Claim but atomically (preventing races). This would work because no new jobs could take the machine and when the old job attempts to Return the machine, devicecheckout will just ignore it (since the claim id won’t match?)

1. Click Take Over
2. Don’t have to wait for CI to Return it
3. …
4. Return it