Fix tests to continue trying to claim after a nil response to claim

[eric]

Right now the tests assume that claim will never return nil if there are any tasks still available to claim.

The reality is (this was explained by @aphyr in #80) that with an AP queue it is expected behavior that sometimes claim will return nil even though there is theoretically work still to be done.

Because of that, we need to update the tests to continue trying to claim until some period of time has passed instead of relying on a nil response.

[aphyr]

Ugh, which is itself a race condition. Jeez, there is just NO good way out of this, is there?

[eric]

Yeah. I'm not actually sold on the idea that during topology changes claim should (is "can often" a better way to phrase it?) return nil — I feel like the system knows when these changes are happening and should at least hold off on responding to the client until it's stable again (if possible).

I mean, maybe the answer is that with a solution to #81 (hopefully one where there is no interruption in a leader holding their leadership) that it'll be good enough for the tests (other than any tests that are specifically testing behavior during a leader that quits responding).

[aphyr]

Scan is O(n), so I suggest not doing that. Letting it block for up to the request timeout duration is OK though!

[eric]

To begin with, I'm going to try to figure out the best way to add a retry until a deadline in stress-test:

    (let [claims (loop [claims {}]
                   (if-let [t (client/claim! *client* 100000)]
                     (do
                       ; Make sure we never double-claim
                       (assert (not (get claims (:id t))))
                       (let [claims (assoc claims (:id t) t)]
                         (if (= (count ids) (count claims))
                           claims
                           (recur claims))))
                     ; Out of claims?
                     (do
                       claims)))]

[eric]

This has been fixed.