test_rebalance_raises_missing_data3[True] flaky on my dev box

[crusaderky]

test_scheduler::test_rebalance_raises_missing_data3[True] is highly flaky (~65% failure rate) on my hi-powered dev box. It seems to be stable on CI, likely due to the much slower CPU on the CI boxes.

    futures = await c.scatter(range(100), workers=[a.address])

    if explicit:
        keys = [f.key for f in futures]
        del futures
        out = await s.rebalance(keys=keys)
        assert out["status"] == "partial-fail"
        assert 1 <= len(out["keys"]) <= 100

There is clearly a race condition going on between the client->scheduler comms and the scheduler->worker comms.

@fjetter I recall you once showed me a way to block Scheduler.rpc on an event?