[tune] Deflake PBT Async test

[krfricke]

Why are these changes needed?

The PopulationBasedTrainingSynchTest:testAsynchFail test is currently flaky.

The test ensures that in asynchronous training, at least some trials do not exploit the best performing trial due to asynchronous evaluation. However, for certain orders this might randomly still be the case - e.g. if the worst trial exploits the best trial early, and the second trial then does the same afterwards.

By introducing different sleep times, we make sure that bad performing trials are evaluated first, continue training earlier, and finish earlier without exploiting well performing trials. In synchronous mode, this should not happen as PBT will wait for all results to arrive, first - exactly what we want to test here.

Related issue number

Closes #15730

Checks

• I've run scripts/format.sh to lint the changes in this PR.
• I've included any doc changes needed for https://docs.ray.io/en/master/.
• I've made sure the tests are passing. Note that there might be a few flaky tests, see the recent failures at https://flakey-tests.ray.io/
• Testing Strategy
  • Unit tests
  • Release tests
  • This PR is not tested :(

[amogkam]

Awesome, thanks!