Fix flaky distributed array test

[stephanie-wang]

Why are these changes needed?

test_array.py::test_distributed_array_methods has been timing out occasionally. This happens when plasma store notifications to the raylet for new objects get delayed. If the delay is long enough, then this causes concurrent fetch calls to timeout and mistakenly mark the object as lost. This PR sets the timeout very high in this particular test to prevent the spurious error. Confirmed locally that this fixes the flaky test.

It's unclear exactly when this has been coming up. It seems more likely to happen if the plasma store or raylet have more load, since then the notifications will not get processed by the raylet quickly enough. The errors may have been exacerbated by #7628 but I'm not sure why.

Checks

• I've run scripts/format.sh to lint the changes in this PR.
• I've included any doc changes needed for https://ray.readthedocs.io/en/latest/.
• I've made sure the tests are passing. Note that there might be a few flaky tests, see the recent failure rates at https://ray-travis-tracker.herokuapp.com/.

[edoakes]

Interesting. Curious why it only seems to come up on this specific test. Were you able to reproduce it locally?

[stephanie-wang]

Interesting. Curious why it only seems to come up on this specific test. Were you able to reproduce it locally?

Yes, it flaked out in <10 runs when run locally with all of the processes pinned to 1 core. I'd guess that it comes up more in this test because it uses plasma and distributed references heavily. Which makes it a good test, I suppose :)

[AmplabJenkins]

Can one of the admins verify this patch?

[AmplabJenkins]

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/Ray-PRB/23591/
Test FAILed.