Could not start gRPC server flakiness in XLA tests

[suo]

For some examples, see here.

Can we add some retries or something to this test?

cc @bdhirsh

[suo]

cc @JackCaoG can you have someone take a look

[JackCaoG]

Hmm, I had pytorch/xla@935b602 which suppose to make this situation slightly better but it got reverted due to some CI issue. Let me try to reland this change.

[suo]

Another example from today on master https://github.com/pytorch/pytorch/runs/6615801243?check_suite_focus=true

[JackCaoG]

Hmm, I merge the fix pytorch/xla#3605 and pytorch pr #78327 but I see that now it is a different test trigger this failure.

I will do some further clean up

[JackCaoG]

pytorch/xla#3615 should help a bit more. Let me know if you still see this kind of error in pytorch CI.

[suo]

Another recent failure on master https://github.com/pytorch/pytorch/runs/6659602672?check_suite_focus=true

[suo]

another recent failure on master; this one is not Could not start gRPC server but also appears multiprocessing related https://github.com/pytorch/pytorch/runs/6676424724?check_suite_focus=true

[JackCaoG]

I think those failure are not recent, if they rebase their pytorch branch, mp test should not being run on CI.

[suo]

This failure was on the master branch :)

Maybe we didn't update the pin in a while?

[JackCaoG]

I think so, I did not see the call of run_mp_op_tests which should be in https://github.com/pytorch/xla/blob/master/test/run_tests.sh#L113. I think we update pin nightly?