XLA tests are flaky on PyTorch master

[suo]

Hey, I've noticed an uptick on flaky XLA test failures on PyTorch master. Some recent examples:

• https://github.com/pytorch/pytorch/runs/6097653892?check_suite_focus=true
• https://github.com/pytorch/pytorch/runs/6041716960?check_suite_focus=true
• https://github.com/pytorch/pytorch/runs/6041275327?check_suite_focus=true
• https://github.com/pytorch/pytorch/runs/6041716960?check_suite_focus=true

Can folks take a look? Our typical remediation in PyTorch core is to disable flaky tests until they can be fixed, but we can't do that for these tests without disabling the whole workflow :/

[JackCaoG]

Thanks I will take a look, seems like error is related to start the grpc server

2022-04-20T15:59:01.9430510Z E0420 15:59:01.942391440   23139 server_chttp2.cc:40]        {"created":"@1650470341.942356286","description":"No address added out of total 1 resolved","file":"external/com_github_grpc_grpc/src/core/ext/transport/chttp2/server/chttp2_server.cc","file_line":395,"referenced_errors":[{"created":"@1650470341.942354533","description":"Failed to add any wildcard listeners","file":"external/com_github_grpc_grpc/src/core/lib/iomgr/tcp_server_posix.cc","file_line":342,"referenced_errors":[{"created":"@1650470341.942336692","description":"Address family not supported by protocol","errno":97,"file":"external/com_github_grpc_grpc/src/core/lib/iomgr/socket_utils_common_posix.cc","file_line":420,"os_error":"Address family not supported by protocol","syscall":"socket","target_address":"[::]:37179"},{"created":"@1650470341.942353922","description":"Unable to configure socket","fd":30,"file":"external/com_github_grpc_grpc/src/core/lib/iomgr/tcp_server_utils_posix_common.cc","file_line":216,"referenced_errors":[{"created":"@1650470341.942351233","description":"Address already in use","errno":98,"file":"external/com_github_grpc_grpc/src/core/lib/iomgr/tcp_server_utils_posix_common.cc","file_line":189,"os_error":"Address already in use","syscall":"bind"}]}]}]}

[JackCaoG]

All of the mp test should not get run on CPU anyway, I think what happened is it will still try to start a new server which in some time failed due to the address issue. Let me try to explicitly disable the test on CPU.

[JackCaoG]

another option is we don't really need to keep restarting xrt server in our test, we can use a long running one. Let me try this idea first.