[Data] Skip recording memory spilled stats when get_memory_info_reply is failed

[c21]

Why are these changes needed?

User found the issue the call of get_memory_info_reply throws GRPC error if memory load on cluster is heavy. The error stack trace is:

  File "/home/ray/anaconda3/lib/python3.9/site-packages/ray/data/_internal/plan.py", line 675, in execute
    reply = get_memory_info_reply(
  File "/home/ray/anaconda3/lib/python3.9/site-packages/ray/_private/internal_api.py", line 82, in get_memory_info_reply
    reply = stub.FormatGlobalMemoryInfo(
  File "/home/ray/anaconda3/lib/python3.9/site-packages/grpc/_channel.py", line 1160, in __call__
    return _end_unary_response_blocking(state, call, False, None)
  File "/home/ray/anaconda3/lib/python3.9/site-packages/grpc/_channel.py", line 1003, in _end_unary_response_blocking
    raise _InactiveRpcError(state)  # pytype: disable=not-instantiable
grpc._channel._InactiveRpcError: <_InactiveRpcError of RPC that terminated with:
	status = StatusCode.DEADLINE_EXCEEDED
	details = "Deadline Exceeded"
	debug_error_string = "UNKNOWN:Error received from peer  {created_time:"...", grpc_status:4, grpc_message:"Deadline Exceeded"}"

This PR is to skip recording the memory spilled stats if the call didn't succeed. This stats reporting is not on critical path of job, so it can be skipped.

Related issue number

Checks

• I've signed off every commit(by using the -s flag, i.e., git commit -s) in this PR.
• I've run scripts/format.sh to lint the changes in this PR.
• I've included any doc changes needed for https://docs.ray.io/en/master/.
  • I've added any new APIs to the API Reference. For example, if I added a
    method in Tune, I've added it in doc/source/tune/api/ under the
    corresponding .rst file.
• I've made sure the tests are passing. Note that there might be a few flaky tests, see the recent failures at https://flakey-tests.ray.io/
• Testing Strategy
  • Unit tests
  • Release tests
  • This PR is not tested :(

[c21]

Did manual test locally to throw one arbitrary error in try code block, made sure the job succeeded.

[c21]

Will merge in the morning tomorrow if no more comments.