[SPARK-13652][CORE] Copy ByteBuffer in sendRpcSync as it will be recy…

…cled

## What changes were proposed in this pull request?

`sendRpcSync` should copy the response content because the underlying buffer will be recycled and reused.

## How was this patch tested?

Jenkins unit tests.

Author: Shixiong Zhu <shixiong@databricks.com>

Closes #11499 from zsxwing/SPARK-13652.

(cherry picked from commit 465c665)
Signed-off-by: Shixiong Zhu <shixiong@databricks.com>

network/common/src/main/java/org/apache/spark/network/client/RpcResponseCallback.java

@@ -24,7 +24,12 @@
  * failure.
  */
 public interface RpcResponseCallback {
-  /** Successful serialized result from server. */
+  /**
+   * Successful serialized result from server.
+   *
+   * After `onSuccess` returns, `response` will be recycled and its content will become invalid.
+   * Please copy the content of `response` if you want to use it after `onSuccess` returns.
+   */
   void onSuccess(ByteBuffer response);
 
   /** Exception either propagated from server or raised on client side. */

network/common/src/main/java/org/apache/spark/network/client/TransportClient.java

@@ -257,7 +257,11 @@ public ByteBuffer sendRpcSync(ByteBuffer message, long timeoutMs) {
     sendRpc(message, new RpcResponseCallback() {
       @Override
       public void onSuccess(ByteBuffer response) {
-        result.set(response);
+        ByteBuffer copy = ByteBuffer.allocate(response.remaining());
+        copy.put(response);
+        // flip "copy" to make it readable
+        copy.flip();
+        result.set(copy);
       }
 
       @Override