[SPARK-34589][CORE] Make BypassMergeSortShuffleWriter close partitionWriters in case of success

[dongjoon-hyun]

What changes were proposed in this pull request?

This PR aims to close partitionWriters when BypassMergeSortShuffleWriter.stop with success.

Why are the changes needed?

In case of failure, we call writer.revertPartialWritesAndClose and delete the files.
Likewise, we had better close the resources at this level.

Does this PR introduce any user-facing change?

No.

How was this patch tested?

Pass the existing UTs.

[dongjoon-hyun]

Hi, @mridulm and @Ngone51 .
How do you think about this?

[SparkQA]

Test build #135634 has started for PR 31704 at commit 79e6c5d.

[dongjoon-hyun]

Oops. Sorry guys. I realized that writer.write has try (DiskBlockObjectWriter writer = partitionWriters[I]). I'm closing this.

      writer.write(
        rdd.iterator(partition, context).asInstanceOf[Iterator[_ <: Product2[Any, Any]]])
      val mapStatus = writer.stop(success = true)

[SparkQA]

Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/40213/

[SparkQA]

Kubernetes integration test status failure
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/40213/