[#2411] fix(spark): Avoid double release memory when trigger spill

[summaryzb]

What changes were proposed in this pull request?

When trigger spill, return 0 instead of the memory number of sent block.

Why are the changes needed?

Fix #2411

Does this PR introduce any user-facing change?

No.

How was this patch tested?

UT

[github-actions]

Test Results

 2 945 files   -  66   2 945 suites   - 66   9h 43m 8s ⏱️ + 2h 58m 31s
 1 147 tests  -  30   1 142 ✅  -  34   1 💤 ±0   2 ❌ + 2   2 🔥 + 2 
14 666 runs   - 248  14 611 ✅  - 288  15 💤 ±0  20 ❌ +20  20 🔥 +20 

For more details on these failures and errors, see this check.

Results for commit 4d4a5d5. ± Comparison against base commit 3874f15.

This pull request removes 30 tests.

org.apache.spark.shuffle.DelegationRssShuffleManagerTest ‑ testCreateFallback
org.apache.spark.shuffle.DelegationRssShuffleManagerTest ‑ testCreateInDriver
org.apache.spark.shuffle.DelegationRssShuffleManagerTest ‑ testCreateInDriverDenied
org.apache.spark.shuffle.DelegationRssShuffleManagerTest ‑ testCreateInExecutor
org.apache.spark.shuffle.DelegationRssShuffleManagerTest ‑ testDefaultIncludeExcludeProperties
org.apache.spark.shuffle.DelegationRssShuffleManagerTest ‑ testExcludeProperties
org.apache.spark.shuffle.DelegationRssShuffleManagerTest ‑ testIncludeProperties
org.apache.spark.shuffle.DelegationRssShuffleManagerTest ‑ testTryAccessCluster
org.apache.spark.shuffle.FunctionUtilsTests ‑ testOnceFunction0
org.apache.spark.shuffle.RssShuffleManagerTest ‑ testCreateShuffleManagerServer
…

[zuston]

From my point, the return value is to tell spark memory manager we have released, that will not duplicate release.

[summaryzb]

I see, but does it really necessary to block here to calculate memory released. since the free memory is called in multiply callback thread

[zuston]

Sorry, I havn't fully gotten your thought. The returned free memory size is to tell spark the spilled size.

If you don't mind, please provide more details?

[summaryzb]

free memory is called in multiply callback thread

freeMemory of MemoryConsumer is called in multiply callback thread, but used in MemoryConsumer is a normal variable, There is a high probability that concurrent issue happens

[zuston]

Emm. I have to say that the spill only will be triggered by self.

if (!memorySpillEnabled || trigger != this) {
      return 0L;
    }