[#825][part-7] feat(spark): Write Stage resubmit and dynamic shuffle server assign integration tests.

[yl09099]

What changes were proposed in this pull request?

Write Stage resubmit and dynamic shuffle server assign integration tests.

Ⅰ. Overall objective:

1. During the shuffle write phase, the ShuffleServer reports faulty nodes and reallocates the ShuffleServer list;
2. Triggers a Stage level retry of SPARK. The shuffleServer node is excluded and reallocated before the retry.

Ⅱ. Implementation logic diagram:

Ⅲ. As shown in the picture above:

1. During Shuffle registration, obtain the ShuffleServer list to be written through the RPC interface of a Coordinator Client by following the solid blue line step. The list is bound using ShuffleID.
   2, the Task of Stage starts, solid steps, in accordance with the green by ShuffleManager Client RPC interface gets to be written for shuffleIdToShuffleHandleInfo ShuffleServer list;
2. In the Stage, if Task fails to write blocks to the ShuffleServer, press the steps in red to report ShuffleServer to FailedShuffleServerList in RSSShuffleManager through the RPC interface.
3. FailedShuffleServerList records the number of ShuffleServer failures. After the number of failures reaches the maximum number of retries of the Task level, follow the steps in dotted orange lines. Through the RPC interface of a Coordinator Client, obtain the list of ShuffleServer files to be written (the ShuffleServer files that fail to be written are excluded). After obtaining the list, go to Step 5 of the dotted orange line. Throwing a FetchFailed Exception triggers a stage-level retry for SPARK;
4. Attempt 1 is generated by the SPARK Stage level again. Pull the corresponding ShuffleServer list according to the green dotted line.

Why are the changes needed?

Fix: #825

Does this PR introduce any user-facing change?

No.

How was this patch tested?

UT.

[codecov-commenter]

Codecov Report

All modified and coverable lines are covered by tests ✅

Comparison is base (d5de299) 53.35% compared to head (348b270) 54.48%.
Report is 1 commits behind head on master.

Additional details and impacted files

@@             Coverage Diff              @@
##             master    #1148      +/-   ##
============================================
+ Coverage     53.35%   54.48%   +1.13%     
- Complexity     2690     2696       +6     
============================================
  Files           410      391      -19     
  Lines         23590    21280    -2310     
  Branches       2003     2006       +3     
============================================
- Hits          12587    11595     -992     
+ Misses        10229     8980    -1249     
+ Partials        774      705      -69     

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

[yl09099]

@jerqi @leixm Look at it for me when you have time.

[leixm]

Could we reuse MockedShuffleServerGrpcService?

[roryqi]

Could you resolve this comment #1147 (comment) in this pr?

[yl09099]

Could you resolve this comment #1147 (comment) in this pr?

I will submit this PR separately, and this integration test is a test of the above 6 PR, which is one.

[leixm]

LGTM, @jerqi please take a look.

[roryqi]

LGTM. thanks @leixm @yl09099