[SPARK-31360][WEBUI] Fix hung-up issue in StagePage

[sarutak]

What changes were proposed in this pull request?

This change (SPARK-31360) fixes a hung-up issue in StagePage.
StagePage will be hung-up with following operations.

1. Run a job with shuffle.
   scala> sc.parallelize(1 to 10).map(x => (x, x)).reduceByKey(_ + _).collect

2. Visit StagePage for the stage writing shuffle data and check Shuffle Write Time.

3. Run a job with no shuffle.
   scala> sc.parallelize(1 to 10).collect

4. Visit StagePage for the last stage.

This issue is caused by following reason.

In stagepage.js, an array optionalColumns has indices for columns for optional metrics.
If a stage doesn't perform shuffle read or write, the corresponding indices are removed from the array.
StagePage doesn't try to create column for such metrics, even if the state of corresponding optional metrics are preserved as "visible".
But, if a stage doesn't perform both shuffle read and write, the index for Shuffle Write Time isn't removed.
In that case, StagePage tries to create a column for Shuffle Write Time even though there are no metrics for shuffle write, leading hungup.

Why are the changes needed?

This is a bug.

Does this PR introduce any user-facing change?

No.

How was this patch tested?

I tested with operations I explained above and confirmed that StagePage won't be hung-up.

[SparkQA]

Test build #120871 has finished for PR 28136 at commit fb89c37.

• This patch passes all tests.
• This patch merges cleanly.
• This patch adds no public classes.

[dongjoon-hyun]

+1, LGTM. Thank you, @sarutak and @srowen .
I also verified the bug. Merged to master.