Commit
This commit does not belong to any branch on this repository, and may belong to a fork outside of the repository.
[SPARK-31360][WEBUI] Fix hung-up issue in StagePage
### What changes were proposed in this pull request? This change (SPARK-31360) fixes a hung-up issue in StagePage. StagePage will be hung-up with following operations. 1. Run a job with shuffle. `scala> sc.parallelize(1 to 10).map(x => (x, x)).reduceByKey(_ + _).collect` 2. Visit StagePage for the stage writing shuffle data and check `Shuffle Write Time`. <img width="401" alt="check-shuffle-write-time" src="https://user-images.githubusercontent.com/4736016/78557730-4513e200-784c-11ea-8b42-a5053b9489a5.png"> 3. Run a job with no shuffle. `scala> sc.parallelize(1 to 10).collect` 4. Visit StagePage for the last stage. <img width="956" alt="hungup" src="https://user-images.githubusercontent.com/4736016/78557746-4f35e080-784c-11ea-83e8-5db745b88535.png"> This issue is caused by following reason. In stagepage.js, an array `optionalColumns` has indices for columns for optional metrics. If a stage doesn't perform shuffle read or write, the corresponding indices are removed from the array. StagePage doesn't try to create column for such metrics, even if the state of corresponding optional metrics are preserved as "visible". But, if a stage doesn't perform both shuffle read and write, the index for `Shuffle Write Time` isn't removed. In that case, StagePage tries to create a column for `Shuffle Write Time` even though there are no metrics for shuffle write, leading hungup. ### Why are the changes needed? This is a bug. ### Does this PR introduce any user-facing change? No. ### How was this patch tested? I tested with operations I explained above and confirmed that StagePage won't be hung-up. Closes apache#28136 from sarutak/fix-ui-hungup. Authored-by: Kousuke Saruta <sarutak@oss.nttdata.com> Signed-off-by: Dongjoon Hyun <dongjoon@apache.org>
- Loading branch information