Skip to content

Commit

Permalink
[SPARK-31360][WEBUI] Fix hung-up issue in StagePage
Browse files Browse the repository at this point in the history
### What changes were proposed in this pull request?

This change (SPARK-31360) fixes a hung-up issue in StagePage.
StagePage will be hung-up with following operations.

1. Run a job with shuffle.
`scala> sc.parallelize(1 to 10).map(x => (x, x)).reduceByKey(_ + _).collect`

2. Visit StagePage for the stage writing shuffle data and check `Shuffle Write Time`.
<img width="401" alt="check-shuffle-write-time" src="https://user-images.githubusercontent.com/4736016/78557730-4513e200-784c-11ea-8b42-a5053b9489a5.png">

3. Run a job with no shuffle.
`scala> sc.parallelize(1 to 10).collect`

4. Visit StagePage for the last stage.
<img width="956" alt="hungup" src="https://user-images.githubusercontent.com/4736016/78557746-4f35e080-784c-11ea-83e8-5db745b88535.png">

This issue is caused by following reason.

In stagepage.js, an array `optionalColumns` has indices for columns for optional metrics.
If a stage doesn't perform shuffle read or write, the corresponding indices are removed from the array.
StagePage doesn't try to create column for such metrics, even if the state of corresponding optional metrics are preserved as "visible".
But, if a stage doesn't perform both shuffle read and write, the index for `Shuffle Write Time` isn't removed.
In that case, StagePage tries to create a column for `Shuffle Write Time` even though there are no metrics for shuffle write, leading hungup.

### Why are the changes needed?

This is a bug.

### Does this PR introduce any user-facing change?

No.

### How was this patch tested?

I tested with operations I explained above and confirmed that StagePage won't be hung-up.

Closes apache#28136 from sarutak/fix-ui-hungup.

Authored-by: Kousuke Saruta <sarutak@oss.nttdata.com>
Signed-off-by: Dongjoon Hyun <dongjoon@apache.org>
  • Loading branch information
sarutak authored and Seongjin Cho committed Apr 14, 2020
1 parent 270997b commit 790a157
Showing 1 changed file with 11 additions and 2 deletions.
13 changes: 11 additions & 2 deletions core/src/main/resources/org/apache/spark/ui/static/stagepage.js
Original file line number Diff line number Diff line change
Expand Up @@ -363,15 +363,24 @@ $(document).ready(function () {
dataToShow.showBytesSpilledData =
(responseBody.diskBytesSpilled > 0 || responseBody.memoryBytesSpilled > 0);

var columnIndicesToRemove = [];
if (!dataToShow.showShuffleReadData) {
$('#shuffle_read_blocked_time').remove();
$('#shuffle_remote_reads').remove();
optionalColumns.splice(2, 2);
columnIndicesToRemove.push(2);
columnIndicesToRemove.push(3);
}

if (!dataToShow.showShuffleWriteData) {
$('#shuffle_write_time').remove();
optionalColumns.splice(7, 1)
columnIndicesToRemove.push(7);
}

if (columnIndicesToRemove.length > 0) {
columnIndicesToRemove.sort(function(a, b) { return b - a; });
columnIndicesToRemove.forEach(function(idx) {
optionalColumns.splice(idx, 1);
});
}

// prepare data for executor summary table
Expand Down

0 comments on commit 790a157

Please sign in to comment.