Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[SPARK-8656][WebUI] Fix the webUI and JSON API number is not synced #7038

Closed
wants to merge 4 commits into from
Closed

[SPARK-8656][WebUI] Fix the webUI and JSON API number is not synced #7038

Show file tree
Hide file tree
Changes from 1 commit
Commits
Show all changes
4 commits
Select commit Hold shift + click to select a range
8b3b8e8
[SPARK-8656] Fix the webUI and JSON API number is not synced
Jun 26, 2015
431d2b0
Worker List should contain DEAD node also
Jun 26, 2015
2c8ea89
Change some styling and add local variable
Jul 3, 2015
9e54bf0
Change variable name to camel case
Jul 3, 2015
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Jump to
Jump to file
Failed to load files.
Diff view
Diff view
10 changes: 5 additions & 5 deletions core/src/main/scala/org/apache/spark/deploy/JsonProtocol.scala
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -77,11 +77,11 @@ private[deploy] object JsonProtocol {

def writeMasterState(obj: MasterStateResponse): JObject = {
("url" -> obj.uri) ~
("workers" -> obj.workers.toList.map(writeWorkerInfo)) ~
("cores" -> obj.workers.map(_.cores).sum) ~
("coresused" -> obj.workers.map(_.coresUsed).sum) ~
("memory" -> obj.workers.map(_.memory).sum) ~
("memoryused" -> obj.workers.map(_.memoryUsed).sum) ~
("workers" -> obj.workers.filter(_.isAlive()).toList.map(writeWorkerInfo)) ~
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Why do the filtering so many times?

Copy link
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I can refactor this better later. Let me find the API author first. Thanks

("cores" -> obj.workers.filter(_.isAlive()).map(_.cores).sum) ~
("coresused" -> obj.workers.filter(_.isAlive()).map(_.coresUsed).sum) ~
("memory" -> obj.workers.filter(_.isAlive()).map(_.memory).sum) ~
("memoryused" -> obj.workers.filter(_.isAlive()).map(_.memoryUsed).sum) ~
("activeapps" -> obj.activeApps.toList.map(writeApplicationInfo)) ~
("completedapps" -> obj.completedApps.toList.map(writeApplicationInfo)) ~
("activedrivers" -> obj.activeDrivers.toList.map(writeDriverInfo)) ~
Expand Down
4 changes: 4 additions & 0 deletions core/src/main/scala/org/apache/spark/deploy/master/WorkerInfo.scala
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -107,4 +107,8 @@ private[spark] class WorkerInfo(
def setState(state: WorkerState.Value): Unit = {
this.state = state
}

def isAlive(): Boolean={
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

No parens, return type or braces are needed. In fact is it worth writing a method?

Copy link
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The reason why I write a method here is due to JsonProtocol.scala cannot access WorkerState.ALIVE.

Changing the WorkerState from private to other permission is not good approach. I think it is the best way to keep the current class/namespace struct and get thing done.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

please add space around =

this.state == WorkerState.ALIVE
}
}