[SPARK-8656][WebUI] Fix the webUI and JSON API number is not synced #7038
Conversation
|
Can one of the admins verify this patch? |
|
This is changing the semantics of the response though. I don't think you can necessarily do this. It is not necessarily true that what is presented in the web UI for human consumption is identical to what's in the JSON response for machine consumption. You should ask the person who created this API to see if there is or should be a way to get just alive workers status. Is this info not in the response? |
| ("coresused" -> obj.workers.map(_.coresUsed).sum) ~ | ||
| ("memory" -> obj.workers.map(_.memory).sum) ~ | ||
| ("memoryused" -> obj.workers.map(_.memoryUsed).sum) ~ | ||
| ("workers" -> obj.workers.filter(_.isAlive()).toList.map(writeWorkerInfo)) ~ |
There was a problem hiding this comment.
Why do the filtering so many times?
There was a problem hiding this comment.
I can refactor this better later. Let me find the API author first. Thanks
|
HI Sean, thanks for your opinion. My feeling is that WebUI and API is just different presentation format. But the content should be the same. Different meaning in the same column(workers,memory,core) will confuse the user. BTW, Let me talk to API author first :) |
|
Base on git blame, I think the JSON API creator is @rxin ( 9db1e50 ) Hi Reynold #6317 has already change some master UI's data's definition (total memory/memory in used , total core/core in used ) . How do you think? |
|
ok to test |
|
I think we should have the JSON protocol do whatever the normal UI does. This change looks fine. |
|
@thegiive have you verified that these are the only sources of mismatch? |
|
Merged build triggered. |
|
Merged build started. |
|
Test build #36030 has started for PR 7038 at commit |
|
Test build #36030 has finished for PR 7038 at commit
|
|
Merged build finished. Test PASSed. |
|
what's the non-compatible part? |
|
I think just that the response will have different information now -- are people relying on and properly expecting to get info for all workers in the JSON? then removing non-alive worker info is incompatible in that sense. |
|
ic - to me it is almost incorrect in the past, and we are fixing a bug here ... |
|
Merged build triggered. |
|
Merged build started. |
|
Hi all reviewer I modified the code base on the comment. |
|
Test build #36464 has started for PR 7038 at commit |
|
Test build #36464 has finished for PR 7038 at commit
|
|
Merged build finished. Test FAILed. |
| @@ -76,12 +76,13 @@ private[deploy] object JsonProtocol { | |||
| } | |||
|
|
|||
| def writeMasterState(obj: MasterStateResponse): JObject = { | |||
| val alive_workers = obj.workers.filter(_.isAlive()) | |||
There was a problem hiding this comment.
in Spark we generally use camel case, so:
val aliveWorkers = obj.workers.filter(_.state == WorkerState.ALIVE)
There was a problem hiding this comment.
Hi Andrew
WorkerState is org.apache.spark.deploy.master's private object. JsonProtocol cannot resolved WorkerState due to class visibility.This code cannot work unless we change WorkerState code.
I think best way to do that is add a one line method on WorkerInfo
How do you think?
There was a problem hiding this comment.
BTW, I will change the variable name to camel case without any problem
|
Merged build triggered. |
|
Merged build started. |
|
Test build #36515 has started for PR 7038 at commit |
|
Test build #36515 has finished for PR 7038 at commit
|
|
Merged build finished. Test PASSed. |
|
HI @andrewor14 The test is passed. If there is any problem I need to change, please let me know. Thanks |
|
LGTM |
|
Ok, I'm merging this into master. Thanks @thegiive. |
Spark standalone master web UI show "Alive Workers" total core, total used cores and "Alive workers" total memory, memory used.
But the JSON API page "http://MASTERURL:8088/json" shows "ALL workers" core, memory number.
This webUI data is not sync with the JSON API.
The proper way is to sync the number with webUI and JSON API.