Commit
This commit does not belong to any branch on this repository, and may belong to a fork outside of the repository.
[SPARK-32823][WEB UI] Fix the master ui resources reporting
### What changes were proposed in this pull request? Fixes the master UI for properly summing the resources total across multiple workers. field: Resources in use: 0 / 8 gpu The bug here is that it was creating MutableResourceInfo and then reducing using the + operator. the + operator in MutableResourceInfo simple adds the address from one to the addresses of the other. But its using a HashSet so if the addresses are the same then you lose the correct amount. ie worker1 has gpu addresses 0,1,2,3 and worker2 has addresses 0,1,2,3 then you only see 4 total GPUs when there are 8. In this case we don't really need to create the MutableResourceInfo at all because we just want the sums for used and total so just remove the use of it. The other uses of it are per Worker so those should be ok. ### Why are the changes needed? fix UI ### Does this PR introduce _any_ user-facing change? UI ### How was this patch tested? tested manually on standalone cluster with multiple workers and multiple GPUs and multiple fpgas Closes apache#29683 from tgravescs/SPARK-32823. Lead-authored-by: Thomas Graves <tgraves@nvidia.com> Co-authored-by: Thomas Graves <tgraves@apache.org> Signed-off-by: HyukjinKwon <gurwls223@apache.org>
- Loading branch information