Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[STORM-2770] Add fragmentation metrics for CPU and Memory #2358

Merged
merged 4 commits into from
Oct 10, 2017
Merged

[STORM-2770] Add fragmentation metrics for CPU and Memory #2358

merged 4 commits into from
Oct 10, 2017

Conversation

kishorvpatil
Copy link
Contributor

The patch addresses following:

  • Calculate fragmentation of CPU/memory
  • Display it on UI under Cluster summary
  • Add following guages:
    -- fragmented_memory
    -- fragmented_cpu
    -- available_memory
    -- available_cpu
    -- total_memory
    -- total_cpu
  • Fix num_supervisors calculations.

storm2770-screenshot

HeartSaVioR
HeartSaVioR reviewed Oct 9, 2017
View reviewed changes
Copy link
Contributor

@HeartSaVioR HeartSaVioR left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

+1 once a nit is addressed.

storm-server/src/main/java/org/apache/storm/scheduler/Cluster.java
@@ -934,7 +942,7 @@ public void setStatusMap(Map<String, String> statusMap) {
public Map<String, SupervisorResources> getSupervisorsResourcesMap() {
Map<String, SupervisorResources> ret = new HashMap<>();
for (SupervisorDetails sd : supervisors.values()) {
ret.put(sd.getId(), new SupervisorResources(sd.getTotalMemory(), sd.getTotalMemory(), 0, 0));
ret.put(sd.getId(), new SupervisorResources(sd.getTotalMemory(), sd.getTotalCPU(), 0, 0));
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Nice catch.

storm-server/src/main/java/org/apache/storm/daemon/nimbus/Nimbus.java Outdated
@@ -1825,6 +1844,11 @@ private void mkAssignments(String scratchTopoId) throws Exception {

if (!newAssignments.equals(existingAssignments)) {
LOG.debug("RESETTING id->resources and id->worker-resources cache!");
LOG.info("Fragmentation after scheduling is: {} MB, {} PCore CPUs", fragmentedMemory(), fragmentedCpu());
//map.forEach((x, y) -> System.out.println(x + ": " + y));
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Let's remove the comment.

revans2
revans2 approved these changes Oct 10, 2017
View reviewed changes
Copy link
Contributor

@revans2 revans2 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

+1 looks good to me.

@asfgit asfgit merged commit 0b3930d into apache:master Oct 10, 2017
asfgit pushed a commit that referenced this pull request Oct 10, 2017
Merge branch 'storm2770' of https://github.com/kishorvpatil/incubator…
f867e3a 
…-storm into STORM-2770

STORM-2770: Add fragmentation metrics for CPU and Memory

This closes #2358
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@HeartSaVioR HeartSaVioR HeartSaVioR left review comments

@revans2 revans2 revans2 approved these changes

Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
4 participants
@kishorvpatil @HeartSaVioR @revans2 @asfgit