Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[flant-integration] fix resources counting and add tests #8291

Merged
merged 2 commits into from
May 2, 2024
Merged

[flant-integration] fix resources counting and add tests #8291

merged 2 commits into from
May 2, 2024

Conversation

RomanenkoDenys
Copy link
Member

@RomanenkoDenys RomanenkoDenys commented May 2, 2024 •
edited by z9r5

Description

Fix count resources for the different nodegroup roles.
Added tests.

Why do we need it, and what problem does it solve?

Wrong counting of the worker nodegroups resources.

Why do we need it in the patch release (if we do)?

What is the expected result?

Checklist

  • The code is covered by unit tests.
  • e2e tests passed.
  • Documentation updated according to the changes.
  • Changes were tested in the Kubernetes cluster manually.

Changelog entries

section: flant-integration
type: fix
summary: Fix resources counting.
impact_level: default

@RomanenkoDenys
[flant-integration] fix resources counting and add tests
39deb88 
Signed-off-by: Denis Romanenko <denis.romanenko@flant.com>
@RomanenkoDenys RomanenkoDenys added the area/monitoring Pull requests that update monitoring modules label May 2, 2024
@RomanenkoDenys RomanenkoDenys added this to the v1.59.9 milestone May 2, 2024
@RomanenkoDenys RomanenkoDenys self-assigned this May 2, 2024
@github-actions github-actions bot added python Pull requests that update Python code and removed area/monitoring Pull requests that update monitoring modules labels May 2, 2024
@RomanenkoDenys
Copy link
Member Author 

ks-master-0:~# kubectl get nodes -o json | jq -r '.items[] | {"name": .metadata.name, "capacity": {"cpu" :.status.capacity.cpu, "memory": .status.capacity.memory}}'
{
  "name": "ks-application-nodes-de5604c3-7bf59-6rq5b",
  "capacity": {
    "cpu": "8",
    "memory": "16351048Ki"
  }
}
{
  "name": "ks-application-nodes-de5604c3-7bf59-7tvqg",
  "capacity": {
    "cpu": "8",
    "memory": "16351056Ki"
  }
}
{
  "name": "ks-application-nodes-de5604c3-7bf59-bmh8q",
  "capacity": {
    "cpu": "8",
    "memory": "16351056Ki"
  }
}
{
  "name": "ks-application-nodes-de5604c3-7bf59-f4wdg",
  "capacity": {
    "cpu": "8",
    "memory": "16351056Ki"
  }
}
{
  "name": "ks-application-nodes-de5604c3-7bf59-fn7cs",
  "capacity": {
    "cpu": "8",
    "memory": "16351056Ki"
  }
}
{
  "name": "ks-application-nodes-de5604c3-7bf59-g6mkz",
  "capacity": {
    "cpu": "8",
    "memory": "16351056Ki"
  }
}
{
  "name": "ks-application-nodes-de5604c3-7bf59-jh2kx",
  "capacity": {
    "cpu": "8",
    "memory": "16351056Ki"
  }
}
{
  "name": "ks-application-nodes-de5604c3-7bf59-k5t6q",
  "capacity": {
    "cpu": "8",
    "memory": "16351056Ki"
  }
}
{
  "name": "ks-application-nodes-de5604c3-7bf59-k6hwq",
  "capacity": {
    "cpu": "8",
    "memory": "16351056Ki"
  }
}
{
  "name": "ks-application-nodes-de5604c3-7bf59-lgmxw",
  "capacity": {
    "cpu": "8",
    "memory": "16351056Ki"
  }
}
{
  "name": "ks-application-nodes-de5604c3-7bf59-mc5z8",
  "capacity": {
    "cpu": "8",
    "memory": "16351048Ki"
  }
}
{
  "name": "ks-application-nodes-de5604c3-7bf59-njgbd",
  "capacity": {
    "cpu": "8",
    "memory": "16351056Ki"
  }
}
{
  "name": "ks-application-nodes-de5604c3-7bf59-nv2wj",
  "capacity": {
    "cpu": "8",
    "memory": "16351056Ki"
  }
}
{
  "name": "ks-application-nodes-de5604c3-7bf59-sghm8",
  "capacity": {
    "cpu": "8",
    "memory": "16351056Ki"
  }
}
{
  "name": "ks-application-nodes-de5604c3-7bf59-sh5t8",
  "capacity": {
    "cpu": "8",
    "memory": "16351056Ki"
  }
}
{
  "name": "ks-application-nodes-de5604c3-7bf59-tv56j",
  "capacity": {
    "cpu": "8",
    "memory": "16351048Ki"
  }
}
{
  "name": "ks-application-nodes-de5604c3-7bf59-tzlbj",
  "capacity": {
    "cpu": "8",
    "memory": "16351056Ki"
  }
}
{
  "name": "ks-application-nodes-de5604c3-7bf59-vdfnr",
  "capacity": {
    "cpu": "8",
    "memory": "16351056Ki"
  }
}
{
  "name": "ks-application-nodes-de5604c3-7bf59-vfdw4",
  "capacity": {
    "cpu": "8",
    "memory": "16351056Ki"
  }
}
{
  "name": "ks-application-nodes-de5604c3-7bf59-vlv84",
  "capacity": {
    "cpu": "8",
    "memory": "16351056Ki"
  }
}
{
  "name": "ks-application-nodes-de5604c3-7bf59-xt59d",
  "capacity": {
    "cpu": "8",
    "memory": "16351056Ki"
  }
}
{
  "name": "ks-application-nodes-de5604c3-7bf59-xth4w",
  "capacity": {
    "cpu": "8",
    "memory": "16351056Ki"
  }
}
{
  "name": "ks-d8-loki-de5604c3-5c5c7-cz7ll",
  "capacity": {
    "cpu": "4",
    "memory": "8105804Ki"
  }
}
{
  "name": "ks-dam-de5604c3-59dcb-dtxw5",
  "capacity": {
    "cpu": "16",
    "memory": "3983184Ki"
  }
}
{
  "name": "ks-dam-de5604c3-59dcb-qgsxr",
  "capacity": {
    "cpu": "16",
    "memory": "3983192Ki"
  }
}
{
  "name": "ks-dam-de5604c3-59dcb-qs6gc",
  "capacity": {
    "cpu": "16",
    "memory": "3983184Ki"
  }
}
{
  "name": "ks-dam-de5604c3-59dcb-vrgt2",
  "capacity": {
    "cpu": "16",
    "memory": "3983192Ki"
  }
}
{
  "name": "ks-dam-de5604c3-59dcb-w7cm4",
  "capacity": {
    "cpu": "16",
    "memory": "3983184Ki"
  }
}
{
  "name": "ks-loadbalancer-de5604c3-7df9c-m72dk",
  "capacity": {
    "cpu": "2",
    "memory": "4001708Ki"
  }
}
{
  "name": "ks-loadbalancer-de5604c3-7df9c-tbfb5",
  "capacity": {
    "cpu": "2",
    "memory": "4001700Ki"
  }
}
{
  "name": "ks-master-0",
  "capacity": {
    "cpu": "8",
    "memory": "16384592Ki"
  }
}
{
  "name": "ks-odfe-common-de5604c3-55dcd-b4z7q",
  "capacity": {
    "cpu": "4",
    "memory": "8105812Ki"
  }
}
{
  "name": "ks-odfe-common-de5604c3-55dcd-b9kv2",
  "capacity": {
    "cpu": "4",
    "memory": "8105804Ki"
  }
}
{
  "name": "ks-odfe-common-de5604c3-55dcd-dq2nw",
  "capacity": {
    "cpu": "4",
    "memory": "8105804Ki"
  }
}
{
  "name": "ks-opendistro-de5604c3-6f9dc-6z49x",
  "capacity": {
    "cpu": "2",
    "memory": "10157908Ki"
  }
}
{
  "name": "ks-opendistro-de5604c3-6f9dc-dcnkf",
  "capacity": {
    "cpu": "2",
    "memory": "10157900Ki"
  }
}
{
  "name": "ks-opendistro-de5604c3-6f9dc-fjqrv",
  "capacity": {
    "cpu": "2",
    "memory": "10157908Ki"
  }
}
{
  "name": "ks-opendistro-de5604c3-6f9dc-m9q8t",
  "capacity": {
    "cpu": "2",
    "memory": "10157908Ki"
  }
}
{
  "name": "ks-router-0",
  "capacity": {
    "cpu": "2",
    "memory": "4030560Ki"
  }
}
{
  "name": "ks-system-de5604c3-b6f5f-rn7zh",
  "capacity": {
    "cpu": "4",
    "memory": "16351056Ki"
  }
}
{
  "name": "ks-system-de5604c3-b6f5f-zk478",
  "capacity": {
    "cpu": "4",
    "memory": "16351056Ki"
  }
}

@RomanenkoDenys
Copy link
Member Author 

kubectl get ng -o json | jq -r '.items[] | {"name": .metadata.name, "nodeTemplate": .spec.nodeTemplate}'
{
  "name": "application-nodes",
  "nodeTemplate": {
    "labels": {
      "node-role/application": ""
    }
  }
}
{
  "name": "d8-loki",
  "nodeTemplate": {
    "labels": {
      "node-role/d8-loki": ""
    },
    "taints": [
      {
        "effect": "NoExecute",
        "key": "dedicated",
        "value": "d8-loki"
      }
    ]
  }
}
{
  "name": "dam",
  "nodeTemplate": {
    "labels": {
      "node-role/application": "",
      "node-role/dam": ""
    },
    "taints": [
      {
        "effect": "NoExecute",
        "key": "dedicated",
        "value": "dam"
      }
    ]
  }
}
{
  "name": "loadbalancer",
  "nodeTemplate": {
    "labels": {
      "node-role/loadbalancer": ""
    },
    "taints": [
      {
        "effect": "NoExecute",
        "key": "node-role/loadbalancer",
        "value": ""
      }
    ]
  }
}
{
  "name": "master",
  "nodeTemplate": {
    "labels": {
      "node-role.kubernetes.io/control-plane": "",
      "node-role.kubernetes.io/master": ""
    },
    "taints": [
      {
        "effect": "NoSchedule",
        "key": "node-role.kubernetes.io/master"
      },
      {
        "effect": "NoSchedule",
        "key": "node-role.kubernetes.io/control-plane"
      }
    ]
  }
}
{
  "name": "odfe-common",
  "nodeTemplate": {
    "labels": {
      "node-role/opendistro-common": ""
    },
    "taints": [
      {
        "effect": "NoSchedule",
        "key": "dedicated",
        "value": "opendistro-common"
      }
    ]
  }
}
{
  "name": "opendistro",
  "nodeTemplate": {
    "labels": {
      "node-role/opendistro": ""
    },
    "taints": [
      {
        "effect": "NoSchedule",
        "key": "dedicated",
        "value": "opendistro"
      }
    ]
  }
}
{
  "name": "router",
  "nodeTemplate": {
    "labels": {
      "node-role/router": ""
    },
    "taints": [
      {
        "effect": "NoExecute",
        "key": "dedicated",
        "value": "router"
      }
    ]
  }
}
{
  "name": "system",
  "nodeTemplate": {
    "labels": {
      "node-role.deckhouse.io/system": "",
      "node-role/logging": "",
      "node-role/monitoring": ""
    },
    "taints": [
      {
        "effect": "NoExecute",
        "key": "dedicated.deckhouse.io",
        "value": "system"
      }
    ]
  }
}

@RomanenkoDenys
Copy link
Member Author

Wrong metrics

bash-4.4$ curl -s 127.0.0.1:9115/metrics/hooks | grep flant_pricing_node_group_cpu_cores
# HELP flant_pricing_node_group_cpu_cores flant_pricing_node_group_cpu_cores
# TYPE flant_pricing_node_group_cpu_cores gauge
flant_pricing_node_group_cpu_cores{hook="node_group_resources_metrics.py",is_frontend="false",is_master="false",is_monitoring="false",is_system="false"} 2
flant_pricing_node_group_cpu_cores{hook="node_group_resources_metrics.py",is_frontend="false",is_master="false",is_monitoring="false",is_system="true"} 8
flant_pricing_node_group_cpu_cores{hook="node_group_resources_metrics.py",is_frontend="false",is_master="true",is_monitoring="false",is_system="false"} 8

@RomanenkoDenys
Copy link
Member Author

Right metrics:

 curl -s 127.0.0.1:9115/metrics/hooks | grep flant_pricing_node_group_cpu_cores                                                                                                                                                                                           pricing-r4bzl: Thu May  2 11:34:13 2024

# HELP flant_pricing_node_group_cpu_cores flant_pricing_node_group_cpu_cores
# TYPE flant_pricing_node_group_cpu_cores gauge
flant_pricing_node_group_cpu_cores{hook="node_group_resources_metrics.py",is_frontend="false",is_master="false",is_monitoring="false",is_system="false"} 286
flant_pricing_node_group_cpu_cores{hook="node_group_resources_metrics.py",is_frontend="false",is_master="false",is_monitoring="false",is_system="true"} 8
flant_pricing_node_group_cpu_cores{hook="node_group_resources_metrics.py",is_frontend="false",is_master="true",is_monitoring="false",is_system="false"} 8

@RomanenkoDenys
[flant-integration] fix resources counting and add tests
942194a 
Signed-off-by: Denis Romanenko <denis.romanenko@flant.com>
@RomanenkoDenys RomanenkoDenys added the status/backport Backport pr label May 2, 2024
@konstantin-axenov konstantin-axenov merged commit 50a8213 into main May 2, 2024
34 of 38 checks passed
@konstantin-axenov konstantin-axenov deleted the fix-flant-integration-metrics branch May 2, 2024 13:03
@deckhouse-BOaTswain deckhouse-BOaTswain mentioned this pull request May 2, 2024
Merged
github-actions bot pushed a commit that referenced this pull request May 2, 2024
@RomanenkoDenys @konstantin-axenov
[flant-integration] fix resources counting and add tests (#8291)
42c3b96 
Signed-off-by: Denis Romanenko <denis.romanenko@flant.com>
deckhouse-BOaTswain added a commit that referenced this pull request May 2, 2024
@deckhouse-BOaTswain @RomanenkoDenys
[flant-integration] fix resources counting and add tests (#8291) (#8294)
9b78f36 
Signed-off-by: Denis Romanenko <denis.romanenko@flant.com>
Co-authored-by: Denys Romanenko <65756796+RomanenkoDenys@users.noreply.github.com>
@deckhouse-BOaTswain
Copy link
Collaborator

Cherry pick PR 8294 to the branch release-1.59 successful!

@deckhouse-BOaTswain deckhouse-BOaTswain removed the status/backport Backport pr label May 2, 2024
aleksey-su pushed a commit that referenced this pull request May 2, 2024
@RomanenkoDenys @aleksey-su
[flant-integration] fix resources counting and add tests (#8291)
fb92184 
Signed-off-by: Denis Romanenko <denis.romanenko@flant.com>
@deckhouse-BOaTswain deckhouse-BOaTswain mentioned this pull request May 6, 2024
Merged
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@konstantin-axenov konstantin-axenov konstantin-axenov approved these changes

@name212 name212 name212 approved these changes

@nabokihms nabokihms Awaiting requested review from nabokihms nabokihms is a code owner

Assignees

@RomanenkoDenys RomanenkoDenys

Labels
python Pull requests that update Python code
Projects
None yet
Milestone
v1.59.9
Development

Successfully merging this pull request may close these issues.

None yet
4 participants
@RomanenkoDenys @deckhouse-BOaTswain @konstantin-axenov @name212