Need simple kubectl command to see cluster resource usage

Users are getting tripped up by pods not being able to schedule due to resource deficiencies. It can be hard to know when a pod is pending because it just hasn't started up yet, or because the cluster doesn't have room to schedule it. http://kubernetes.io/v1.1/docs/user-guide/compute-resources.html#monitoring-compute-resource-usage helps, but isn't that discoverable (I tend to try a 'get' on a pod in pending first, and only after waiting a while and seeing it 'stuck' in pending, do I use 'describe' to realize it's a scheduling problem).

This is also complicated by system pods being in a namespace that is hidden. Users forget that those pods exist, and 'count against' cluster resources.

There are several possible fixes offhand, I don't know what would be ideal:

Develop a new pod state other than Pending to represent "tried to schedule and failed for lack of resources".
Have kubectl get po or kubectl get po -o=wide display a column to detail why something is pending (perhaps the container.state that is Waiting in this case, or the most recent event.message).
Create a new kubectl command to more easily describe resources. I'm imagining a "kubectl usage" that gives an overview of total cluster CPU and Mem, per node CPU and Mem and each pod/container's usage. Here we would include all pods, including system ones. This might be useful long term alongside more complex schedulers, or when your cluster has enough resources but no single node does (diagnosing the 'no holes large enough' problem).

Something along the lines of (2) seems reasonable, though the UX folks would know better than me.

(3) seems vaguely related to #15743 but I'm not sure they're close enough to combine.

In addition to the case above, it would be nice to see what resource utilization we're getting.

kubectl utilization requests might show (maybe kubectl util or kubectl usage are better/shorter):

cores: 4.455/5 cores (89%)
memory: 20.1/30 GiB (67%)
...

In this example, the aggregate container requests are 4.455 cores and 20.1 GiB and there are 5 cores and 30GiB total in the cluster.

There is:

$ kubectl top nodes
NAME                    CPU(cores)   CPU%      MEMORY(bytes)   MEMORY%   
cluster1-k8s-master-1   312m         15%       1362Mi          68%       
cluster1-k8s-node-1     124m         12%       233Mi           11%

I use below command to get a quick view for the resource usage. It is the simplest way I found.

kubectl describe nodes

If there was a way to "format" the output of kubectl describe nodes, I wouldn't mind scripting my way to summarize all node's resource requests/limits.

here is my hack kubectl describe nodes | grep -A 2 -e "^\\s*CPU Requests"

@from-nibly

@from-nibly thanks, just what i was looking for

Yup, this is mine:

$ cat bin/node-resources.sh 
#!/bin/bash
set -euo pipefail

echo -e "Iterating...\n"

nodes=$(kubectl get node --no-headers -o custom-columns=NAME:.metadata.name)

for node in $nodes; do
  echo "Node: $node"
  kubectl describe node "$node" | sed '1,/Non-terminated Pods/d'
  echo
done

@goltermann

@goltermann There are no sig labels on this issue. Please add a sig label by:
(1) mentioning a sig: @kubernetes/sig-<team-name>-misc
(2) specifying the label manually: /sig <label>

Note: method (1) will trigger a notification to the team. You can find the team list here.

@kubernetes/sig-cli-misc

From the long history of comments here it seems everyone has different expectations by the many issues and requests reported in this thread. This thread is more of a wiki now.

We'd be happy to see one of these plugins be proposed for upstreaming via a KEP. If someone wants to own this and bias for action with a decision, please open a KEP for discussion.

/close

@eddiezane

@eddiezane: Closing this issue.

Details

In response to this:

From the long history of comments here it seems everyone has different expectations by the many issues and requests reported in this thread. This thread is more of a wiki now.

We'd be happy to see one of these plugins be proposed for upstreaming via a KEP. If someone wants to own this and bias for action with a decision, please open a KEP for discussion.

/close

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

In case folks are still listening in on this issue...Has anyone attempted using the standard resource usage apis like getrusage() for sw running inside containers/pods(). For cpu stats it does not seem that it will far off from what the node level cgroup would have to report.

mem stats seem more problematic. Unclear whether say /sys/fs/crgoup/memory/<> from inside a container really reflects memory usage correctly. Being able to monitor the resource usage from within an app (and then changing behavior in the app etc.) is a neat capability. Seems unclear when that will be available in k8s so casting around for workarounds.

another tool to see resources node-wise, namespace-wise: https://github.com/dguyhasnoname/k8s-day2-ops/tree/master/resource_calcuation/k8s-toppur

My hack (on k8s 1.18; EKS)

kubectl describe nodes | grep 'Name:\|Allocated' -A 5 | grep 'Name\|memory'

Lots of gems in this thread, :) thanks all! (wish some good writer could summarize and publish a quick sheet for it)

@jackdpeterson

@jackdpeterson answer adapted for Powershell :)

kubectl describe nodes | Select-String -Pattern 'Allocated resources:' -Context 0,5

kubectl describe nodes | grep "Allocated resources" -A 9

Without counting the lines

kubectl describe nodes | awk '/Allocated resources/,/Events/' | grep -v "^Events:"

It's not perfect, but we can get a serviceable summary with sed:

:;  kubectl describe nodes |
    sed -n '/^Allocated /,/^Events:/ { /^  [^(]/ p; } ; /^Name: / p'
Name:               ip100.k8s.computer
  Resource                    Requests           Limits
  --------                    --------           ------
  cpu                         6773m (90%)        14300m (190%)
  memory                      12851005952 (40%)  18577645056 (57%)
  ephemeral-storage           0 (0%)             0 (0%)
  hugepages-1Gi               0 (0%)             0 (0%)
  hugepages-2Mi               0 (0%)             0 (0%)
Name:               ip200.k8s.computer
  Resource                    Requests           Limits
  --------                    --------           ------
  cpu                         7082m (94%)        9500m (126%)
  memory                      26405455360 (83%)  24630806144 (77%)
  ephemeral-storage           0 (0%)             0 (0%)
  hugepages-1Gi               0 (0%)             0 (0%)
  hugepages-2Mi               0 (0%)             0 (0%)
Name:               ip300.k8s.computer
  Resource                    Requests           Limits
  --------                    --------           ------
  cpu                         7153m (95%)        8800m (117%)
  memory                      27759605888 (86%)  22996783232 (71%)
  ephemeral-storage           0 (0%)             0 (0%)
  hugepages-1Gi               0 (0%)             0 (0%)
  hugepages-2Mi               0 (0%)             0 (0%)

Below script only works in the kubectl describe node values are in unit "m/Ki".

  # Assume unit CPU: m, Memory: Ki
  
  allocatable_cpu=$(kubectl describe node |grep Allocatable -A 5|grep cpu   | awk '{if (index($NF,"m") == 0) $NF=$NF*1000;sum+=$NF;} END{print sum;}')
  allocatable_mem=$(kubectl describe node |grep Allocatable -A 5|grep memory| awk '{sum+=$NF;} END{print sum;}')
  ## the allocated resource in request field
  allocated_req_cpu=$(kubectl describe node |grep Allocated -A 5|grep cpu   | awk '{sum+=$2; } END{print sum;}')
  allocated_req_mem=$(kubectl describe node |grep Allocated -A 5|grep memory| awk '{sum+=$2; } END{print sum;}')

  # finally, we got the resource spaces left
  usable_req_cpu=$(( allocatable_cpu - allocated_req_cpu ))
  usable_req_mem=$(( allocatable_mem - allocated_req_mem ))

  echo Usable       CPU request $usable_req_cpu   M
  echo Usable Memory request $usable_req_mem Ki

Getting in on this shell scripting party. I have an older cluster running the CA with scale down disabled. I wrote this script to determine roughly how much I can scale down the cluster when it starts to bump up on its AWS route limits:

Updated version of this shell function

function kusage() {
    # Function returning resources usage on current kubernetes cluster
	local node_count=0
	local total_percent_cpu=0
	local total_percent_mem=0

    echo "NODE\t\t CPU_allocatable\t Memory_allocatable\t CPU_requests%\t Memory_requests%\t CPU_limits%\t Memory_limits%\t"
	for n in $(kubectl get nodes --no-headers -o custom-columns=NAME:.metadata.name); do
		local requests=$(kubectl describe node $n | grep -A2 -E "Resource" | tail -n1 | tr -d '(%)')
        local abs_cpu=$(echo $requests | awk '{print $2}')
		local percent_cpu=$(echo $requests | awk '{print $3}')
        local node_cpu=$(echo $abs_cpu $percent_cpu | tr -d 'mKi' | awk '{print int($1/$2*100)}')
        local allocatable_cpu=$(echo $node_cpu $abs_cpu | tr -d 'mKi' | awk '{print int($1 - $2)}')
        local percent_cpu_lim=$(echo $requests | awk '{print $5}')
        local requests=$(kubectl describe node $n | grep -A3 -E "Resource" | tail -n1 | tr -d '(%)')
        local abs_mem=$(echo $requests | awk '{print $2}')
		local percent_mem=$(echo $requests | awk '{print $3}')
        local node_mem=$(echo $abs_mem $percent_mem | tr -d 'mKi' | awk '{print int($1/$2*100)}')
        local allocatable_mem=$(echo $node_mem $abs_mem | tr -d 'mKi' | awk '{print int($1 - $2)}')
        local percent_mem_lim=$(echo $requests | awk '{print $5}')
		echo "$n\t ${allocatable_cpu}m\t\t\t ${allocatable_mem}Ki\t\t ${percent_cpu}%\t\t ${percent_mem}%\t\t\t ${percent_cpu_lim}%\t\t ${percent_mem_lim}%\t"

		node_count=$((node_count + 1))
		total_percent_cpu=$((total_percent_cpu + percent_cpu))
		total_percent_mem=$((total_percent_mem + percent_mem))
	done

	local avg_percent_cpu=$((total_percent_cpu / node_count))
	local avg_percent_mem=$((total_percent_mem / node_count))

	echo "Average usage (requests) : ${avg_percent_cpu}% CPU, ${avg_percent_mem}% memory."
}

@julienlau

Update of @julienlau's code, modified it a bit to show usage, added 2 flags (--nodes <node1> <node2> <node3> to print data from specific nodes only, and --pool <pool name> for specific pool - those 2 flags are exclusive) and made indention dynamic, based on nodes' names. Tested with bash 5.2, zsh 5.9, kubectl 1.27

function kusage() {
    # Function returning resources usage on current kubernetes cluster
    local node_count=0
    local total_percent_cpu_req=0
    local total_percent_mem_req=0
    local total_percent_cpu_lim=0
    local total_percent_mem_lim=0
    local total_percent_cpu_usage=0
    local total_percent_mem_usage=0
    local init_tab_position=$(( $(kubectl get nodes | awk '{print $1}' | grep -v NAME | awk '{ print length }' | sort -n | tail -1) + 5))
    tab_width=10
    terminal_width=$( stty size | awk '{print $2}' )
    set_tab_stops $init_tab_position $tab_width $terminal_width

    if [ -z "$1" ]; then
        nodes=( $(kubectl get nodes --no-headers -o custom-columns=NAME:.metadata.name) )
    elif [[ "$1" == "--nodes" ]]; then
            shift
            nodes=( $(echo "$@") )
    elif [[ "$1" == "--pool" ]]; then
            nodes=( $(kubectl get nodes --no-headers -o custom-columns=NAME:.metadata.name | grep "${2}") )
    else
        echo "Wrong arg!"
        exit 1
    fi

    echo -e "NODE\t CPU_allocatable\t Memory_allocatable\t CPU_request%\t Memory_request%\t CPU_limit%\t Memory_limit%\t CPU_usage%\t Memory_usage%"
    for node in ${nodes[@]}; do
        local requests=$(kubectl describe node $node | grep -A2 -E "Resource" | tail -n1 | tr -d '(%)')
        local abs_cpu=$(echo $requests | awk '{print $2}')
        local percent_cpu_req=$(echo $requests | awk '{print $3}')
        local node_cpu_req=$(echo $abs_cpu $percent_cpu_req | tr -d 'mKi' | awk '{print int($1/$2*100)}')
        local allocatable_cpu=$(echo $node_cpu_req $abs_cpu | tr -d 'mKi' | awk '{print int($1 - $2)}')
        local percent_cpu_lim=$(echo $requests | awk '{print $5}')
        local requests=$(kubectl describe node $node | grep -A3 -E "Resource" | tail -n1 | tr -d '(%)')
        local abs_mem=$(echo $requests | awk '{print $2}')
        local percent_mem_req=$(echo $requests | awk '{print $3}')
        local node_mem_req=$(echo $abs_mem $percent_mem_req | tr -d 'mKi' | awk '{print int($1/$2*100)}')
        local allocatable_mem=$(echo $node_mem_req $abs_mem | tr -d 'mKi' | awk '{print int($1 - $2)}')
        local percent_mem_lim=$(echo $requests | awk '{print $5}')
        local percent_cpu_usage=$(kubectl top nodes | grep $node | awk '{print $3}' | rev | cut -c2- | rev)
        local percent_mem_usage=$(kubectl top nodes | grep $node | awk '{print $5}' | rev | cut -c2- | rev)
        echo -e "$node\t ${allocatable_cpu}m\t\t ${allocatable_mem}Ki\t\t ${percent_cpu_req}%\t\t ${percent_mem_req}%\t\t ${percent_cpu_lim}%\t\t ${percent_mem_lim}%\t\t ${percent_cpu_usage}%\t\t ${percent_mem_usage}%"

        node_count=$((node_count + 1))
        total_percent_cpu_req=$((total_percent_cpu_req + percent_cpu_req))
        total_percent_mem_req=$((total_percent_mem_req + percent_mem_req))
        total_percent_cpu_lim=$((total_percent_cpu_lim + percent_cpu_lim))
        total_percent_mem_lim=$((total_percent_mem_lim + percent_mem_lim))
        total_percent_cpu_usage=$((total_percent_cpu_usage + percent_cpu_usage))
        total_percent_mem_usage=$((total_percent_mem_usage + percent_mem_usage))
    done

    local avg_percent_cpu_req=$((total_percent_cpu_req / node_count))
    local avg_percent_mem_req=$((total_percent_mem_req / node_count))

    local avg_percent_cpu_lim=$((total_percent_cpu_lim / node_count))
    local avg_percent_mem_lim=$((total_percent_mem_lim / node_count))

    local avg_percent_cpu_usage=$((total_percent_cpu_usage / node_count))
    local avg_percent_mem_usage=$((total_percent_mem_usage / node_count))

    echo ""
    echo "Average requests: ${avg_percent_cpu_req}% CPU, ${avg_percent_mem_req}% memory."
    echo "Average limits: ${avg_percent_cpu_lim}% CPU, ${avg_percent_mem_lim}% memory."
    echo "Average usage: ${avg_percent_cpu_usage}% CPU, ${avg_percent_mem_usage}% memory."
    tabs 4
}

set_tab_stops() {
    local init_tab_position=$1 tab_width=$2 terminal_width=$3 tab_stops=''
    for (( i=$init_tab_position; $i<$terminal_width; i+=$tab_width )); do
        tab_stops+=$i','
    done
    tabs $tab_stops
}

as an alternative - kube-capcity seems to do the job

Try KubeNodeUsage

https://github.com/AKSarav/KubeNodeUsage

Unfortunately KubeNodeUsage is pretty but does not output all information in one command.
The bare minimum is to be able to get these 3 metrics :
memory usage (real), memory request, memory limits
This is what does kube-capacity --util

Navigation Menu

Repository navigation

Need simple kubectl command to see cluster resource usage #17512

94 remaining items

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Need simple kubectl command to see cluster resource usage #17512

Description

Activity

davidopp commented on Nov 20, 2015

chrishiestand commented on Sep 15, 2016

xmik commented on Dec 19, 2016

ozbillwang commented on Jan 11, 2017

tonglil commented on Jan 20, 2017

from-nibly commented on Feb 1, 2017

jesseredl commented on May 25, 2017

tonglil commented on May 25, 2017

k8s-github-robot commented on May 31, 2017

0xmichalis commented on Jun 10, 2017

94 remaining items

eddiezane commented on Mar 31, 2021

k8s-ci-robot commented on Mar 31, 2021

champak commented on Apr 15, 2021

dguyhasnoname commented on Jun 18, 2021

jackdpeterson commented on Sep 21, 2021

shawncao commented on Nov 7, 2021

valxv commented on Nov 10, 2021

sanderdescamps commented on Nov 19, 2021

solidsnack commented on Jul 6, 2022

panpan0000 commented on Sep 18, 2022

julienlau commented on Nov 23, 2022

d-mankowski-synerise commented on Sep 23, 2023

AKSarav commented on Mar 28, 2025

julienlau commented on Mar 31, 2025

Metadata

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Issue actions