Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Resources calculation of a pod with restartable init containers is wrong in kubectl describe nodes #119406

Closed
gjkim42 opened this issue Jul 18, 2023 · 5 comments
Assignees
Labels
kind/bug Categorizes issue or PR as related to a bug. sig/node Categorizes an issue or PR as relevant to SIG Node. triage/accepted Indicates an issue or PR is ready to be actively worked on.

Comments

@gjkim42
Copy link
Member

gjkim42 commented Jul 18, 2023

What happened?

Resources calculation of a pod with restartable init containers is wrong in kubectl describe nodes
It does not take into account the restartable init containers.

/cc @SergeyKanzhelev @tzneal
/sig node
ref: #115934

What did you expect to happen?

kubectl describe nodes should show the right resource requirement considering restartable init containers.

How can we reproduce it (as minimally and precisely as possible)?

  1. Create a cluster enabling SidecarContainers feature gate.

  2. Create a pod with a restartable init container

$ cat <<EOF | kubectl apply -f -                                                                  
apiVersion: v1                                                                                    
kind: Pod                                      
metadata:                                                                                                                                                                                                  
  name: sidecar                                                                                                                                                                                            
spec:                                                                                                                                                                                                      
  initContainers:                                                                                                                                                                                          
  - name: init-1                                                                                                                                                                                           
    image: ubuntu                                                                                                                                                                                          
    command:                                                                                                                                                                                               
    - sh                                                                                                                                                                                                   
    - -c                                                                                                                                                                                                   
    - |                                                                                                                                                                                                    
      sleep 1s                                                                                                                                                                                             
    resources:                                                                                                                                                                                             
      limits:                        
        cpu: 100m                                                                                    
        memory: 100Mi                                                                                
  - name: sidecar-1                                                                                                                                                                                        
    image: ubuntu                                                                                                                                                                                          
    command:                                                                                                                                                                                               
    - sh                                                                                                                                                                                                   
    - -c                                                                                                                                                                                                   
    - |                                                                                                                                                                                                    
      sleep 600s                                                                                                                                                                                           
    resources:                                                                                                                                                                                             
      limits:                                                                                        
        cpu: 200m                                                                                                                                                                                          
        memory: 200Mi                                                                                
    restartPolicy: Always                                                                                                                                                                                  
  containers:                                                                                                                                                                                              
  - name: main                                                                                                                                                                                             
    image: ubuntu                                                                                                                                                                                          
    command:                                                                                                                                                                                               
    - sh                                                                                                                                                                                                   
    - -c                                                                                                                                                                                                   
    - |                                                                                                                                                                                                    
      sleep 600s                                                                                                                                                                                           
    resources:                                                                                                                                                                                             
      limits:                                                                                                                                                                                              
        cpu: 300m                                                                                                                                                                                          
        memory: 300Mi                                                                                                                                                                                      
  terminationGracePeriodSeconds: 1                                                                                                                                                                         
EOF                                                                                               
pod/sidecar created
  1. wait for the pod is running.

  2. kubectl describe node THE_NODE_RUNNING_THAT_POD

$ kubectl describe nodes | grep sidecar -B 3
Non-terminated Pods:          (10 in total)
  Namespace                   Name                                          CPU Requests  CPU Limits  Memory Requests  Memory Limits  Age
  ---------                   ----                                          ------------  ----------  ---------------  -------------  ---
  default                     sidecar                                       300m (2%)     300m (2%)   300Mi (1%)       300Mi (1%)     6m43s

The resource requirement should be 500m CPU, 500Mi memory

Anything else we need to know?

No response

Kubernetes version

$ kubectl version
# paste output here

Cloud provider

OS version

# On Linux:
$ cat /etc/os-release
# paste output here
$ uname -a
# paste output here

# On Windows:
C:\> wmic os get Caption, Version, BuildNumber, OSArchitecture
# paste output here

Install tools

Container runtime (CRI) and version (if applicable)

Related plugins (CNI, CSI, ...) and versions (if applicable)

@gjkim42 gjkim42 added the kind/bug Categorizes issue or PR as related to a bug. label Jul 18, 2023
@k8s-ci-robot k8s-ci-robot added sig/node Categorizes an issue or PR as relevant to SIG Node. needs-triage Indicates an issue or PR lacks a `triage/foo` label and requires one. labels Jul 18, 2023
@tzneal
Copy link
Contributor

tzneal commented Jul 18, 2023

/triage accepted
/assign tzneal

@k8s-ci-robot k8s-ci-robot added triage/accepted Indicates an issue or PR is ready to be actively worked on. and removed needs-triage Indicates an issue or PR lacks a `triage/foo` label and requires one. labels Jul 18, 2023
@SergeyKanzhelev SergeyKanzhelev added this to Triage in SIG Node Bugs Jul 18, 2023
@SergeyKanzhelev SergeyKanzhelev moved this from Triage to Triaged in SIG Node Bugs Jul 19, 2023
@gjkim42
Copy link
Member Author

gjkim42 commented Aug 22, 2023

We can close this now as #119509 has been merged.

/close

@k8s-ci-robot
Copy link
Contributor

@gjkim42: Closing this issue.

In response to this:

We can close this now as #119509 has been merged.

/close

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

SIG Node Bugs automation moved this from Triaged to Done Aug 22, 2023
@Sivakajan-tech
Copy link

@gjkim42 @tzneal QQ: Why are only the resources of the restartable init containers expected to be counted, while normal init containers are not included? Your response is highly appreciated.

@gjkim42
Copy link
Member Author

gjkim42 commented Mar 1, 2024

@gjkim42 @tzneal QQ: Why are only the resources of the restartable init containers expected to be counted, while normal init containers are not included? Your response is highly appreciated.

The reason we had excluded the normal init containers was that they temporarily occupy resources and release the resources after the initialization phase. However, the restartable init container occupies the resources for the entire lifecyle of the pod.

See https://kubernetes.io/docs/concepts/workloads/pods/sidecar-containers/ for the detail.
"Restartable init container" is the internal concept to implement the native sidecar container.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
kind/bug Categorizes issue or PR as related to a bug. sig/node Categorizes an issue or PR as relevant to SIG Node. triage/accepted Indicates an issue or PR is ready to be actively worked on.
Projects
Development

No branches or pull requests

4 participants