New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Report CPU usageNanoCores for pods/containers in summary API on Windows #72788

Open
yujuhong opened this Issue Jan 10, 2019 · 5 comments

Comments

4 participants
@yujuhong
Copy link
Contributor

yujuhong commented Jan 10, 2019

Related: #62047

On Windows, kubelet reports only the cumulativeusageCoreNanoSeconds for cpu usage.

    "containers": [                                                                                                             
     {                                                                                                                          
      "name": "foo",                                                                                                         
      "startTime": "2019-01-09T23:26:05Z",                                                                                      
      "cpu": {                                                                                                                  
       "time": "2019-01-10T01:02:31Z",                                                                                          
       "usageCoreNanoSeconds": 25156250000                                                                                      
      },

On Linux, kubelet reports both usageCoreNanoSeconds and the instantaneous usageNanoCores.

    "containers": [
     {
      "name": "foo",
      "startTime": "2019-01-09T23:18:42Z",
      "cpu": {
       "time": "2019-01-10T01:08:29Z",
       "usageNanoCores": 2825438,
       "usageCoreNanoSeconds": 19549406996
      },

Although In the future, most components/monitoring pipelines will switch to using the cumulative metrics, a few features still depend on the instantaneous usageNanoCores metrics today. Examples are: kubectl top, HPA/VPA. Kubelet should report this metrics on Windows to support the existing features.

/cc @dashpole, who explained to me what features depend on the cpu metrics.
/cc @kubernetes/sig-windows-feature-requests @PatrickLang @feiskyer
/cc @pjh @mtaufen

@PatrickLang

This comment has been minimized.

Copy link
Contributor

PatrickLang commented Jan 16, 2019

@feiskyer - is this something that's clear enough in the CRI layer that you can implement it, or do we need to fix that first?

@PatrickLang

This comment has been minimized.

Copy link
Contributor

PatrickLang commented Jan 16, 2019

/assign @feiskyer

@PatrickLang

This comment has been minimized.

Copy link
Contributor

PatrickLang commented Jan 16, 2019

@yujuhong do you have any notes or a doc you can point to that better describes what features depend on which metrics? It would be good for us to share and see if anything other than kubectl top is broken

@dashpole

This comment has been minimized.

Copy link
Contributor

dashpole commented Jan 16, 2019

The horizontal and vertical pod autoscalers also depend on the resource metrics api

@yujuhong

This comment has been minimized.

Copy link
Contributor

yujuhong commented Jan 16, 2019

Documentation on metrics API and metrics server (i.e., the aggregator): https://kubernetes.io/docs/tasks/debug-application-cluster/core-metrics-pipeline/

The metrics-server pod keeps spewing errors like this:

unable to fully collect metrics: [unable to fully scrape metrics from source kubelet_summary:e2e-test-yjhong-minion-windows-group-7gg8...

Running kubectl top pod:

# Against a Linux pod
$ kubectl top pod ubuntu
NAME      CPU(cores)   MEMORY(bytes)   
ubuntu    0m           0Mi    

# Against a Windows pod
$ kubectl top pod foo-server
Error from server (NotFound): podmetrics.metrics.k8s.io "default/foo-server" not found

I haven't verified whether pod autoscaling works in my cluster, but I assume they won't since metrics server seems to discard the metrics from the windows pods.

@feiskyer - is this something that's clear enough in the CRI layer that you can implement it, or do we need to fix that first?

I believe the implementation is in kubelet (for the docker integration), so this should not need CRI changes.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment