Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

The Kubernetes Metrics Server / HPA Custom Metrics Support #2751

Open
diclophis opened this issue Mar 30, 2018 · 20 comments
Open

The Kubernetes Metrics Server / HPA Custom Metrics Support #2751

diclophis opened this issue Mar 30, 2018 · 20 comments

Comments

@diclophis
Copy link

@diclophis diclophis commented Mar 30, 2018

Expected behavior

I would like to be able to develop in my docker-for-mac kubernetes cluster features like the "Kubernetes Metrics Server" ( https://github.com/kubernetes-incubator/metrics-server ) and "Custom/External Metrics" for the HPA modules ( https://kubernetes.io/docs/tasks/run-application/horizontal-pod-autoscale/#requirements )

Actual behavior

I am unable to configure these features

Information

Version 18.03.0-ce-mac58 (23607)
@mac2000
Copy link

@mac2000 mac2000 commented Mar 31, 2018

Will be also nice if metrics server and heapster will be shipped by default, e.g. to make environment as close as possible to what we get in GCP/AWS/Azure

e.g. at moment, locally I can not play with horizontal pod autoscaler because it can not collect current cpu usage from deployment

@diclophis
Copy link
Author

@diclophis diclophis commented Apr 2, 2018

@mac2000 Indeed, either by default, or via some gui checkboxes that allow finer tuned control over the kubelet and apiserver flags would be great!

@jasonwc
Copy link

@jasonwc jasonwc commented May 13, 2018

@mac2000 @diclophis did you ever figure out how to get heapster installed and reporting on Docker for Mac? Just ran into the same issue. Huge bummer cause I was hoping to demonstrate auto scaling locally for a talk I'm giving.

@jasonwc
Copy link

@jasonwc jasonwc commented May 13, 2018

Running Version 18.04.0-ce-mac62 (23965)

@YRM64
Copy link

@YRM64 YRM64 commented May 13, 2018

Configure Kubernetes with Autoscaling with Custom Metrics (docs.bitnami.com). If you haven't seen it yet, this article discusses how to configure Kubernetes with autoscaling, Kubernetes Aggregator Layer, Prometheus, Cluster Configuration, Checking the Status of the Cluster, Cluster Configuration, Deploying a Prometheus Monitoring System, Deploying a Custom API Server, and Deploying a Sample Application, with a Summary. I hope this helps; let me know if you're still experiencing configuration issues.

@jasonwc
Copy link

@jasonwc jasonwc commented May 13, 2018

Thanks those are some great resources!

I ended up getting it to work by setting up the metrics-server instead. Would be cool if that shipped by default like most standard kube-up setups, but you can go here to clone it and deploy it as a workaround.

@YRM64
Copy link

@YRM64 YRM64 commented May 14, 2018

Thanks Jasonwc for your contribution.

@docker-desktop-robot
Copy link
Collaborator

@docker-desktop-robot docker-desktop-robot commented Aug 12, 2018

Issues go stale after 90d of inactivity.
Mark the issue as fresh with /remove-lifecycle stale comment.
Stale issues will be closed after an additional 30d of inactivity.

Prevent issues from auto-closing with an /lifecycle frozen comment.

If this issue is safe to close now please do so.

Send feedback to Docker Community Slack channels #docker-for-mac or #docker-for-windows.
/lifecycle stale

@diclophis
Copy link
Author

@diclophis diclophis commented Aug 13, 2018

/remove-lifecycle stale

Is there any chance we can see this be included by default with docker-for-mac ?

@jasonwc
Copy link

@jasonwc jasonwc commented Aug 13, 2018

Agreed. The cluster that ships with docker-for-mac should be as close to a "standard" Kube cluster as possible.

@paultiplady
Copy link

@paultiplady paultiplady commented Sep 8, 2018

I'm hitting certificate validation issues running the latests metrics-server (v0.3.0) against a Docker for Mac k8s cluster (Kubernetes: v1.10.3, Engine: 18.06.1-ce).

The error is this one: kubernetes-sigs/metrics-server#133 -

unable to fully collect metrics: unable to fully scrape metrics from source kubelet_summary:docker-for-desktop: unable to fetch metrics from Kubelet docker-for-desktop (docker-for-desktop): Get https://docker-for-desktop:10250/stats/summary/: x509: certificate signed by unknown authority

Seems like the Docker for Mac node might not be wired up to the cluster CA properly? (Either that or it's a metrics-server bug).

Anyone else seen this / found a workaround?

@vfarcic
Copy link

@vfarcic vfarcic commented Oct 4, 2018

I have the same issue as @paultiplady. Last time I used Metrics Server (a few weeks ago), it worked. Now it doesn't. I'm installing the same Metrics Server version now as before so I guess that's not the issue. My best guess is that my Docker For Mac was upgraded since the last time I run Metrics Server successfully.

@marcellodesales
Copy link

@marcellodesales marcellodesales commented Nov 16, 2018

@paultiplady @vfarcic Any solutions on this? I'm hitting the same roadblock :(

@paultiplady
Copy link

@paultiplady paultiplady commented Nov 16, 2018

@marcellodesales I gave up on Kubernetes for Docker for Mac, and went back to Minikube.

@fallwith
Copy link

@fallwith fallwith commented Nov 26, 2018

@paultiplady @vfarcic @marcellodesales - I was able to get past the cert verification issue that exists between metrics-server and a Docker based instance of Kubernetes via the instructions found at https://qiita.com/rk05231977/items/64825c095a6ec37e49f8

Here are the full steps I took to get things working:

  1. Have Kubernetes running within Docker
  2. Delete any previous instance of metrics-server from your Kubernetes instance with kubectl delete -n kube-system deployments.apps metrics-server
  3. Clone metrics-server with git clone https://github.com/kubernetes-incubator/metrics-server.git
  4. Edit deploy/1.8+/metrics-server-deployment.yaml to override the default command by adding a command section that didn't exist before. The new section will instruct metrics-server to allow for an insecure communications session (don't verify the certs involved). Do this only for Docker, and not for production deployments of metrics-server:
% git diff
diff --git a/deploy/1.8+/metrics-server-deployment.yaml b/deploy/1.8+/metrics-server-deployment.yaml
index ad2abaf..517166b 100644
--- a/deploy/1.8+/metrics-server-deployment.yaml
+++ b/deploy/1.8+/metrics-server-deployment.yaml
@@ -30,6 +30,9 @@ spec:
      containers:
      - name: metrics-server
        image: k8s.gcr.io/metrics-server-amd64:v0.3.1
+        command:
+          - /metrics-server
+          - --kubelet-insecure-tls
        imagePullPolicy: Always
        volumeMounts:
        - name: tmp-dir
  1. Add metrics-server to your Kubernetes instance with kubectl create -f deploy/1.8+

It would ne nice if Docker (for any OS) bundled metrics-server, but I don't think anything needs to be done on the Docker side to make metrics-server work. Docker is correctly using a cert, and metrics-server needs to either be taught to trust that cert or bypass verification entirely.

Recommendations:

  • Close this issue
  • Open a new Docker issue (not just for Mac) to propose the bundling of metrics-server
  • Create an issue with kubernetes-incubator/metrics-server to trust Docker's cert or perhaps recognize Docker and automatically pass --kubelet-insecure-tls
@narg95
Copy link

@narg95 narg95 commented Dec 6, 2018

@fallwith thanks it worked like a charm, you saved me a lot of time :) The metrics-server docu does not mention how to apply those flags.

In my case after modifing the metrics-server-deployment.yaml file I just had to apply those changes kubectl apply -f deploy/1.8+/

@bitsofinfo
Copy link

@bitsofinfo bitsofinfo commented Feb 7, 2019

SOLVED: (i was missing resources definition on the ReplicaSet's container spec) i.e.

          resources:
            limits:
              cpu: "1"
            requests:
              cpu: "0.5"

I've followed @fallwith example (thanks) and I no longer see the TLS errors

I'm running latest edge for mac (k8 1.13)

kubectl top pod shows my pod CPU and MEMORY fine

Created my hpa with kubectl autoscale rs my-app --min=1 --max=5 --cpu-percent=1

The logs for the metrics-server look ok, I see HPA requests returning 200's

But my HPA doesn't ever have any data, I just get this for a kubectl describe hpa my-app.

Any thoughts?

Name:                                                  my-app
Namespace:                                             default
Labels:                                                <none>
Annotations:                                           <none>
CreationTimestamp:                                     Thu, 07 Feb 2019 09:30:51 -0700
Reference:                                             ReplicaSet/my-app
Metrics:                                               ( current / target )
  resource cpu on pods  (as a percentage of request):  <unknown> / 1%
Min replicas:                                          1
Max replicas:                                          5
ReplicaSet pods:                                       1 current / 0 desired
Conditions:
  Type           Status  Reason                   Message
  ----           ------  ------                   -------
  AbleToScale    True    SucceededGetScale        the HPA controller was able to get the target's current scale
  ScalingActive  False   FailedGetResourceMetric  the HPA was unable to compute the replica count: missing request for cpu
Events:
  Type     Reason                        Age                From                       Message
  ----     ------                        ----               ----                       -------
  Warning  FailedGetResourceMetric       4s (x8 over 109s)  horizontal-pod-autoscaler  missing request for cpu
  Warning  FailedComputeMetricsReplicas  4s (x8 over 109s)  horizontal-pod-autoscaler  failed to get cpu utilization: missing request for cpu
@secat
Copy link

@secat secat commented Mar 20, 2019

You could also use the official helm chart stable/metrics-server using the following values.yaml file:

args:
  - --logtostderr
# enable this if you have self-signed certificates, see: https://github.com/kubernetes-incubator/metrics-server
  - --kubelet-insecure-tls

hostNetwork:
  # Specifies if metrics-server should be started in hostNetwork mode.
  #
  # You would require this enabled if you use alternate overlay networking for pods and
  # API server unable to communicate with metrics-server. As an example, this is required
  # if you use Weave netwok on EKS
  enabled: true

The node metrics works:

$ kubectl get --raw "/apis/metrics.k8s.io/v1beta1/nodes" | jq

will return:

{                                                                       
  "kind": "NodeMetricsList",                                            
  "apiVersion": "metrics.k8s.io/v1beta1",                               
  "metadata": {                                                         
    "selfLink": "/apis/metrics.k8s.io/v1beta1/nodes"                    
  },                                                                    
  "items": [                                                            
    {                                                                   
      "metadata": {                                                     
        "name": "docker-desktop",                                       
        "selfLink": "/apis/metrics.k8s.io/v1beta1/nodes/docker-desktop",
        "creationTimestamp": "2019-03-20T13:26:15Z"                     
      },                                                                
      "timestamp": "2019-03-20T13:25:52Z",                              
      "window": "30s",                                                  
      "usage": {                                                        
        "cpu": "521240620n",                                            
        "memory": "3148748Ki"                                           
      }                                                                 
    }                                                                   
  ]                                                                     
}                                                                       

However, the pod metrics doesn't work with error type like:

E0320 13:21:08.077551       1 reststorage.go:144] unable to fetch pod metrics for pod [...] : no metrics known for pod
@jianglijie
Copy link

@jianglijie jianglijie commented Aug 14, 2019

@fallwith After I follow the steps,metrics-server is runing, also I can use kubectl get pod and kubectl get node But I cant kubectl apply anything. It report error: SchemaError(io.k8s.apimachinery.pkg.apis.meta.v1.APIGroup_v2): invalid object doesn't have additional properties
the docker-for-mac-desktop version is 2.0.0.3 and k8s version is 1.10.11

@gaurish
Copy link

@gaurish gaurish commented May 10, 2020

@secat I was about to reach to the solution where host metrics work but pod metrics are not working. Wondering, if you or anyone else was able to make progress to get pod metrics working?

Tested on latest version, Docker 2.2.0.5.

[Edit]
I got have the raw metrics for the pod as well:

$ kubectl get --raw "/apis/metrics.k8s.io/v1beta1/pods" | jq '.items[] | select(.containers[].name=="httpd")'

{
  "metadata": {
    "name": "httpd-dc5bb96b-sfv7t",
    "namespace": "default",
    "selfLink": "/apis/metrics.k8s.io/v1beta1/namespaces/default/pods/httpd-dc5bb96b-sfv7t",
    "creationTimestamp": "2020-05-10T06:47:01Z"
  },
  "timestamp": "2020-05-10T06:46:18Z",
  "window": "30s",
  "containers": [
    {
      "name": "httpd",
      "usage": {
        "cpu": "64288n",
        "memory": "6376Ki"
      }
    }
  ]
}

But then I goto the HPA, it doesn't work:

$ kubectl describe hpa/httpd                                                                           
Name:                                                  httpd
Namespace:                                             default
Labels:                                                <none>
Annotations:                                           <none>
CreationTimestamp:                                     Sun, 10 May 2020 01:30:04 -0500
Reference:                                             Deployment/httpd
Metrics:                                               ( current / target )
  resource cpu on pods  (as a percentage of request):  1% (1m) / 50%
Min replicas:                                          1
Max replicas:                                          10
Deployment pods:                                       1 current / 1 desired
Conditions:
  Type            Status  Reason              Message
  ----            ------  ------              -------
  AbleToScale     True    ReadyForNewScale    recommended size matches current size
  ScalingActive   True    ValidMetricFound    the HPA was able to successfully calculate a replica count from cpu resource utilization (percentage of request)
  ScalingLimited  False   DesiredWithinRange  the desired count is within the acceptable range
Events:
  Type     Reason                        Age                From                       Message
  ----     ------                        ----               ----                       -------
  Warning  FailedGetResourceMetric       17m                horizontal-pod-autoscaler  unable to get metrics for resource cpu: no metrics returned from resource metrics API
  Warning  FailedComputeMetricsReplicas  17m                horizontal-pod-autoscaler  Invalid metrics (1 invalid out of 1), last error was: failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API
  Warning  FailedGetResourceMetric       16m (x4 over 17m)  horizontal-pod-autoscaler  did not receive metrics for any ready pods
  Warning  FailedComputeMetricsReplicas  16m (x4 over 17m)  horizontal-pod-autoscaler  Invalid metrics (1 invalid out of 1), last error was: failed to get cpu utilization: did not receive metrics for any ready pods

so the raw metics exists but the HPA is not able to find them. Any idea why?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Linked pull requests

Successfully merging a pull request may close this issue.

None yet