Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

The Kubernetes Metrics Server / HPA Custom Metrics Support #2751

Open
diclophis opened this Issue Mar 30, 2018 · 18 comments

Comments

Projects
None yet
@diclophis
Copy link

diclophis commented Mar 30, 2018

Expected behavior

I would like to be able to develop in my docker-for-mac kubernetes cluster features like the "Kubernetes Metrics Server" ( https://github.com/kubernetes-incubator/metrics-server ) and "Custom/External Metrics" for the HPA modules ( https://kubernetes.io/docs/tasks/run-application/horizontal-pod-autoscale/#requirements )

Actual behavior

I am unable to configure these features

Information

Version 18.03.0-ce-mac58 (23607)
@mac2000

This comment has been minimized.

Copy link

mac2000 commented Mar 31, 2018

Will be also nice if metrics server and heapster will be shipped by default, e.g. to make environment as close as possible to what we get in GCP/AWS/Azure

e.g. at moment, locally I can not play with horizontal pod autoscaler because it can not collect current cpu usage from deployment

@diclophis

This comment has been minimized.

Copy link
Author

diclophis commented Apr 2, 2018

@mac2000 Indeed, either by default, or via some gui checkboxes that allow finer tuned control over the kubelet and apiserver flags would be great!

@jasonwc

This comment has been minimized.

Copy link

jasonwc commented May 13, 2018

@mac2000 @diclophis did you ever figure out how to get heapster installed and reporting on Docker for Mac? Just ran into the same issue. Huge bummer cause I was hoping to demonstrate auto scaling locally for a talk I'm giving.

@jasonwc

This comment has been minimized.

Copy link

jasonwc commented May 13, 2018

Running Version 18.04.0-ce-mac62 (23965)

@YRM64

This comment has been minimized.

Copy link

YRM64 commented May 13, 2018

Configure Kubernetes with Autoscaling with Custom Metrics (docs.bitnami.com). If you haven't seen it yet, this article discusses how to configure Kubernetes with autoscaling, Kubernetes Aggregator Layer, Prometheus, Cluster Configuration, Checking the Status of the Cluster, Cluster Configuration, Deploying a Prometheus Monitoring System, Deploying a Custom API Server, and Deploying a Sample Application, with a Summary. I hope this helps; let me know if you're still experiencing configuration issues.

@jasonwc

This comment has been minimized.

Copy link

jasonwc commented May 13, 2018

Thanks those are some great resources!

I ended up getting it to work by setting up the metrics-server instead. Would be cool if that shipped by default like most standard kube-up setups, but you can go here to clone it and deploy it as a workaround.

@YRM64

This comment has been minimized.

Copy link

YRM64 commented May 14, 2018

Thanks Jasonwc for your contribution.

@docker-desktop-robot

This comment has been minimized.

Copy link
Collaborator

docker-desktop-robot commented Aug 12, 2018

Issues go stale after 90d of inactivity.
Mark the issue as fresh with /remove-lifecycle stale comment.
Stale issues will be closed after an additional 30d of inactivity.

Prevent issues from auto-closing with an /lifecycle frozen comment.

If this issue is safe to close now please do so.

Send feedback to Docker Community Slack channels #docker-for-mac or #docker-for-windows.
/lifecycle stale

@diclophis

This comment has been minimized.

Copy link
Author

diclophis commented Aug 13, 2018

/remove-lifecycle stale

Is there any chance we can see this be included by default with docker-for-mac ?

@jasonwc

This comment has been minimized.

Copy link

jasonwc commented Aug 13, 2018

Agreed. The cluster that ships with docker-for-mac should be as close to a "standard" Kube cluster as possible.

@paultiplady

This comment has been minimized.

Copy link

paultiplady commented Sep 8, 2018

I'm hitting certificate validation issues running the latests metrics-server (v0.3.0) against a Docker for Mac k8s cluster (Kubernetes: v1.10.3, Engine: 18.06.1-ce).

The error is this one: kubernetes-incubator/metrics-server#133 -

unable to fully collect metrics: unable to fully scrape metrics from source kubelet_summary:docker-for-desktop: unable to fetch metrics from Kubelet docker-for-desktop (docker-for-desktop): Get https://docker-for-desktop:10250/stats/summary/: x509: certificate signed by unknown authority

Seems like the Docker for Mac node might not be wired up to the cluster CA properly? (Either that or it's a metrics-server bug).

Anyone else seen this / found a workaround?

@vfarcic

This comment has been minimized.

Copy link

vfarcic commented Oct 4, 2018

I have the same issue as @paultiplady. Last time I used Metrics Server (a few weeks ago), it worked. Now it doesn't. I'm installing the same Metrics Server version now as before so I guess that's not the issue. My best guess is that my Docker For Mac was upgraded since the last time I run Metrics Server successfully.

@marcellodesales

This comment has been minimized.

Copy link

marcellodesales commented Nov 16, 2018

@paultiplady @vfarcic Any solutions on this? I'm hitting the same roadblock :(

@paultiplady

This comment has been minimized.

Copy link

paultiplady commented Nov 16, 2018

@marcellodesales I gave up on Kubernetes for Docker for Mac, and went back to Minikube.

@fallwith

This comment has been minimized.

Copy link

fallwith commented Nov 26, 2018

@paultiplady @vfarcic @marcellodesales - I was able to get past the cert verification issue that exists between metrics-server and a Docker based instance of Kubernetes via the instructions found at https://qiita.com/rk05231977/items/64825c095a6ec37e49f8

Here are the full steps I took to get things working:

  1. Have Kubernetes running within Docker
  2. Delete any previous instance of metrics-server from your Kubernetes instance with kubectl delete -n kube-system deployments.apps metrics-server
  3. Clone metrics-server with git clone https://github.com/kubernetes-incubator/metrics-server.git
  4. Edit deploy/1.8+/metrics-server-deployment.yaml to override the default command by adding a command section that didn't exist before. The new section will instruct metrics-server to allow for an insecure communications session (don't verify the certs involved). Do this only for Docker, and not for production deployments of metrics-server:
% git diff
diff --git a/deploy/1.8+/metrics-server-deployment.yaml b/deploy/1.8+/metrics-server-deployment.yaml
index ad2abaf..517166b 100644
--- a/deploy/1.8+/metrics-server-deployment.yaml
+++ b/deploy/1.8+/metrics-server-deployment.yaml
@@ -30,6 +30,9 @@ spec:
      containers:
      - name: metrics-server
        image: k8s.gcr.io/metrics-server-amd64:v0.3.1
+        command:
+          - /metrics-server
+          - --kubelet-insecure-tls
        imagePullPolicy: Always
        volumeMounts:
        - name: tmp-dir
  1. Add metrics-server to your Kubernetes instance with kubectl create -f deploy/1.8+

It would ne nice if Docker (for any OS) bundled metrics-server, but I don't think anything needs to be done on the Docker side to make metrics-server work. Docker is correctly using a cert, and metrics-server needs to either be taught to trust that cert or bypass verification entirely.

Recommendations:

  • Close this issue
  • Open a new Docker issue (not just for Mac) to propose the bundling of metrics-server
  • Create an issue with kubernetes-incubator/metrics-server to trust Docker's cert or perhaps recognize Docker and automatically pass --kubelet-insecure-tls
@narg95

This comment has been minimized.

Copy link

narg95 commented Dec 6, 2018

@fallwith thanks it worked like a charm, you saved me a lot of time :) The metrics-server docu does not mention how to apply those flags.

In my case after modifing the metrics-server-deployment.yaml file I just had to apply those changes kubectl apply -f deploy/1.8+/

@bitsofinfo

This comment has been minimized.

Copy link

bitsofinfo commented Feb 7, 2019

SOLVED: (i was missing resources definition on the ReplicaSet's container spec) i.e.

          resources:
            limits:
              cpu: "1"
            requests:
              cpu: "0.5"

I've followed @fallwith example (thanks) and I no longer see the TLS errors

I'm running latest edge for mac (k8 1.13)

kubectl top pod shows my pod CPU and MEMORY fine

Created my hpa with kubectl autoscale rs my-app --min=1 --max=5 --cpu-percent=1

The logs for the metrics-server look ok, I see HPA requests returning 200's

But my HPA doesn't ever have any data, I just get this for a kubectl describe hpa my-app.

Any thoughts?

Name:                                                  my-app
Namespace:                                             default
Labels:                                                <none>
Annotations:                                           <none>
CreationTimestamp:                                     Thu, 07 Feb 2019 09:30:51 -0700
Reference:                                             ReplicaSet/my-app
Metrics:                                               ( current / target )
  resource cpu on pods  (as a percentage of request):  <unknown> / 1%
Min replicas:                                          1
Max replicas:                                          5
ReplicaSet pods:                                       1 current / 0 desired
Conditions:
  Type           Status  Reason                   Message
  ----           ------  ------                   -------
  AbleToScale    True    SucceededGetScale        the HPA controller was able to get the target's current scale
  ScalingActive  False   FailedGetResourceMetric  the HPA was unable to compute the replica count: missing request for cpu
Events:
  Type     Reason                        Age                From                       Message
  ----     ------                        ----               ----                       -------
  Warning  FailedGetResourceMetric       4s (x8 over 109s)  horizontal-pod-autoscaler  missing request for cpu
  Warning  FailedComputeMetricsReplicas  4s (x8 over 109s)  horizontal-pod-autoscaler  failed to get cpu utilization: missing request for cpu
@secat

This comment has been minimized.

Copy link

secat commented Mar 20, 2019

You could also use the official helm chart stable/metrics-server using the following values.yaml file:

args:
  - --logtostderr
# enable this if you have self-signed certificates, see: https://github.com/kubernetes-incubator/metrics-server
  - --kubelet-insecure-tls

hostNetwork:
  # Specifies if metrics-server should be started in hostNetwork mode.
  #
  # You would require this enabled if you use alternate overlay networking for pods and
  # API server unable to communicate with metrics-server. As an example, this is required
  # if you use Weave netwok on EKS
  enabled: true

The node metrics works:

$ kubectl get --raw "/apis/metrics.k8s.io/v1beta1/nodes" | jq

will return:

{                                                                       
  "kind": "NodeMetricsList",                                            
  "apiVersion": "metrics.k8s.io/v1beta1",                               
  "metadata": {                                                         
    "selfLink": "/apis/metrics.k8s.io/v1beta1/nodes"                    
  },                                                                    
  "items": [                                                            
    {                                                                   
      "metadata": {                                                     
        "name": "docker-desktop",                                       
        "selfLink": "/apis/metrics.k8s.io/v1beta1/nodes/docker-desktop",
        "creationTimestamp": "2019-03-20T13:26:15Z"                     
      },                                                                
      "timestamp": "2019-03-20T13:25:52Z",                              
      "window": "30s",                                                  
      "usage": {                                                        
        "cpu": "521240620n",                                            
        "memory": "3148748Ki"                                           
      }                                                                 
    }                                                                   
  ]                                                                     
}                                                                       

However, the pod metrics doesn't work with error type like:

E0320 13:21:08.077551       1 reststorage.go:144] unable to fetch pod metrics for pod [...] : no metrics known for pod
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
You can’t perform that action at this time.