Use Heapster as an in-built monitoring solution for Kubernetes in GCE. #2208

vishh · 2014-11-06T19:43:01Z

Users will have an option to enable it when they setup their cluster (kube-up).

vmarmol · 2014-11-06T19:49:33Z

examples/monitoring/README.md

+
+Heapster enables monitoring of Kubernetes Clusters using [cAdvisor](https://github.com/google/cadvisor). It currently works only on GCE.
+
+#####Run Heapster in a Kubernetes cluster with an Influxdb backend and [Grafana](http://grafana.org/docs/features/influxdb)


Hmmm I'm torn. I don't think we should include these instructions here (they are in the heapster repo and we can point people there). Maybe just have the ones for how to access heapster? Here we should probably put the instructions of how to make kube-up enable heapster.

I thought users can choose to try it outside of kube-up too. I can just place a pointer to the heapster repo. Does that sound reasonable?

Sounds perfect

vmarmol · 2014-11-06T19:55:48Z

LGTM

lavalamp · 2014-11-06T20:22:26Z

examples/monitoring/influx-grafana-pod.json

+      "containers": [{
+        "name": "influxdb",
+        "image": "kubernetes/heapster_influxdb",
+        "ports": [{"containerPort": 8083, "hostPort": 8083},


Does this really need hostPort?

The backend DB (InfluxDB) has its UI available at this port.

dchen1107 · 2014-11-06T20:28:22Z

Are we sure we want to enable this by default? This means even e2e test will start Heapster service, which might potentially slow down the core service / cluster for testing. How about having a flag to enable / disable this?

lavalamp · 2014-11-06T20:30:41Z

examples/monitoring/influx-grafana-service.json

+  "apiVersion": "v1beta1",
+  "port": 8085,
+  "containerPort": 8086,
+  "selector": { "name": "influxdb" }


Seems overly generic.

"provider": "kubernetes-default",
"component": "influxdb"

?

I added the fields you mentioned. My Kube-fu is limited as of now.

Users will have an option to enable it when they setup their cluster (kube-up).

…sable it for tests. Updated heapster readme.

vishh · 2014-11-07T01:33:29Z

@lavalamp: PTAL. I tested and it works! It takes more than 10 minutes for the dashboard to be online. But I assume docker pull slowness is being handled separately.
@thockin: Ping.

…nfig.

vishh · 2014-11-07T19:28:54Z

@dchen1107: As per your suggestion, I have added a config option for monitoring and disabled it for tests.

lavalamp · 2014-11-07T20:31:31Z

LGTM.

@vishh any thoughts about making an e2e test for this?

Use Heapster as an in-built monitoring solution for Kubernetes in GCE.

lavalamp · 2014-11-07T21:25:57Z

Somehow I think this broke our e2e test?

The user name and password to use is located in ~/.kubernetes_auth.

Kubelet is successfully installed on e2e-test-jenkins-minion-1
Kubelet is successfully installed on e2e-test-jenkins-minion-2
Cluster validation succeeded
Error: The resource 'projects/google.com:brendan/global/firewalls/monitoring-heapster' was not found
+------+---------+
| name | network |
+------+---------+
INFO: Waiting for insert of firewall monitoring-heapster. Sleeping for 3s.
INFO: Waiting for insert of firewall monitoring-heapster. Sleeping for 3s.

Table of resources:

+---------------------+---------+
| name                | network |
+---------------------+---------+
| monitoring-heapster | default |
+---------------------+---------+

Table of operations:

+---------------------------------------------------------+--------+-------------------------------+----------------+
| name                                                    | status | insert-time                   | operation-type |
+---------------------------------------------------------+--------+-------------------------------+----------------+
| operation-1415394081083-5074b1d6b8e78-ca7381a8-0f712326 | DONE   | 2014-11-07T13:01:21.236-08:00 | insert         |
+---------------------------------------------------------+--------+-------------------------------+----------------+
/var/lib/jenkins/jobs/kubernetes/workspace/cluster/../cluster/gce/util.sh: line 589: kubectl.sh: command not found
Failed to Setup Monitoring
INFO: Waiting for delete of firewall monitoring-heapster. Sleeping for 3s.
INFO: Waiting for delete of firewall monitoring-heapster. Sleeping for 3s.

vishh · 2014-11-07T21:34:37Z

Posted #2240 to fix that.

On Fri, Nov 7, 2014 at 1:26 PM, Daniel Smith notifications@github.com
wrote:

Somehow I think this broke our e2e test?

The user name and password to use is located in ~/.kubernetes_auth.

Kubelet is successfully installed on e2e-test-jenkins-minion-1
Kubelet is successfully installed on e2e-test-jenkins-minion-2
Cluster validation succeeded
Error: The resource 'projects/google.com:brendan/global/firewalls/monitoring-heapster' was not found
+------+---------+
| name | network |
+------+---------+
INFO: Waiting for insert of firewall monitoring-heapster. Sleeping for 3s.
INFO: Waiting for insert of firewall monitoring-heapster. Sleeping for 3s.

Table of resources:

+---------------------+---------+
| name | network |
+---------------------+---------+
| monitoring-heapster | default |
+---------------------+---------+

Table of operations:

+---------------------------------------------------------+--------+-------------------------------+----------------+
| name | status | insert-time | operation-type |
+---------------------------------------------------------+--------+-------------------------------+----------------+
| operation-1415394081083-5074b1d6b8e78-ca7381a8-0f712326 | DONE | 2014-11-07T13:01:21.236-08:00 | insert |
+---------------------------------------------------------+--------+-------------------------------+----------------+
/var/lib/jenkins/jobs/kubernetes/workspace/cluster/../cluster/gce/util.sh: line 589: kubectl.sh: command not found
Failed to Setup Monitoring
INFO: Waiting for delete of firewall monitoring-heapster. Sleeping for 3s.
INFO: Waiting for delete of firewall monitoring-heapster. Sleeping for 3s.

—
Reply to this email directly or view it on GitHub
#2208 (comment)
.

vmarmol reviewed Nov 6, 2014
View reviewed changes

lavalamp reviewed Nov 6, 2014
View reviewed changes

vishh added 2 commits November 7, 2014 00:47

Use Heapster as an in-built monitoring solution for Kubernetes in GCE.

edf6d8e

Users will have an option to enable it when they setup their cluster (kube-up).

Add an option for enabling monitoring in gce/config-default.sh and di…

ad9cb98

…sable it for tests. Updated heapster readme.

Improve the monitoring setup script and fix bugs in the monitoring co…

37af921

…nfig.

lavalamp added a commit that referenced this pull request Nov 7, 2014

Merge pull request #2208 from vishh/monitoring

ec246ad

Use Heapster as an in-built monitoring solution for Kubernetes in GCE.

lavalamp merged commit ec246ad into kubernetes:master Nov 7, 2014

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Use Heapster as an in-built monitoring solution for Kubernetes in GCE. #2208

Use Heapster as an in-built monitoring solution for Kubernetes in GCE. #2208

vishh commented Nov 6, 2014

vmarmol Nov 6, 2014

vishh Nov 6, 2014

vmarmol Nov 6, 2014

vmarmol commented Nov 6, 2014

lavalamp Nov 6, 2014

vishh Nov 6, 2014

dchen1107 commented Nov 6, 2014

lavalamp Nov 6, 2014

vishh Nov 6, 2014

vishh commented Nov 7, 2014

vishh commented Nov 7, 2014

lavalamp commented Nov 7, 2014

lavalamp commented Nov 7, 2014

vishh commented Nov 7, 2014


		Heapster enables monitoring of Kubernetes Clusters using [cAdvisor](https://github.com/google/cadvisor). It currently works only on GCE.

		#####Run Heapster in a Kubernetes cluster with an Influxdb backend and [Grafana](http://grafana.org/docs/features/influxdb)

Use Heapster as an in-built monitoring solution for Kubernetes in GCE. #2208

Use Heapster as an in-built monitoring solution for Kubernetes in GCE. #2208

Conversation

vishh commented Nov 6, 2014

vmarmol Nov 6, 2014

Choose a reason for hiding this comment

vishh Nov 6, 2014

Choose a reason for hiding this comment

vmarmol Nov 6, 2014

Choose a reason for hiding this comment

vmarmol commented Nov 6, 2014

lavalamp Nov 6, 2014

Choose a reason for hiding this comment

vishh Nov 6, 2014

Choose a reason for hiding this comment

dchen1107 commented Nov 6, 2014

lavalamp Nov 6, 2014

Choose a reason for hiding this comment

vishh Nov 6, 2014

Choose a reason for hiding this comment

vishh commented Nov 7, 2014

vishh commented Nov 7, 2014

lavalamp commented Nov 7, 2014

lavalamp commented Nov 7, 2014

vishh commented Nov 7, 2014