Integration with cAdvisor

[monnand]

cAdvisor right now already have some basic statistical information about containers running on the machine. We currently hold most recent stats in memory but we have a framework to dump stats into some backend storage. google/cadvisor#39 has a discussion about support influxdb and I think @erikh is working on that now.

I think the information collected by cAdvisor may be useful for kubernetes. I'm currently considering to add some code into kubelet so that kubelet could pull information from cAdvisor periodically.

Before getting started, I would like to discuss the approach we should take and other issues.

Currently, cAdvisor collects stats and stores them into memory. It will only remember recent stats and provide resource usage percentiles (currently, there are only CPU and memory). I think the resource usage percentiles would be useful for kubernetes master to do scheduling.

There are two possible ways to retrieve such information from cAdvisor:

Solution 1: kubelet pulls information from cAdvisor through REST API and expose another REST API for the master. The master will periodically check the containers' stats through kubelet's REST API. In this case, any information is sent through REST API. Currently, kubelet communicate with master through etcd only. So this approach will add one more communicate channel between kubelet and the master.

Solution 2: Since the percentiles information (e.g. what is the 90th percentile memory usage of a container) is not too big and may be small enough to fit into etcd. We could let cAdvisor update containers' information in etcd and let the master retrieve it whenever it wants to. Or, we could let kubelet pulls such information from cAdvisor and update the corresponding ectd key. In both cases, the communication is through etcd. The disadvantage of this approach is that the information that the master needs may increase to some big message not suitable to be put into etcd.

I would like to see other approaches or discussions of proposed ones.

[brendandburns]

I think I prefer option 1. There is already a communication channel from master to kublet (see kublet_server.go) and I think we can merge the info in there.

Let me know if you want/need more details.

Let's not store this in etcd for now, I don't see too much utility in storing this kind of transient info in etcd.

[monnand]

@brendanburns Oh, I didn't notice kubelet has already exposed a REST API. OK. I could work on it this week. Thank you @brendandburns .

[proppy]

I'd be great if the scheduler was also leverage this data for pod placement, should I file a separate bug?

[vmarmol]

I believe that is planned, but we can file an issue to track it
On Jun 27, 2014 4:12 PM, "Johan Euphrosine" notifications@github.com
wrote:

I'd be great if the scheduler was also leverage this data for pod
placement, should I file a separate bug?

—
Reply to this email directly or view it on GitHub
#160 (comment)
.

[monnand]

Currently, #328 and #174 could let kubelet retrieve some basic stats info from cAdvisor. On cAdvisor side, google/cadvisor#74 could put all stats into influxdb.

If we want to display the data, we could write a separate UI program to retrieve stats from the backend storage (There is only influxdb now) directly (reading the influxdb) or indirectly (retrieving from cAdvisor). Or, we could put this feature into the api server.

[timothysc]

Why can't the kublet just regularly send a 'nvp thunk' including cAdvisor data to a store which the scheduler can then use well established variables for initial algorithm, then extra discrimination via constraint.

[monnand]

@timothysc cAdvisor now could dump all stats into influxdb. Kubelet will retrieve a summary of such stats from cAdvisor and in turn pulled by master to do scheduling decisions. I think its similar to what you described?