I think I prefer option 1. There is already a communication channel from master to kublet (see kublet_server.go) and I think we can merge the info in there.

Let me know if you want/need more details.

Let's not store this in etcd for now, I don't see too much utility in storing this kind of transient info in etcd.

@brendanburns Oh, I didn't notice kubelet has already exposed a REST API. OK. I could work on it this week. Thank you @brendandburns .

I'd be great if the scheduler was also leverage this data for pod placement, should I file a separate bug?

I believe that is planned, but we can file an issue to track it
On Jun 27, 2014 4:12 PM, "Johan Euphrosine" notifications@github.com
wrote:

I'd be great if the scheduler was also leverage this data for pod
placement, should I file a separate bug?

—
Reply to this email directly or view it on GitHub
#160 (comment)
.

Currently, #328 and #174 could let kubelet retrieve some basic stats info from cAdvisor. On cAdvisor side, google/cadvisor#74 could put all stats into influxdb.

If we want to display the data, we could write a separate UI program to retrieve stats from the backend storage (There is only influxdb now) directly (reading the influxdb) or indirectly (retrieving from cAdvisor). Or, we could put this feature into the api server.

Why can't the kublet just regularly send a 'nvp thunk' including cAdvisor data to a store which the scheduler can then use well established variables for initial algorithm, then extra discrimination via constraint.

@timothysc cAdvisor now could dump all stats into influxdb. Kubelet will retrieve a summary of such stats from cAdvisor and in turn pulled by master to do scheduling decisions. I think its similar to what you described?

Re-assign this back to me since @vmarmol is out this week. I will update the status by end of today, then decide the next step.

Here is a quick summary on the status:

1. cAdvisor is running on all minions by default.

Status: Done.

2. Kubelet does have REST API through which master could retrieve per container's stats.

Status: API is there, and implementation is done, but I failed to retrieve any container stats due to: "Internal Error: couldn't find container". I believe this is just a bug. Will fix next.

3. Kubelet does have REST API through which master could retrieve per Pod's stats.

Status: API is there, but not implemented yet. I think I can add a simple support easily by returning the stats all containers within a pod

4. Currently master never retrieve and consume those stats, even there is no REST API for apiserver to get such stats. The closest thing we had is a TODO I added recently:
   type ContainerStatus struct {
   State ContainerState json:"state,omitempty" yaml:"state,omitempty"
   RestartCount int json:"restartCount" yaml:"restartCount"
   ...
   // TODO(dchen1107): Once we have done with integration with cadvisor, resource
   // usage should be included.
   }
   // PodInfo contains one entry for every container with available info.
   type PodInfo map[string]ContainerStatus

But I guess this is belonging to a separate issue, especially downward API? @bgrant0607

5. On stats side, currently only cpu stats are available. Issue Enable memory cgroup on GCE VMs #1548 and Using latest containervm image when bring up a Kubernetes cluster. #738 were filed.

Sent too soon, there are more to update:

6. Machine capacity discovery
   Status: Kubelet has REST API to retrieve machine capacity from cAdvisor, but master doesn't retrieve such stats from kubelet. It also belongs to downward API.

7. Root stats
   Status: Kubelet has REST API to retrieve root usage stats from cAdvisor, but there is no way to propagate it to master too. Root stats should include the usages of kernel threads, kubelet and other daemons today which are not running in a docker container. Those stats should be reported back master for proper scheduling.

cc/ @bgrant0607 @thockin @brendandburns @smarterclayton
For 0.5 milestone, I think we should only fix 2) and maybe 3), then fix 4), 7) along with v1beta3 support separately. 5) should be fixed through #738 since we want to have that available to track the instance hours and other metrics anyway.

We'll need (6) for scheduling won't we?

On Tue, Oct 7, 2014 at 4:41 PM, Dawn Chen notifications@github.com wrote:

Sent too soon, there are more to update:

6. Machine capacity discovery
   Status: Kubelet has REST API to retrieve machine capacity from cAdvisor,
   but master doesn't retrieve such stats from kubelet. It also belongs to
   downward API.

7. Root stats
   Status: Kubelet has REST API to retrieve root usage stats from cAdvisor,
   but there is no way to propagate it to master too. Root stats should
   include the usages of kernel threads, kubelet and other daemons today which
   are not running in a docker container. Those stats should be reported back
   master for proper scheduling.

cc/ @bgrant0607 https://github.com/bgrant0607 @thockin
https://github.com/thockin @brendandburns
https://github.com/brendandburns @smarterclayton
https://github.com/smarterclayton
For 0.5 milestone, I think we should only fix 2) and maybe 3), then fix
4), 7) along with v1beta3 support separately. 5) should be fixed through
#738 #738 since
we want to have that available to track the instance hours and other
metrics anyway.

Reply to this email directly or view it on GitHub
#160 (comment)
.

@thockin, we do need 6) for scheduling, and it should be fixed together with 4) and 7) along with v1beta3 support. Thanks for pointing it out.

Also I will file a separate issue to discuss if including namespace in kubelet's REST api.

Terminology-wise the "downward API" relates to APIs exposed within containers and to the container environment (filesystem, env. vars., etc.). A downward API is needed, but that's a separate discussion, happening on a Docker issue right now.

For the v0.5 milestone, bug fixes seem sufficient.

v1beta3 is on v0.7. Once we have that, the Kubelet and apiserver APIs will be distinct, which will give us the freedom to decide what to expose via the general apiserver /pods API. Resource data changes continuously and will become more and more voluminous, with more resources represented (e.g., I/O), histograms, avg/max, load/latency, effective soft/hard limits, etc. We won't want to expose all of that via /pods.

I'm ok with the scheduling-related bits being moved to milestone v0.8, which is where I put the other scheduling and isolation issues.

Based on above discussion. The fix is in, and v1beta3 related is already on v0.7, and scheduling-related is already on v0.8. I plan to have ContainerVM image by default for GCE in v0.7, so that it automatically resolve the issue of enabling memcg cgroup. Closing the issue.