Heapster not getting network rx and tx bytes info

[akash010]

Hi

I am running heapster on kubernetes. Several pods are running but all of them giving 0 values in rx and tx network stats.
On doing docker stats , I found that network stats are also coming 0 for that container but network stats are coming in cadvisor correctly.
Network stats for the pod gcr.io/google_containers/pause:0.8.0 are coming correctly but they are not communicated into the influxdb by the heapster. The containers corresponding to this gcr.io/google_containers/pause:0.8.0 are showing 0 network stats.

What should I do to get correct network stats ?
Please help !

[vishh]

This is a known issue. Kubelet does not expose network stats. We will fix this as soon as Kubernetes v1.0 is rolled out. Network stats should be available for the node though.

cc @dchen1107

[akash010]

Thanks for the reply !

Assuming node means 'worker nodes where kubelet and proxy runs', I am not getting any node specific network stats. Though I am getting cpu and memory based stats.

[vishh]

Thats not expected. I will investigate! Thanks for reporting the issue.

On Thu, Jun 25, 2015 at 4:17 AM, Akash Agarwal notifications@github.com
wrote:

Thanks for the reply !

Assuming node means 'worker nodes where kubelet and proxy runs', I am not
getting any node specific network stats. Though I am getting cpu and memory
based stats.

—
Reply to this email directly or view it on GitHub
#368 (comment)
.

[smarterclayton]

Any update on this? Haven't seen it working yet - not sure if I missed it.

[vishh]

Waiting for kubelet to expose pod level metrics. @timstclair is working on the kubelet APIs now.

[jimmidyson]

@vishh @timstclair What's the status of this? Any kubernetes issues/PRs I can look at?

[timstclair]

@jimmidyson we just decided that we need to re-think the API, rather than trying to force the cAdvisor API into k8s. I should have a PR out in the next couple of days, but I suspect it will not get merged until after thanksgiving since a lot of people are out next week. You can see the draft here (actively changing): https://github.com/timstclair/kubernetes/blob/metrics-beta/pkg/apis/metrics/types.go

[jimmidyson]

@timstclair Thanks for the update. Look forward to your PR!

[jimmidyson]

@timstclair @vishh When do you aim for a stable stats v2 endpoint in kubernetes? A couple of months perhaps?

Thinking about users who want to use network stats today & can see a simple fix to heapster to ingest them - network stats from kubelet are correctly exposed today, ready for ingestion, albeit in a raw cadvisor form.

What do you think about just adding in network stats to existing heapster API while we wait for @timstclair's PR & any subsequent stabilisation of the new metrics APIs?

/cc @jcantrill

[timstclair]

When do you aim for a stable stats v2 endpoint in kubernetes?

I'd like to deprecate the /stats endpoint, and put any data we need going forward into the metrics API. In order to move heapster over to the new API for v1.2, we need to have the new API in beta in a couple weeks. To this end, I'm only planning on including the minimum stats requirements for heapster.

What do you think about just adding in network stats to existing heapster API while we wait for @timstclair's PR & any subsequent stabilisation of the new metrics APIs?

Which network stats are you looking for? It looks like heapster already has network/{tx,rx}{,_errors}. I just had a meeting this morning where we agreed to remove a lot of the more detailed networking stats from the metrics API. cc/ @mwielgus

[vishh]

@jimmidyson: A workaround for now SGTM. Let's get this working. The
cadvisor stats we query directly should let us figure out the network usage
of a pod's infra container. The infra container is tagged with special
docker labels.

On Mon, Nov 23, 2015 at 12:23 PM, Tim St. Clair notifications@github.com
wrote:

When do you aim for a stable stats v2 endpoint in kubernetes?

I'd like to deprecate the /stats endpoint, and put any data we need going
forward into the metrics API. In order to move heapster over to the new API
for v1.2, we need to have the new API in beta in a couple weeks. To this
end, I'm only planning on including the minimum stats requirements for
heapster.

What do you think about just adding in network stats to existing heapster
API while we wait for @timstclair https://github.com/timstclair's PR &
any subsequent stabilisation of the new metrics APIs?

Which network stats are you looking for? It looks like heapster already
has
https://github.com/kubernetes/heapster/blob/master/sinks/api/supported_metrics.go
network/{tx,rx}{,_errors}. I just had a meeting this morning where we
agreed to remove a lot of the more detailed networking stats from the
metrics API. cc/ @mwielgus https://github.com/mwielgus

—
Reply to this email directly or view it on GitHub
#368 (comment)
.

[jimmidyson]

So this is a bit more involved than I thought it would be. AFAIK there is no way to query the infra container in a pod with the current kubelet API & as the network metrics are associated only to the infra container that means there is no way to get the network metrics.

I guess this will have to wait for the new metrics APIs.

@vishh Does that sound right to you or can you think of a simple way to work around this without waiting for new metrics APIs?

[vishh]

SO we pull all the containers from cadvisor already. We filter out the
system containers using special labels. We can similarly filter out pod
containers.

Infact, we can get avoid accessing kubelet APIs completely, if we had a map
between kube pod container name to docker container ID.

Look here:
https://github.com/kubernetes/heapster/blob/master/sources/datasource/kubelet.go#L180

On Fri, Dec 4, 2015 at 8:42 AM, Jimmi Dyson notifications@github.com
wrote:

So this is a bit more involved than I thought it would be. AFAIK there is
no way to query the infra container in a pod with the current kubelet API &
as the network metrics are associated only to the infra container that
means there is no way to get the network metrics.

I guess this will have to wait for the new metrics APIs.

@vishh https://github.com/vishh Does that sound right to you or can you
think of a simple way to work around this without waiting for new metrics
APIs?

—
Reply to this email directly or view it on GitHub
#368 (comment)
.

[simon3z]

@vishh @jimmidyson right now stats are per-container, any idea on how to represent these per-pod network metrics? I think you mentioned the problem here above but I didn't understand the conclusion.

You mentioned the pod infra container, are you thinking to expose/collect them (with their network metrics) and then leave to external applications to do the grouping? (It should properly reference the pod uid, etc.).

[jimmidyson]

I am erring on waiting until the new metrics APIs are finalised as this will be pod based rather than container based. However if this is a requirement in the near future then we should probably get this added to existing APIs. This will be a bit trickier (& dirtier) to do but should be doable.

[simon3z]

@jimmidyson with regard to openshift (openshift-metrics) when is expected the new API to be finalized? Is it for 3.2? We expected to have network metrics in 3.1.z. cc @smarterclayton to keep me honest on the expectancy.

[smarterclayton]

We would certainly like to have network metrics, since it can be a
significant bit of info for administrators. How dirty and ugly do we think
it will be?

On Tue, Dec 29, 2015 at 4:37 PM, Federico Simoncelli <
notifications@github.com> wrote:

@jimmidyson https://github.com/jimmidyson with regard to openshift
(openshift-metrics) when is expected the new API to be finalized? Is it for
3.2? We expected to have network metrics in 3.1.z. cc @smarterclayton
https://github.com/smarterclayton to keep me honest on the expectancy.

—
Reply to this email directly or view it on GitHub
#368 (comment)
.

[jimmidyson]

The problem is that the infrastructure container isn't included in the pod status retrievable from the kube API (the API server is unaware of the infrastructure container from what I can see). The way that Heapster currently works is to get container info for each pod from API server then request stats for each container from the appropriate kubelet (of course this is really inefficient & addressed in the ongoing rewrite of heapster AFAIK).

Currently the infra container is totally ignored by heapster (https://github.com/kubernetes/heapster/blob/master/sources/datasource/kubelet.go#L180) as it's labelled as a member of a kubernetes pod, but not known to the API server. I cannot see anything else available that identifies it as the infra container for a pod, other than the format of the container name & the network mode (normal pod containers have a network mode of container:<containerID>).

The current kubelet APIs only allow querying for a) a container of a pod (which the infra container isn't listed as so is not queryable this way); b) a raw container via a Docker ID (no way to know); c) by getting all raw containers, extracting pod name from the container name (follows known regexp ATM) & adding this to pod info somehow. So c is the only option I can see, but it seems pretty fragile (no way to identify it's actually an infra container that I can think of).

How does that sound? Any other ideas? Dirty & fragile but without a major rework (the Heapster rewrite & new metrics APIs) this is the only option I can see. I would prefer to wait for the updated versions as I said before but if it's wanted I can probably hack it together.

[jimmidyson]

So looking again it seems the infra container name is hard coded to POD (https://github.com/kubernetes/kubernetes/blob/b9cfab87e33ea649bdd13a1bd243c502d76e5d22/pkg/kubelet/leaky/leaky.go#L24 , https://github.com/kubernetes/kubernetes/blob/810181fb7b4e969d1c1aa84f32377ef1b65f8c65/pkg/kubelet/dockertools/docker.go#L41). Of course this is terrible & fragile but would simplify finding infra containers & pairing them to a pod. Not at all happy with this though.

[timstclair]

The kubelet-metrics API was finally merged
kubernetes/kubernetes#18544, and includes what
you're looking for (pod level network stats). There is a little more work
to do on the metrics (most notably filesystem & volume metrics), but I can
start implementing what we have. This shouldn't be too much work, nor
should it see much push back (compared with the API), so I'm optimistic
that I can get an initial implementation out by the EOW, or by next week at
the latest.

On Mon, Jan 4, 2016 at 10:14 AM, Jimmi Dyson notifications@github.com
wrote:

So looking again it seems the infra container name is hard coded to POD (
https://github.com/kubernetes/kubernetes/blob/b9cfab87e33ea649bdd13a1bd243c502d76e5d22/pkg/kubelet/leaky/leaky.go#L24
,
https://github.com/kubernetes/kubernetes/blob/810181fb7b4e969d1c1aa84f32377ef1b65f8c65/pkg/kubelet/dockertools/docker.go#L41).
Of course this is terrible & fragile but would simplify finding infra
containers & pairing them to a pod. Not at all happy with this though.

—
Reply to this email directly or view it on GitHub
#368 (comment)
.