Per Container Metrics not availble

[sattishv]

Hi team,

We weren't able to view the per container metrics through Vector. Do we need to install any modules inside the container to get those metrics. We could see only the Total Container Memory usage not per container measures.

Our current installation has pcp installed on the host system and configured pmwebd as a simple HTTP file server, in order to server Vector itself to a web browser. Could you please review and let us know.

Vector Version: v1.1.0-3-gc851eac

Thanks
Sattish.

[spiermar]

@sattishv can you give us more details? You have no data in the per-container widgets?

[sattishv]

Hi Martin,

Thanks for the response. Yes, when we open Per container graphs it displays "No Data Available"

Sattish.

[spiermar]

Can you share your request/response from the _fetch request that is happening in the background?
Also, do you get a list of container Ids? Or is it just showing All?

[sattishv]

I am not sure on how to get the req/res will search for the _fetch request and let u know, we dont get the container ids its shows only ALL

[spiermar]

If you're on Chrome, just open the developer console and go to the Network tab.

[sattishv]

Hi ,

I could see series of fetch requests. Have listed the request headers for one of the request.Let me know if you any other details

GET /pmapi/1921743143/_fetch?names=containers.cgroup,containers.name,kernel.all.cpu.sys,kernel.all.cpu.user,hinv.ncpu,kernel.percpu.cpu.sys,kernel.percpu.cpu.user,kernel.all.runnable,kernel.all.load,network.interface.in.bytes,network.interface.out.bytes,network.tcpconn.established,network.tcpconn.time_wait,network.tcpconn.close_wait,network.interface.in.packets,network.interface.out.packets,network.tcp.retranssegs,network.tcp.timeouts,network.tcp.listendrops,network.tcp.fastretrans,network.tcp.slowstartretrans,network.tcp.syncretrans,mem.util.cached,mem.util.used,mem.util.free,mem.util.bufmem,mem.vmstat.pgfault,mem.vmstat.pgmajfault,kernel.all.pswitch,disk.dev.read,disk.dev.write,disk.dev.read_bytes,disk.dev.write_bytes,disk.dev.avactive,disk.dev.read_rawactive,disk.dev.write_rawactive,cgroup.cpuacct.usage,cgroup.blkio.all.throttle.io_serviced.read,cgroup.blkio.all.throttle.io_serviced.write,cgroup.memory.usage,cgroup.cpusched.shares,cgroup.cpusched.periods HTTP/1.1
Host: 10.25.129.111:44329
Connection: keep-alive
Accept: application/json, text/plain, /
Origin: http://kncntusrstage14007.qa.paypal.com
User-Agent: Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/52.0.2743.116 Safari/537.36
Referer: http://kncntusrstage14007.qa.paypal.com/index.html
Accept-Encoding: gzip, deflate, sdch
Accept-Language: en-US,en;q=0.8

[spiermar]

The response is what I mostly interested in.

[sattishv]

Hi
Sorry for the discommode. here is the response

{"timestamp":{"s":1471562912,"us":736254 }, "values":[{"pmid":251658262,"name":"kernel.all.cpu.sys","instances":[
{"instance":-1, "value":39399360 }]},
{"pmid":251658260,"name":"kernel.all.cpu.user","instances":[
{"instance":-1, "value":141824400 }]},
{"pmid":251658272,"name":"hinv.ncpu","instances":[
{"instance":-1, "value":2 }]},
{"pmid":251658242,"name":"kernel.percpu.cpu.sys","instances":[
{"instance":0, "value":19222780 },{"instance":1, "value":20176580 }]},
{"pmid":251658240,"name":"kernel.percpu.cpu.user","instances":[
{"instance":0, "value":67092550 },{"instance":1, "value":74731850 }]},
{"pmid":251660290,"name":"kernel.all.runnable","instances":[
{"instance":-1, "value":3 }]},
{"pmid":251660288,"name":"kernel.all.load","instances":[
{"instance":1, "value":0.029999999 },{"instance":5, "value":0.15000001 },{"instance":15, "value":0.13 }]},
{"pmid":251661312,"name":"network.interface.in.bytes","instances":[
{"instance":0, "value":15597289897 },{"instance":1, "value":36222246 },{"instance":2, "value":0 },{"instance":3, "value":1281572765 },{"instance":4, "value":8712 },{"instance":8, "value":648 },{"instance":9, "value":648 }]},
{"pmid":251661320,"name":"network.interface.out.bytes","instances":[
{"instance":0, "value":126577123534 },{"instance":1, "value":88277811 },{"instance":2, "value":0 },{"instance":3, "value":1281572765 },{"instance":4, "value":418 },{"instance":8, "value":1296 },{"instance":9, "value":648 }]},
{"pmid":251677697,"name":"network.tcpconn.established","instances":[
{"instance":-1, "value":29 }]},
{"pmid":251677702,"name":"network.tcpconn.time_wait","instances":[
{"instance":-1, "value":6 }]},
{"pmid":251677704,"name":"network.tcpconn.close_wait","instances":[
{"instance":-1, "value":0 }]},
{"pmid":251661313,"name":"network.interface.in.packets","instances":[
{"instance":0, "value":60195928 },{"instance":1, "value":357347 },{"instance":2, "value":0 },{"instance":3, "value":4499088 },{"instance":4, "value":120 },{"instance":8, "value":8 },{"instance":9, "value":8 }]},
{"pmid":251661321,"name":"network.interface.out.packets","instances":[
{"instance":0, "value":43733344 },{"instance":1, "value":365570 },{"instance":2, "value":0 },{"instance":3, "value":4499088 },{"instance":4, "value":5 },{"instance":8, "value":16 },{"instance":9, "value":8 }]},
{"pmid":251672637,"name":"network.tcp.retranssegs","instances":[
{"instance":-1, "value":98747 }]},
{"pmid":251712576,"name":"network.tcp.timeouts","instances":[
{"instance":-1, "value":425 }]},
{"pmid":251712549,"name":"network.tcp.listendrops","instances":[
{"instance":-1, "value":0 }]},
{"pmid":251712573,"name":"network.tcp.fastretrans","instances":[
{"instance":-1, "value":2894 }]},
{"pmid":251712575,"name":"network.tcp.slowstartretrans","instances":[
{"instance":-1, "value":7 }]},
{"pmid":251659269,"name":"mem.util.cached","instances":[
{"instance":-1, "value":2539400 }]},
{"pmid":251659265,"name":"mem.util.used","instances":[
{"instance":-1, "value":6033700 }]},
{"pmid":251659266,"name":"mem.util.free","instances":[
{"instance":-1, "value":78016 }]},
{"pmid":251659268,"name":"mem.util.bufmem","instances":[
{"instance":-1, "value":328576 }]},
{"pmid":251686928,"name":"mem.vmstat.pgfault","instances":[
{"instance":-1, "value":276208928 }]},
{"pmid":251686929,"name":"mem.vmstat.pgmajfault","instances":[
{"instance":-1, "value":1318888 }]},
{"pmid":251658253,"name":"kernel.all.pswitch","instances":[
{"instance":-1, "value":4368607612 }]},
{"pmid":251658244,"name":"disk.dev.read","instances":[
{"instance":0, "value":3063571 },{"instance":1, "value":497 }]},
{"pmid":251658245,"name":"disk.dev.write","instances":[
{"instance":0, "value":3248261 },{"instance":1, "value":6 }]},
{"pmid":251658278,"name":"disk.dev.read_bytes","instances":[
{"instance":0, "value":157252197 },{"instance":1, "value":1594 }]},
{"pmid":251658279,"name":"disk.dev.write_bytes","instances":[
{"instance":0, "value":62260780 },{"instance":1, "value":3 }]},
{"pmid":251658286,"name":"disk.dev.avactive","instances":[
{"instance":0, "value":2464272 },{"instance":1, "value":112 }]},
{"pmid":251658312,"name":"disk.dev.read_rawactive","instances":[
{"instance":0, "value":3202892 },{"instance":1, "value":128 }]},
{"pmid":251658313,"name":"disk.dev.write_rawactive","instances":[
{"instance":0, "value":54306280 },{"instance":1, "value":8 }]},
{"pmid":12624898,"name":"cgroup.cpuacct.usage","instances":[
{"instance":0, "value":205983596699473 },{"instance":1, "value":26733384299102 },{"instance":2, "value":1342501118123 },{"instance":3, "value":660633885282 },{"instance":4, "value":141215345760 },{"instance":5, "value":5998524552659 },{"instance":6, "value":5954871455884 },{"instance":7, "value":18581909680859 },{"instance":8, "value":18581909680859 },{"instance":9, "value":473958848 },{"instance":11, "value":169476234138533 },{"instance":12, "value":61796315764 },{"instance":13, "value":278563648438 },{"instance":14, "value":5522839869875 },{"instance":15, "value":2138633348232 },{"instance":18, "value":76733959948 },{"instance":19, "value":71566233338 },{"instance":20, "value":25059257554 },{"instance":21, "value":12404619580 },{"instance":22, "value":473958848 }]},
{"pmid":12629072,"name":"cgroup.memory.usage","instances":[
{"instance":0, "value":4832018432 },{"instance":1, "value":0 },{"instance":2, "value":0 },{"instance":3, "value":977965056 },{"instance":4, "value":8065024 },{"instance":5, "value":0 },{"instance":6, "value":301592576 },{"instance":7, "value":0 },{"instance":8, "value":103653376 },{"instance":9, "value":0 },{"instance":11, "value":0 },{"instance":12, "value":195485696 },{"instance":13, "value":0 },{"instance":14, "value":2359296 },{"instance":15, "value":12574720 },{"instance":16, "value":83804160 },{"instance":17, "value":69337088 },{"instance":18, "value":37167104 },{"instance":19, "value":1299599360 },{"instance":20, "value":8192 }]}]}

Thanks
Sattish

[spiermar]

It looks like the response is missing containers.cgroup and containers.name, so Vector is filtering out all containers because they are not on the container list. The list is empty because no Ids are being returned.
Something might have changed in the latest Docker version. I'll double-check and ping PCP if necessary.

[spiermar]

According to @sattishv

# pminfo --fetch containers.name containers.state.running

containers.name
Error: Unknown or illegal instance identifier

Since Vector checks if a container is in that list before adding it to the graphs, (ContainerMetadataService.containerIdExist(instance.key)), that's likely the issue.

@natoscott Looks like a similar issue to the one we had migrating from 1.9 to 1.10. Should I open a issue for PCP?

[sattishv]

Hi Martin,

Were you able to get any update from PCP team on this.

Sattish.

[lberk]

Just checked with docker v1.12 here (and pminfo -f containers works for me), what version of PCP is being used on that host?

[spiermar]

1.11.2-cs3 according to Sattish.

• Martin

On Thu, Aug 25, 2016 at 9:23 AM, Lukas Berk notifications@github.com
wrote:

Just checked with docker v1.12 here, what version of PCP is being used on
that host?

—
You are receiving this because you modified the open/close state.
Reply to this email directly, view it on GitHub
#135 (comment), or mute
the thread
https://github.com/notifications/unsubscribe-auth/ACGut9BcUhr-703WEn6R8gXnyaXhGmGoks5qjcGKgaJpZM4Jn-ja
.

[lberk]

@spiermar I'm not familiar with PCP version 1.11.2-cs3 however the config.v2.json fix that you were alluding to earlier is in pcp-3.11.2 and later.

[spiermar]

I mean Docker 1.11.

• Martin

On Thu, Aug 25, 2016 at 10:43 AM, Lukas Berk notifications@github.com
wrote:

@spiermar https://github.com/spiermar I'm not familiar with PCP version
1.11.2-cs3 however the config.v2.json fix that you were alluding to earlier
is in pcp-3.11.2 and later.

—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
#135 (comment), or mute
the thread
https://github.com/notifications/unsubscribe-auth/ACGut1v4F7cjO1-Tg4yT9tathMpOCAGpks5qjdQwgaJpZM4Jn-ja
.

[spiermar]

@sattishv, @lberk tested the latest Docker version and it's working with the current release of PCP. Can you confirm you're running PCP 3.11.2 or newer?

[sattishv]

Hi martin,

Sure will check and let you know soon

Thanks
Sattish.

On Tue, Aug 30, 2016 at 9:26 AM, Martin Spier notifications@github.com
wrote:

@sattishv https://github.com/sattishv, @lberk https://github.com/lberk
tested the latest Docker version and it's working with the current release
of PCP. Can you confirm you're running PCP 3.11.2 or newer?

—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
#135 (comment), or mute
the thread
https://github.com/notifications/unsubscribe-auth/AHGqmTbNrbBOaXRMHII1H8QtVMZD_e1uks5qlFmagaJpZM4Jn-ja
.

with regards,

Sattish
9940080070

[spiermar]

Assuming this was solved. Closing the issue.