/vector

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Already on GitHub? Sign in to your account

container disk i/o widgets are always reporting 0 #122

Closed
spiermar opened this Issue Apr 20, 2016 · 7 comments

Comments

Assignees
No one assigned
Labels
bug
Projects
None yet
Milestone
  v1.1
2 participants
@spiermar @natoscott
@spiermar
Contributor

spiermar commented Apr 20, 2016

No description provided.

@spiermar spiermar added this to the v1.1.0 milestone Apr 20, 2016

@spiermar spiermar added the bug label Apr 20, 2016

@spiermar
Contributor

spiermar commented Apr 20, 2016

@natoscott I still need to investigate this further, but the widgets are using cgroup.blkio.all.io_service_bytes and cgroup.blkio.all.io_serviced. Those are always returning 0, and that matches what I'm seeing in the pseudo-filesystem.

Didn't have much time to confirm this, but according to a datadog blog post:

On many systems, however, many of these pseudo-files only return zero values. In this case there are usually still two pseudo-files that work: blkio.throttle.io_service_bytes and blkio.throttle.io_serviced, which report total I/O bytes and operations, respectively. Contrary to their names, these numbers do not report throttled I/O but actual I/O bytes and ops.

This confirms what I'm seeing here. In PCP, all cgroup.blkio.all metrics are 0:

$ pminfo -f cgroup.blkio.all.io_serviced

cgroup.blkio.all.io_serviced.read
    inst [1 or "/"] value 0
    inst [2 or "/docker"] value 0
    inst [3 or "/docker/cdf81c35b5154a790cb8203a762335888ba8732688a68fad401da35b6a5d05c9"] value 0
    inst [4 or "/docker/168553e66eee8589a830e32fb6c9a1ca67dc2e960c7158513e15ac6c871380c6"] value 0
    inst [5 or "/docker/a470bdb21c48627e4ff9728143ac2eb20559fe96d9a6e59dd6d02400187936a2"] value 0
    inst [6 or "/docker/cb7d4cc4412ff8339de0c5de14304f50f7c06ec6b49982ff6d51643935961c3e"] value 0
    inst [7 or "/docker/14979371d9a0f0594e61a78c7bdc373fb7714b5510d61e7d2f791af8cf908c4a"] value 0
    inst [8 or "/docker/ec44e28c017665ed88b7e3eab402f1e787ad6d9ce82f5c6e6b63df69aca431a1"] value 0

cgroup.blkio.all.io_serviced.write
    inst [1 or "/"] value 0
    inst [2 or "/docker"] value 0
    inst [3 or "/docker/cdf81c35b5154a790cb8203a762335888ba8732688a68fad401da35b6a5d05c9"] value 0
    inst [4 or "/docker/168553e66eee8589a830e32fb6c9a1ca67dc2e960c7158513e15ac6c871380c6"] value 0
    inst [5 or "/docker/a470bdb21c48627e4ff9728143ac2eb20559fe96d9a6e59dd6d02400187936a2"] value 0
    inst [6 or "/docker/cb7d4cc4412ff8339de0c5de14304f50f7c06ec6b49982ff6d51643935961c3e"] value 0
    inst [7 or "/docker/14979371d9a0f0594e61a78c7bdc373fb7714b5510d61e7d2f791af8cf908c4a"] value 0
    inst [8 or "/docker/ec44e28c017665ed88b7e3eab402f1e787ad6d9ce82f5c6e6b63df69aca431a1"] value 0

cgroup.blkio.all.io_serviced.sync
    inst [1 or "/"] value 0
    inst [2 or "/docker"] value 0
    inst [3 or "/docker/cdf81c35b5154a790cb8203a762335888ba8732688a68fad401da35b6a5d05c9"] value 0
    inst [4 or "/docker/168553e66eee8589a830e32fb6c9a1ca67dc2e960c7158513e15ac6c871380c6"] value 0
    inst [5 or "/docker/a470bdb21c48627e4ff9728143ac2eb20559fe96d9a6e59dd6d02400187936a2"] value 0
    inst [6 or "/docker/cb7d4cc4412ff8339de0c5de14304f50f7c06ec6b49982ff6d51643935961c3e"] value 0
    inst [7 or "/docker/14979371d9a0f0594e61a78c7bdc373fb7714b5510d61e7d2f791af8cf908c4a"] value 0
    inst [8 or "/docker/ec44e28c017665ed88b7e3eab402f1e787ad6d9ce82f5c6e6b63df69aca431a1"] value 0

cgroup.blkio.all.io_serviced.async
    inst [1 or "/"] value 0
    inst [2 or "/docker"] value 0
    inst [3 or "/docker/cdf81c35b5154a790cb8203a762335888ba8732688a68fad401da35b6a5d05c9"] value 0
    inst [4 or "/docker/168553e66eee8589a830e32fb6c9a1ca67dc2e960c7158513e15ac6c871380c6"] value 0
    inst [5 or "/docker/a470bdb21c48627e4ff9728143ac2eb20559fe96d9a6e59dd6d02400187936a2"] value 0
    inst [6 or "/docker/cb7d4cc4412ff8339de0c5de14304f50f7c06ec6b49982ff6d51643935961c3e"] value 0
    inst [7 or "/docker/14979371d9a0f0594e61a78c7bdc373fb7714b5510d61e7d2f791af8cf908c4a"] value 0
    inst [8 or "/docker/ec44e28c017665ed88b7e3eab402f1e787ad6d9ce82f5c6e6b63df69aca431a1"] value 0

cgroup.blkio.all.io_serviced.total
    inst [1 or "/"] value 0
    inst [2 or "/docker"] value 0
    inst [3 or "/docker/cdf81c35b5154a790cb8203a762335888ba8732688a68fad401da35b6a5d05c9"] value 0
    inst [4 or "/docker/168553e66eee8589a830e32fb6c9a1ca67dc2e960c7158513e15ac6c871380c6"] value 0
    inst [5 or "/docker/a470bdb21c48627e4ff9728143ac2eb20559fe96d9a6e59dd6d02400187936a2"] value 0
    inst [6 or "/docker/cb7d4cc4412ff8339de0c5de14304f50f7c06ec6b49982ff6d51643935961c3e"] value 0
    inst [7 or "/docker/14979371d9a0f0594e61a78c7bdc373fb7714b5510d61e7d2f791af8cf908c4a"] value 0
    inst [8 or "/docker/ec44e28c017665ed88b7e3eab402f1e787ad6d9ce82f5c6e6b63df69aca431a1"] value 0

In the other hand, blkio.throttle.io_service_bytes and blkio.throttle.io_serviced actually have something.

$ cat blkio.throttle.io_serviced
9:0 Read 838
9:0 Write 293334825
9:0 Sync 146551
9:0 Async 293189112
9:0 Total 293335663
202:64 Read 409
202:64 Write 132152808
202:64 Sync 147881
202:64 Async 132005336
202:64 Total 132153217
202:48 Read 507
202:48 Write 132163746
202:48 Sync 147992
202:48 Async 132016261
202:48 Total 132164253
202:32 Read 400
202:32 Write 132175158
202:32 Sync 141900
202:32 Async 132033658
202:32 Total 132175558
202:16 Read 533
202:16 Write 132179910
202:16 Sync 142011
202:16 Async 132038432
202:16 Total 132180443
202:0 Read 22863
202:0 Write 16012089
202:0 Sync 9491219
202:0 Async 6543733
202:0 Total 16034952
7:7 Read 0
7:7 Write 0
7:7 Sync 0
7:7 Async 0
7:7 Total 0
7:6 Read 0
7:6 Write 0
7:6 Sync 0
7:6 Async 0
7:6 Total 0
7:5 Read 0
7:5 Write 0
7:5 Sync 0
7:5 Async 0
7:5 Total 0
7:4 Read 0
7:4 Write 0
7:4 Sync 0
7:4 Async 0
7:4 Total 0
7:3 Read 0
7:3 Write 0
7:3 Sync 0
7:3 Async 0
7:3 Total 0
7:2 Read 0
7:2 Write 0
7:2 Sync 0
7:2 Async 0
7:2 Total 0
7:1 Read 0
7:1 Write 0
7:1 Sync 0
7:1 Async 0
7:1 Total 0
7:0 Read 0
7:0 Write 0
7:0 Sync 0
7:0 Async 0
7:0 Total 0
1:15 Read 182
1:15 Write 0
1:15 Sync 0
1:15 Async 182
1:15 Total 182
1:14 Read 182
1:14 Write 0
1:14 Sync 0
1:14 Async 182
1:14 Total 182
1:13 Read 182
1:13 Write 0
1:13 Sync 0
1:13 Async 182
1:13 Total 182
1:12 Read 182
1:12 Write 0
1:12 Sync 0
1:12 Async 182
1:12 Total 182
1:11 Read 182
1:11 Write 0
1:11 Sync 0
1:11 Async 182
1:11 Total 182
1:10 Read 182
1:10 Write 0
1:10 Sync 0
1:10 Async 182
1:10 Total 182
1:9 Read 182
1:9 Write 0
1:9 Sync 0
1:9 Async 182
1:9 Total 182
1:8 Read 182
1:8 Write 0
1:8 Sync 0
1:8 Async 182
1:8 Total 182
1:7 Read 182
1:7 Write 0
1:7 Sync 0
1:7 Async 182
1:7 Total 182
1:6 Read 182
1:6 Write 0
1:6 Sync 0
1:6 Async 182
1:6 Total 182
1:5 Read 182
1:5 Write 0
1:5 Sync 0
1:5 Async 182
1:5 Total 182
1:4 Read 182
1:4 Write 0
1:4 Sync 0
1:4 Async 182
1:4 Total 182
1:3 Read 182
1:3 Write 0
1:3 Sync 0
1:3 Async 182
1:3 Total 182
1:2 Read 182
1:2 Write 0
1:2 Sync 0
1:2 Async 182
1:2 Total 182
1:1 Read 182
1:1 Write 0
1:1 Sync 0
1:1 Async 182
1:1 Total 182
1:0 Read 182
1:0 Write 0
1:0 Sync 0
1:0 Async 182
1:0 Total 182
Total 838046998

It is possible to include blkio.throttle.* ??
@natoscott
Contributor

natoscott commented Apr 20, 2016

It is possible to include blkio.throttle.* ??

Sure! I'll add it to the list to get done for pcp-3.11.2 (scheduled for next Friday)
@spiermar
Contributor

spiermar commented Apr 20, 2016

Great! I'm not sure why the different name yet, but it seems to be the case. Wanted to bring this up as soon as possible because of that. I'm planning to release a new version of Vector in the next week or so, with all the container features working properly and enabled by default. Tks!

natoscott added a commit to performancecopilot/pcp that referenced this issue Apr 21, 2016

@natoscott
pmdaproc: support cgroup blkio throttle throughput+iops metrics 
Add per-device and aggregate blkio throttle metrics requested by
the Vector folk working on containers support.  More details in:
Netflix/vector#122

Tests 730 and 731 already had test data in place, conveniently.
1274146
@natoscott
Contributor

natoscott commented Apr 21, 2016

@spiermar OK, the new metrics are there now in pcp git master branch ...

$ pminfo cgroup.blkio | grep throttle
cgroup.blkio.dev.throttle.io_service_bytes.read
cgroup.blkio.dev.throttle.io_service_bytes.write
cgroup.blkio.dev.throttle.io_service_bytes.sync
cgroup.blkio.dev.throttle.io_service_bytes.async
cgroup.blkio.dev.throttle.io_service_bytes.total
cgroup.blkio.dev.throttle.io_serviced.read
cgroup.blkio.dev.throttle.io_serviced.write
cgroup.blkio.dev.throttle.io_serviced.sync
cgroup.blkio.dev.throttle.io_serviced.async
cgroup.blkio.dev.throttle.io_serviced.total
cgroup.blkio.all.throttle.io_service_bytes.read
cgroup.blkio.all.throttle.io_service_bytes.write
cgroup.blkio.all.throttle.io_service_bytes.sync
cgroup.blkio.all.throttle.io_service_bytes.async
cgroup.blkio.all.throttle.io_service_bytes.total
cgroup.blkio.all.throttle.io_serviced.read
cgroup.blkio.all.throttle.io_serviced.write
cgroup.blkio.all.throttle.io_serviced.sync
cgroup.blkio.all.throttle.io_serviced.async
cgroup.blkio.all.throttle.io_serviced.total
@spiermar
Contributor

spiermar commented Apr 21, 2016

Cool! I'll give it a try tomorrow probably! Tks @natoscott
@spiermar
Contributor

spiermar commented Apr 26, 2016

Just deployed and it's working. Docker 1.10+ is also working fine.

@spiermar spiermar closed this Apr 26, 2016

@natoscott
Contributor

natoscott commented Apr 26, 2016

Just deployed and it's working. Docker 1.10+ is also working fine.

Great! Nice work @spiermar
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment