no container data at vagrant precise64 #154

xh3b4sd · 2014-08-07T11:06:47Z

Using cAdvisor under my vagrant precise64 does not provide container data. All I see is this:

Navigating to /docker results in

Failed to get container "/docker" with error: json: cannot unmarshal number into Go value of type string

system info

$ uname -a
Linux precise64 3.2.0-23-generic #36-Ubuntu SMP Tue Apr 10 20:39:51 UTC 2012 x86_64 x86_64 x86_64 GNU/Linux
$ docker -v
Docker version 1.1.0, build 79812e3

$ vagrant -v
Vagrant 1.6.2

virtualbox: 4.3.12

Using cAdvisor under coreos-vagrant works as expected and I get information about docker container data. Does anybody has a clue what is going on here?

All the best

The text was updated successfully, but these errors were encountered:

vmarmol · 2014-08-07T15:48:06Z

A few questions to help with debugging:

What version of cAdvisor are you running btw?
How is it running, just a raw run or in a Docker container?
What does the cAdvisor log say?

vmarmol · 2014-08-07T15:48:36Z

You are at least getting machine-wide data, so it seems to be isolated to Docker containers.

monnand · 2014-08-07T15:58:19Z

According to the error message, it's trying to unmarshal some JSON data but failed. The only place I could find is in container/docker/handler.go, readLibcontainerConfig() method. It seems that some fields in docker's config.json was changed from string to number.

vmarmol · 2014-08-07T15:59:06Z

@monnand yeah that's what I'm thinking too

monnand · 2014-08-07T15:59:54Z

Seems like we are compiling against an old libcontainer. If @zyndiecate is using our official docker image, then that might be the reason?

xh3b4sd · 2014-08-07T18:07:34Z

Hey there. So I just run it like told in the readme. There you guys are using the latest tag.

sudo docker run \
  --volume=/var/run:/var/run:rw \
  --volume=/sys:/sys:ro \
  --volume=/var/lib/docker/:/var/lib/docker:ro \
  --publish=8080:8080 \
  --detach=true \
  --name=cadvisor \
  google/cadvisor:latest

xh3b4sd · 2014-08-07T18:16:25Z

A look in the logs is not that helpful for me. Requesting http://localhost:8080/containers/docker shows me the following in the container logs:

2014/08/07 18:14:44 Get(/docker); &{NumStats:60 NumSamples:60 CpuUsagePercentiles:[] MemoryUsagePercentiles:[]}
2014/08/07 18:14:44 Get(/); &{NumStats:60 NumSamples:60 CpuUsagePercentiles:[] MemoryUsagePercentiles:[]}
2014/08/07 18:14:44 Request took 2.010435ms

vmarmol · 2014-08-07T18:30:29Z

@zyndiecate I'd be interested in the start of the log as well as whether the log shows anything (outside of the requests)

vmarmol · 2014-08-07T18:31:35Z

Also, what are the contents of /var/lib/docker/execdriver/native? There should be some container dirs with a config inside, can you get the contents of one such container?

xh3b4sd · 2014-08-07T18:57:43Z

So here you go:

vagrant@precise64 $ docker ps -a
CONTAINER ID        IMAGE                       COMMAND             CREATED             STATUS              PORTS                    NAMES
6ccac23a5b6b        google/cadvisor:latest      /usr/bin/cadvisor   41 minutes ago      Up 41 minutes       0.0.0.0:8080->8080/tcp   cadvisor
9f3b63971fa2        busybox:buildroot-2014.02   /bin/sh             9 hours ago                                                      trusting_leakey
vagrant@precise64 $ docker logs 6ccac23a5b6b
2014/08/07 18:14:02 Machine: {NumCores:2 MemoryCapacity:1043816448}
2014/08/07 18:14:02 Version: {KernelVersion:3.2.0-23-generic ContainerOsVersion:Unknown DockerVersion:1.1.0 CadvisorVersion:0.2.1}
2014/08/07 18:14:02 Registering Docker factory
2014/08/07 18:14:02 Registering Raw factory
2014/08/07 18:14:02 Starting cAdvisor version: "0.2.1"
2014/08/07 18:14:02 About to serve on port 8080
2014/08/07 18:14:02 Using factory "raw" for container "/"
2014/08/07 18:14:02 Added container: / (aliases: [])
2014/08/07 18:14:02 Start housekeeping for container "/"
2014/08/07 18:14:02 Starting recovery of all containers
2014/08/07 18:14:02 Using factory "docker" for container "/docker"
2014/08/07 18:14:03 Added container: /docker (aliases: [])
2014/08/07 18:14:03 Start housekeeping for container "/docker"
2014/08/07 18:14:03 Using factory "raw" for container "/docker/6ccac23a5b6b2e0273ccde73b3000ac5461aa44b4af447d1d9a97b4983de1c1f"
2014/08/07 18:14:03 Added container: /docker/6ccac23a5b6b2e0273ccde73b3000ac5461aa44b4af447d1d9a97b4983de1c1f (aliases: [])
2014/08/07 18:14:03 Start housekeeping for container "/docker/6ccac23a5b6b2e0273ccde73b3000ac5461aa44b4af447d1d9a97b4983de1c1f"
2014/08/07 18:14:03 Recovery completed
2014/08/07 18:14:21 Get(/); &{NumStats:60 NumSamples:60 CpuUsagePercentiles:[] MemoryUsagePercentiles:[]}
2014/08/07 18:14:21 Request took 4.646685ms
2014/08/07 18:14:21 Api - Machine
2014/08/07 18:14:21 Request took 101.13us
2014/08/07 18:14:21 Get(/); &{NumStats:60 NumSamples:60 CpuUsagePercentiles:[] MemoryUsagePercentiles:[]}
2014/08/07 18:14:21 Request took 1.872023ms
2014/08/07 18:14:22 Api - Container(/)
2014/08/07 18:14:22 Get(/); &{NumStats:60 NumSamples:0 CpuUsagePercentiles:[] MemoryUsagePercentiles:[]}
2014/08/07 18:14:22 Request took 3.457616ms
2014/08/07 18:14:23 Api - Container(/)
2014/08/07 18:14:23 Get(/); &{NumStats:60 NumSamples:0 CpuUsagePercentiles:[] MemoryUsagePercentiles:[]}
2014/08/07 18:14:23 Request took 2.697574ms
2014/08/07 18:14:24 Api - Container(/)
2014/08/07 18:14:24 Get(/); &{NumStats:60 NumSamples:0 CpuUsagePercentiles:[] MemoryUsagePercentiles:[]}
2014/08/07 18:14:24 Request took 3.152968ms
2014/08/07 18:14:25 Api - Container(/)
2014/08/07 18:14:25 Get(/); &{NumStats:60 NumSamples:0 CpuUsagePercentiles:[] MemoryUsagePercentiles:[]}
2014/08/07 18:14:25 Request took 3.545393ms
2014/08/07 18:14:26 Api - Container(/)
2014/08/07 18:14:26 Get(/); &{NumStats:60 NumSamples:0 CpuUsagePercentiles:[] MemoryUsagePercentiles:[]}
2014/08/07 18:14:26 Request took 3.095489ms
2014/08/07 18:14:27 Api - Container(/)
2014/08/07 18:14:27 Get(/); &{NumStats:60 NumSamples:0 CpuUsagePercentiles:[] MemoryUsagePercentiles:[]}
2014/08/07 18:14:27 Request took 3.964282ms
2014/08/07 18:14:28 Api - Container(/)
2014/08/07 18:14:28 Get(/); &{NumStats:60 NumSamples:0 CpuUsagePercentiles:[] MemoryUsagePercentiles:[]}
2014/08/07 18:14:28 Request took 3.159696ms
2014/08/07 18:14:29 Api - Container(/)
2014/08/07 18:14:29 Get(/); &{NumStats:60 NumSamples:0 CpuUsagePercentiles:[] MemoryUsagePercentiles:[]}
2014/08/07 18:14:29 Request took 3.628381ms
2014/08/07 18:14:30 Get(/docker); &{NumStats:60 NumSamples:60 CpuUsagePercentiles:[] MemoryUsagePercentiles:[]}
2014/08/07 18:14:30 Get(/); &{NumStats:60 NumSamples:60 CpuUsagePercentiles:[] MemoryUsagePercentiles:[]}
2014/08/07 18:14:30 Request took 1.368883ms
2014/08/07 18:14:44 Get(/docker); &{NumStats:60 NumSamples:60 CpuUsagePercentiles:[] MemoryUsagePercentiles:[]}
2014/08/07 18:14:44 Get(/); &{NumStats:60 NumSamples:60 CpuUsagePercentiles:[] MemoryUsagePercentiles:[]}
2014/08/07 18:14:44 Request took 2.010435ms
2014/08/07 18:15:05 Housekeeping(/) took 221.027482ms
2014/08/07 18:17:41 Housekeeping(/) took 150.450656ms
2014/08/07 18:19:55 Housekeeping(/) took 125.275463ms
2014/08/07 18:21:33 Housekeeping(/) took 134.988726ms
2014/08/07 18:21:59 Housekeeping(/) took 131.300502ms
2014/08/07 18:22:32 Housekeeping(/) took 204.394238ms
2014/08/07 18:23:30 Housekeeping(/) took 222.348049ms
2014/08/07 18:24:21 Housekeeping(/) took 227.333215ms
2014/08/07 18:26:54 Housekeeping(/) took 185.973827ms
2014/08/07 18:27:16 Housekeeping(/) took 175.716752ms
2014/08/07 18:30:22 Housekeeping(/) took 218.857861ms
2014/08/07 18:32:11 Housekeeping(/) took 211.102751ms
2014/08/07 18:32:20 Housekeeping(/) took 221.949129ms
2014/08/07 18:34:09 Housekeeping(/) took 176.068546ms
2014/08/07 18:41:07 Housekeeping(/) took 204.506564ms
2014/08/07 18:42:06 Housekeeping(/) took 198.938433ms
2014/08/07 18:45:02 Housekeeping(/) took 218.0303ms
2014/08/07 18:53:50 Housekeeping(/) took 131.078457ms
vagrant@precise64 $ sudo tree /var/lib/docker/execdriver/native
/var/lib/docker/execdriver/native
|-- 4280483933d311ccd673f0ca72a49221b6e8d2b59651023c2e9fff7a9d72790f
|   |-- container.json
|   `-- state.json
|-- 6ccac23a5b6b2e0273ccde73b3000ac5461aa44b4af447d1d9a97b4983de1c1f
|   |-- container.json
|   `-- state.json
|-- 82b944bf8a08598e96fd969aec383a84a938bb59b2b9141b5a15ad855e6e4a8b
|   |-- container.json
|   `-- state.json
|-- 9f3b63971fa25207405be6dd272dc1e0a22eb9b6852ee8133a0c40a31473aab0
|   |-- container.json
|   `-- state.json
`-- ef581301d0aa60a31db4f05f6531ea2d76f370e5b58d0f654770e2063f4d866f
    |-- container.json
    `-- state.json

5 directories, 10 files

vmarmol · 2014-09-09T17:13:37Z

@zyndiecate we pushed 0.3.0 wanna give that a shot? It adds support for a lot of systems so it may work for you now.

rjnagal · 2014-09-12T20:11:27Z

I think this works now.

denderello · 2014-09-18T12:57:30Z

Updating to 0.3.0 worked for us on local machines. Now we are also able to see network traffic information. Unfortunately the network information is only displayed on our local vagrant machines while it is not on our CoreOS cluster. All other data like CPU and memory usage is displayed.

Any suggestion why we can't see the network information on CoreOS?

denderello · 2014-09-18T13:12:37Z

As addition, these are the versions used on the Vagrant machine and the CoreOS cluster.

Vagrant Machine:

cAdvisor Image: 0.3.0
Docker: 1.2.0

CoreOS Cluster:

CoreOS: 410.0.0
cAdvisor Image: 0.3.0
Docker: 1.1.2

vmarmol · 2014-09-18T15:30:46Z

@denderello, is it unfortunately a known issue. PR #220 fixes that, but due to a libcontainer bug we can't merge that until we fix the libcontainer bug :( we're working on getting that done.

Issue #143 is tracking that work btw.

denderello · 2014-09-18T19:25:31Z

Thanks for catching up so quick on this, I will keep an eye on #220 then. :)

Do you have any hint to point me to on the libcontainer bug blocking this? Can't find anything related in the libcontainer issues or PRs and #143's comments.

vmarmol · 2014-09-22T15:50:27Z

Just filed https://github.com/docker/libcontainer/issues/198 to track that.

rjnagal closed this as completed Sep 12, 2014

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

no container data at vagrant precise64 #154

no container data at vagrant precise64 #154

xh3b4sd commented Aug 7, 2014

vmarmol commented Aug 7, 2014

vmarmol commented Aug 7, 2014

monnand commented Aug 7, 2014

vmarmol commented Aug 7, 2014

monnand commented Aug 7, 2014

xh3b4sd commented Aug 7, 2014

xh3b4sd commented Aug 7, 2014

vmarmol commented Aug 7, 2014

vmarmol commented Aug 7, 2014

xh3b4sd commented Aug 7, 2014

vmarmol commented Sep 9, 2014

rjnagal commented Sep 12, 2014

denderello commented Sep 18, 2014

denderello commented Sep 18, 2014

vmarmol commented Sep 18, 2014

denderello commented Sep 18, 2014

vmarmol commented Sep 22, 2014

no container data at vagrant precise64 #154

no container data at vagrant precise64 #154

Comments

xh3b4sd commented Aug 7, 2014

vmarmol commented Aug 7, 2014

vmarmol commented Aug 7, 2014

monnand commented Aug 7, 2014

vmarmol commented Aug 7, 2014

monnand commented Aug 7, 2014

xh3b4sd commented Aug 7, 2014

xh3b4sd commented Aug 7, 2014

vmarmol commented Aug 7, 2014

vmarmol commented Aug 7, 2014

xh3b4sd commented Aug 7, 2014

vmarmol commented Sep 9, 2014

rjnagal commented Sep 12, 2014

denderello commented Sep 18, 2014

denderello commented Sep 18, 2014

vmarmol commented Sep 18, 2014

denderello commented Sep 18, 2014

vmarmol commented Sep 22, 2014