GET /containers/(id)/stats broken in 1.7?

[ejholmes]

Description of problem:
Has the spec for this endpoint been changed in docker 1.7? Using the latest version of https://github.com/fsouza/go-dockerclient against https://master.dockerproject.com/ results in Unrecognized input header being returned when calling client.Stats(). Works correctly in 1.6. I can dig deeper if this isn't already a known issue.

$ docker version
Client version: 1.6.0
Client API version: 1.18
Go version (client): go1.4.2
Git commit (client): 4749651
OS/Arch (client): darwin/amd64
Server version: 1.7.0-dev
Server API version: 1.19
Go version (server): go1.4.2
Git commit (server): 4bcfa47
OS/Arch (server): linux/amd64

$ docker info
Containers: 5
Images: 12
Storage Driver: aufs
 Root Dir: /mnt/sda/var/lib/docker/aufs
 Backing Filesystem: extfs
 Dirs: 22
 Dirperm1 Supported: true
Execution Driver: native-0.2
Kernel Version: 3.16.1-tinycore64
Operating System: Boot2Docker 1.2.0 (TCL 5.3); 3.16.1-config-file : e75396e - Fri Aug 22 06:45:30 UTC 2014
CPUs: 1
Total Memory: 495.1 MiB
Name: boot2docker
ID: 4MLS:BGLV:7FJH:UIG7:EOH5:FEME:W6LK:535D:JGRT:JXQI:JZUH:ROXH
Debug mode (server): true
Debug mode (client): false
Fds: 81
Goroutines: 89
System Time: Sat May 23 05:10:21 UTC 2015
EventsListeners: 13
Init Path: /usr/local/bin/docker
Docker Root Dir: /mnt/sda/var/lib/docker
Http Proxy:
Https Proxy:
No Proxy:
Username: ejholmes
Registry: [https://index.docker.io/v1/]

$ uname -a
Linux boot2docker 3.16.1-tinycore64 #1 SMP Fri Aug 22 06:40:10 UTC 2014 x86_64 GNU/Linux

Environment details (AWS, VirtualBox, physical, etc.):
Able to reproduce in boot2docker and also on AWS EC2 instances.

How reproducible:
100%

Steps to Reproduce:

1. Run docker 1.7 daemon
2. Run a container: docker run -d alpine:3.1 /bin/sh -c "while true; do echo hello; sleep 1; done"
3. Issue a request to stream the stats for the container with telnet:

$ telnet <docker host> <port>
Trying ::1...
telnet: connect to address ::1: Connection refused
Trying 127.0.0.1...
Connected to localhost.
Escape character is '^]'.
GET /containers/4cda41c21e8d0e41e6da635b12f9e8e13b61f4dc62f7b65a05ff9d60d5393fdb/stats HTTP/1.1

Actual Results:

One stat message is streamed, then a 0 is sent and no more stats are sent after that:

HTTP/1.1 200 OK
Date: Sat, 23 May 2015 05:06:25 GMT
Content-Type: text/plain; charset=utf-8
Transfer-Encoding: chunked

560
{"read":"2015-05-23T05:06:24.831430951Z","network":{"rx_bytes":0,"rx_packets":0,"rx_errors":0,"rx_dropped":0,"tx_bytes":0,"tx_packets":0,"tx_errors":0,"tx_dropped":0},"cpu_stats":{"cpu_usage":{"total_usage":847371027,"percpu_usage":[847371027],"usage_in_kernelmode":50000000,"usage_in_usermode":170000000},"system_cpu_usage":1878570000000,"throttling_data":{"periods":0,"throttled_periods":0,"throttled_time":0}},"memory_stats":{"usage":163840,"max_usage":1179648,"stats":{"active_anon":65536,"active_file":0,"cache":4096,"hierarchical_memory_limit":18446744073709551615,"hierarchical_memsw_limit":18446744073709551615,"inactive_anon":0,"inactive_file":0,"mapped_file":0,"pgfault":58722,"pgmajfault":0,"pgpgin":19824,"pgpgout":19800,"rss":94208,"rss_huge":0,"swap":0,"total_active_anon":65536,"total_active_file":0,"total_cache":4096,"total_inactive_anon":0,"total_inactive_file":0,"total_mapped_file":0,"total_pgfault":58722,"total_pgmajfault":0,"total_pgpgin":19824,"total_pgpgout":19800,"total_rss":94208,"total_rss_huge":0,"total_swap":0,"total_unevictable":0,"total_writeback":0,"unevictable":0,"writeback":0},"failcnt":0,"limit":519147520},"blkio_stats":{"io_service_bytes_recursive":[],"io_serviced_recursive":[],"io_queue_recursive":[],"io_service_time_recursive":[],"io_wait_time_recursive":[],"io_merged_recursive":[],"io_time_recursive":[],"sectors_recursive":[]}}

0

Expected Results:

I expect a continuous stream of stats to be sent, like below, which was captured against docker 1.6.

HTTP/1.1 200 OK
Content-Type: application/json
Date: Sat, 23 May 2015 05:03:14 GMT
Transfer-Encoding: chunked

582
{"read":"2015-05-23T05:03:14.412287818Z","network":{"rx_bytes":648,"rx_packets":8,"rx_errors":0,"rx_dropped":0,"tx_bytes":648,"tx_packets":8,"tx_errors":0,"tx_dropped":0},"cpu_stats":{"cpu_usage":{"total_usage":327891630,"percpu_usage":[61105665,115203059,53758803,97824103],"usage_in_kernelmode":130000000,"usage_in_usermode":20000000},"system_cpu_usage":75857340000000,"throttling_data":{"periods":0,"throttled_periods":0,"throttled_time":0}},"memory_stats":{"usage":454656,"max_usage":655360,"stats":{"active_anon":61440,"active_file":4096,"cache":8192,"hierarchical_memory_limit":18446744073709551615,"hierarchical_memsw_limit":18446744073709551615,"inactive_anon":0,"inactive_file":0,"mapped_file":0,"pgfault":6277,"pgmajfault":0,"pgpgin":2199,"pgpgout":2161,"rss":147456,"rss_huge":0,"swap":0,"total_active_anon":61440,"total_active_file":4096,"total_cache":8192,"total_inactive_anon":0,"total_inactive_file":0,"total_mapped_file":0,"total_pgfault":6277,"total_pgmajfault":0,"total_pgpgin":2199,"total_pgpgout":2161,"total_rss":147456,"total_rss_huge":0,"total_swap":0,"total_unevictable":0,"total_writeback":0,"unevictable":0,"writeback":0},"failcnt":0,"limit":2105860096},"blkio_stats":{"io_service_bytes_recursive":[],"io_serviced_recursive":[],"io_queue_recursive":[],"io_service_time_recursive":[],"io_wait_time_recursive":[],"io_merged_recursive":[],"io_time_recursive":[],"sectors_recursive":[]}}

582
{"read":"2015-05-23T05:03:15.412531749Z","network":{"rx_bytes":648,"rx_packets":8,"rx_errors":0,"rx_dropped":0,"tx_bytes":648,"tx_packets":8,"tx_errors":0,"tx_dropped":0},"cpu_stats":{"cpu_usage":{"total_usage":329912723,"percpu_usage":[61402741,116102576,53758803,98648603],"usage_in_kernelmode":130000000,"usage_in_usermode":20000000},"system_cpu_usage":75861300000000,"throttling_data":{"periods":0,"throttled_periods":0,"throttled_time":0}},"memory_stats":{"usage":401408,"max_usage":655360,"stats":{"active_anon":53248,"active_file":4096,"cache":8192,"hierarchical_memory_limit":18446744073709551615,"hierarchical_memsw_limit":18446744073709551615,"inactive_anon":0,"inactive_file":0,"mapped_file":0,"pgfault":6326,"pgmajfault":0,"pgpgin":2216,"pgpgout":2174,"rss":163840,"rss_huge":0,"swap":0,"total_active_anon":53248,"total_active_file":4096,"total_cache":8192,"total_inactive_anon":0,"total_inactive_file":0,"total_mapped_file":0,"total_pgfault":6326,"total_pgmajfault":0,"total_pgpgin":2216,"total_pgpgout":2174,"total_rss":163840,"total_rss_huge":0,"total_swap":0,"total_unevictable":0,"total_writeback":0,"unevictable":0,"writeback":0},"failcnt":0,"limit":2105860096},"blkio_stats":{"io_service_bytes_recursive":[],"io_serviced_recursive":[],"io_queue_recursive":[],"io_service_time_recursive":[],"io_wait_time_recursive":[],"io_merged_recursive":[],"io_time_recursive":[],"sectors_recursive":[]}}

Additional info:

[thaJeztah]

I'm not aware of anything changed in this, apart from adding an optional parameter to disable streaming.
Could you add the information requested above?

/cc @cpuguy83 think you made the last changes to the stats?

[cpuguy83]

Nothing other than a new param to disable streaming.

[thaJeztah]

That's what I thought, thanks

@ejholmes if you could add the requested information and a way to reproduce this, that would be very welcome!

[ejholmes]

@thaJeztah cool! I'll try to get a repro case for this. Thanks!

[ejholmes]

@thaJeztah @cpuguy83 updated the description with info and steps to reproduce. Seems to be 100% reproducible with docker 1.7-dev; I get one set of stats sent to me, then it just stops streaming.

[ejholmes]

Ah! Browsing the source for the container stats endpoint on master and noticed the new stream param. Works as expected once that param is provided, so I'll try to get go-dockerclient to add this by default, since it shouldn't have any effect against docker 1.6.

Sorry for the trouble!

[thaJeztah]

@ejholmes thanks for researching this! Actually, I am going to re-open; the stream parameter should be optional, not a required parameter, see the description;

stream – 1/True/true or 0/False/false, pull stats once then disconnect. Default true

@cpuguy83 This looks like a bug/regression to me, I haven't tested yet, but this doesn't sound right

[cpuguy83]

Yeah, I bet that happened with engine removal.

[thaJeztah]

@cpuguy83 will you look at it? Or @ejholmes are you interested in creating a PR to fix this (and a regression test)?

[runcom]

so problem is boolValue evaluates "" to false

[thaJeztah]

@runcom yup

[thaJeztah]

@ejholmes we've discussed this briefly on IRC and think it's important to have this fixed ASAP.

@cpuguy83 or @runcom will pick this up so that it'll be fixed soon.

Thanks for reporting!! And if you feel like contributing for other issues, you're more than welcome!

[cpuguy83]

Should probably have "boolValue" accept a default value.

[runcom]

@cpuguy83 yes I did a branch but without a default value for now because I talked with @tiborvass on IRC and I'll come up with another PR changing how the stats endpoind behave (like having it no stream by default and adding stream=1 will stream or having nofollow query param that do the opposite because stats is the only endpoint to have a query param that is default'ed to true)

[thaJeztah]

@runcom It does make sense, but I'm hesitant on changing that so short before code/feature freeze. There might be quite some projects relying on the current behavior.