docker plugin crash

[babaev]

If docker becomes unresponsive to requests docker plugin panics and crashes telegraf.

My case:

• Running telegraf in docker container
• Querying docker via docker.sock, provided via docker volume
• At some point due to heavy load/unknown reasons connection to docker fails with "context deadline exceeded". (Full message from telegraf in logs: "Error getting docker stats: An error occurred trying to connect: context deadline exceeded")
• telegraf crashes with panic:

goroutine 4571798 [running]:
panic(0x1072740, 0xc820012070)
    /usr/local/go/src/runtime/panic.go:464 +0x3e6
github.com/influxdata/telegraf/plugins/inputs/docker.(*Docker).gatherContainer(0xc8207c4200, 0xc820654b40, 0x40, 0xc820654b80, 0x1, 0x4, 0xc82059ce10, 0x47, 0xc82059ce60, 0x47, ...)
    /home/user/go/src/github.com/influxdata/telegraf/plugins/inputs/docker/docker.go:226 +0xa8c
github.com/influxdata/telegraf/plugins/inputs/docker.(*Docker).Gather.func1(0xc820519f80, 0xc8207c4200, 0x7ff06e6fbc00, 0xc82072d3c0, 0xc820654b40, 0x40, 0xc820654b80, 0x1, 0x4, 0xc82059ce10, ...)
    /home/user/go/src/github.com/influxdata/telegraf/plugins/inputs/docker/docker.go:117 +0xa6
created by github.com/influxdata/telegraf/plugins/inputs/docker.(*Docker).Gather
    /home/user/go/src/github.com/influxdata/telegraf/plugins/inputs/docker/docker.go:122 +0x5fb

The reason seems to be in plugins/inputs/docker/docker.go:226

r, err := d.client.ContainerStats(ctx, container.ID, false)
if err != nil {
    log.Printf("Error getting docker stats: %s\n", err.Error())
}
defer r.Close()

Since r might be nil if err != nil shouldn`t it just return err instead of printing it?

[zstyblik]

@babaev, which version of telegraf?

[babaev]

@zstyblik , 0.12.1-83-g40dccf5 but the code I mention is still in master

[sporokh]

I have the same issue running telegraf 0.13. Inputs like: system, CPU, MEM, DISK, DISKIO are working fine, but docker input plugin fails with context deadline exceeded

[sparrc]

I know a fix for this, but the workaround would be to extend the timeout of your docker plugin. Currently it defaults to 5s but you can set it higher.

just do:

[[inputs.docker]]
  ...
  timeout = "10s"

[babaev]

Let me make it clear. The problem is not that docker doesn`t respond - it may happen due to many reasons. My concern is that if it happens telegraf crashes. And it happens because here:

https://github.com/influxdata/telegraf/blob/master/plugins/inputs/docker/docker.go#L226

r might be nil in which case it is nil pointer dereference panic.

Modifying the check

if err != nil {
        log.Printf("Error getting docker stats: %s\n", err.Error())
    }

To something like

if err != nil {
        return fmt.Errorf("Error getting docker stats: %s\n", err.Error())
    }

Should make it work correctly.

[zstyblik]

@babaev correct.

[dvenza]

Please include this fix in the next point release.

[sparrc]

it will be in 0.13.1

[jvalencia]

Just hit this one. @sparrc , when does 13.1 come out?