DEPRECATED: Please use a more robust solution, such as fluentd on Kubernetes. This repository is no longer maintained. It is archived as an example for future users.
docklogbeat
is a tool which combines filebeat
and docker-gen to transport data
from Docker container logs to Logstash and Elasticsearch.
It is intended as a more reliable replacement for logspout.
The implementation was inspired by this blog post.
Run the docklogbeat
container on each physical host in your cluster. It will
watch for new Docker containers to be spawned. If a container has a specific
label, it will automatically begin shipping the logs to your Logstash or
Elasticsearch cluster.
An example:
$ docker run -d \
-v /var/lib/docker:/var/lib/docker:ro \
-v /var/run/docker.sock:/var/run/docker.sock \
-e LOGSTASH_HOSTS=logstash1.example.com:5044,logstash2.example.com:5044 \
docklogbeat
Any containers with the label docklogbeat=true
will have their Docker logs
sent to the configured Logstash / Elasticsearch cluster.
An example:
$ docker run -l docklogbeat=true ...
The docker-gen
tool takes about 2 seconds to notice new containers and
regenerate the filebeat
configuration. If your containers are short lived,
they may not have their log output sent to your Logstash / Elasticsearch cluster.
This container will read all log messages from Log Producing Containers when it
starts, similar to docker logs --tail=all
. If you need to restart docklogbeat
without producing duplicate messages, you must make sure to store
/var/lib/filebeat/registry
on a persistent volume.
This container automatically detects if it is being run on a Rancher
cluster and sets the DOCKLOGBEAT_REGISTRY_FILE_NAME
to a unique value based on the
hostname of the physical host running the container. This allows docklogbeat to be
restarted safely without producing duplicate messages.
These environment variables can be used to configure the docklogbeat
container.
LOGSTASH_HOSTS
- A comma-separated lists of the Elastic Beats servers which will receive logs.DOCKLOGBEAT_REGISTRY_FILE_NAME
- File to store the filebeat registry (default:/var/lib/filebeat/registry
).DOCKLOGBEAT_LABEL_KEY
- Container Filter Label Key (default:docklogbeat
).DOCKLOGBEAT_LABEL_VALUE
- Container Filter Label Value (default:true
).DOCKLOGBEAT_SCAN_FRECUENCY
- Container Scan Logs Frecuency (default:11s
).
docklogbeat
- must be set totrue
for docklogbeat to collect this container's logs.docklogbeat_document_type
- set thedocument_type
field for this container (optional).