Docker-Fluentd: the Container to Log Other Containers' Logs

Hosted on Docker Hub

This container is extremely ECS-friendly, although it may be used for other environments.

Supported Environment Variables

Environment Variable	Purpose
DEBUG	Set to true if local development or non EC2 instance host
LOG_ENVIRONMENT	Logging environment name (e.g. QA or Production). This will be a field in the resulting log entries. Defaults to development if not set
AWS_REGION	Set AWS region for CloudWatch. Defaults to us-east-1 if not set
AWS_ACCESS_KEY_ID	AWS Access Key (Optimally, you can instead use Instance Profiles
AWS_SECRET_ACCESS_KEY	AWS Secret Access Key (Optimally, you can instead use Instance Profiles

What

By running this container with the following command, one can aggregate the logs of Docker containers running on the same host:

docker run -d --restart=always -e "LOG_ENVIRONMENT=qa" -v /var/lib/docker/containers:/var/lib/docker/containers -v /var/run/docker.sock:/var/run/docker.sock:ro --name=fluentd nritholtz/docker-fluentd

The container logs will forwarded to CloudWatch, with the log_group_name as the Task definition's family, followed by the set LOG_ENVIRONMENT environment variable. The log_stream_name will be the ECS container's name, followed by the container's ID (this is necessary due to issues with having more than 1 source for a single log_stream, e.g. the stored sequence parameter).

The outputted entries looks exactly like Docker's JSON formatted logs, except it will contain additional fields depending on which environment you are using for starting up your other Docker containers.

The following scenarios are supported and evaluated in the order that they appear:

Environment	Results
ECS	Log entries will use the running ECS Task Definition's Family and container name for the container_project and container_name log fields, respectively. In addition, a field task_definition_version will be added to the entries as well.
Docker Compose	Log entries will use the running Docker Compose's Project and Service for the container_project and container_name log fields, respectively.
Default	If container does not fall within either of the previous categories, will not add either of the fields.

Docker Compose example

{"log":"2015-10-15T11:44:27Z 54 TID-gomn1ol8g Appworker/Worker/4cab697c-7e67-40ee-a443-b8f52a034183 INFO: completed in: 232.16888 ms\n","stream":"stdout","container_image":"app_web","container_id":"a8c0128c0b47d5a7a5ed702be3c5f57cd62dc73371d5d7743b63a49ac1e09074","container_project":"app","container_service":"worker","time": "2015-10-23T13:00:42Z","ip_address": "10.212.151.112"}

How

docker-fluentd uses Fluentd inside to tail log files that are mounted on /var/lib/docker/containers/<CONTAINER_ID>/<CONTAINER_ID>-json.log. It uses the tail input plugin to tail JSON-formatted log files that each Docker container emits.

Then, Fluentd adds the additional fields using the included record_transformer plugin, before writing out to local files using the file output plugin. It will also using the Fluentd CloudWatch Plugin when sending logs to CloudWatch.

In addition, docker-fluentd registers any new containers started from the Docker service on the same host, by using Docker Gen. All of which are managed by a supervisord process.

Fluentd has a lot of plugins and can probably support your data output destination.