willfarrell/filebeat

Official Image: https://github.com/elastic/beats-docker. Note uses centos:7 as it's base. See #12

Filebeat: Analyze Log Files in Real Time Get ready for the next-generation Logstash Forwarder: Filebeat. Filebeat collects, pre-processes, and forwards log files from remote sources so they can be further enriched and combined with other data sources using Logstash. https://www.elastic.co/products/beats/filebeat

• documentation
• sample filebeat.yml

Supported tags and Dockerfile links

• 5.2.0, 5.2, 5, latest (Dockerfile)

• 5-stdin (Dockerfile)

• 5.1.2, 5.1 (Dockerfile)

• 1.3.0, 1.3, 1 (Dockerfile)

• 1.2.3, 1.2 (Dockerfile)

• 1.1.2, 1.1 (Dockerfile)

• 1.0.1, 1.0 (Dockerfile)

Run Examples

ENV

HOSTNAME: Server Name
LOGSTASH_HOST: Recommended name for Logstash Hostname [default=logstash]
LOGSTASH_PORT: Recommended name for Logstash Port [default=5044]

docker-cli

docker run \
	-v /path/to/filebeat.yml:/etc/filebeat/filebeat.yml \
	willfarrell/filebeat:5

Dockerfile

FROM willfarrell/filebeat:5
COPY filebeat.yml /filebeat.yml

docker-compose

version "3"

services:
  filebeat:
    image: willfarrell/filebeat:5
    #command: "filebeat -e -c /etc/filebeat/filebeat.yml"
    environment:
      HOSTNAME: "my-server"
      LOGSTASH_HOST: "192.168.99.100"
      LOGSTASH_PORT: "5044"
    volumes:
     - "./filebeat.yml:/etc/filebeat/filebeat.yml:rw"

stdin

There is also a wrapper image over the base image provided here that allows piping of docker stdout into filebeat.

ENV

HOSTNAME: Same as above
LOGSTASH_HOST: Logstash Hostname [default=logstash]
LOGSTASH_PORT: Logstash Port [default=5044]
STDIN_CONTAINER_LABEL: Container label to filter what containers to monitor. Set label to `true` to enable. Set ENV to `all` in ignore labels. [default=filebeat.stdin]

docker-cli

docker run \
	-v /path/to/filebeat.yml:/etc/filebeat/filebeat.yml \
	-v /var/run/docker.sock:/tmp/docker.sock \
	willfarrell/filebeat:5-stdin

Dockerfile

FROM willfarrell/filebeat:5-stdin
COPY filebeat.yml /filebeat.yml

docker-compose

version "3"

services:
  filebeat:
    image: willfarrell/filebeat:5-stdin
    #command: "filebeat -e -c /etc/filebeat/filebeat.yml"
    environment:
      HOSTNAME: "my-server"
      LOGSTASH_HOST: "192.168.99.100"
      LOGSTASH_PORT: "5044"
      STDIN_CONTAINER_LABEL: "all"
    volumes:
     - "./filebeat.yml:/etc/filebeat/filebeat.yml:rw"
     - "/var/run/docker.sock:/tmp/docker.sock:ro"

Filebeat

filebeat:
  prospectors:
    - input_type: "stdin"
      document_type: "filebeat-docker-logs"

Logstash

filter {

  if [type] == "filebeat-docker-logs" {

    grok {
      match => { 
        "message" => "\[%{WORD:containerName}\] %{GREEDYDATA:message_remainder}"
      }
    }

    mutate {
      replace => { "message" => "%{message_remainder}" }
    }

    mutate {
      remove_field => [ "message_remainder" ]
    }

  }

}

Testing

docker run --label filebeat.stdin=true -d alpine /bin/sh -c 'while true; do echo "Hello $(date)"; sleep 1; done'
docker build -t filebeat
docker run -v /var/run/docker.sock:/tmp/docker.sock filebeat

Contributors

• @bargenson - main logic for the docker stdin
• @gdubya - idea to use labels to choose what containers to log