Deploy Elastic Stack 5.2.0+ on swarm mode cluster, access container logs in elk.
Switch branches/tags
Nothing to show
Clone or download
Fetching latest commit…
Cannot retrieve the latest commit at this time.
Permalink
Failed to load latest commit information.
docs/zh
env_files/test
.gitignore
LICENSE
README.md
docker-compose.override.yml
docker-compose.yml
env.sh
logstash-shipper.yml

README.md

Docs

Introduction

Deploy Elastic Stack 5.2.0+ on swarm mode cluster, access container logs in elk.

+------------+            +------------+            +------------+
|  Dockerd   |            |  Dockerd   |            |  Dockerd   |
|  GELF UDP  |            |  GELF UDP  |            |  GELF UDP  |
+------------+            +------------+            +------------+
      |                         |                         |
+------------+            +------------+            +------------+
|  Logstash  |            |  Logstash  |            |  Logstash  |
|  Shipper   |            |  Shipper   |            |  Shipper   |
+------------+            +------------+            +------------+
      |                         |                         |
      |                   +------------+                  |
      |___________________|   Redis    |__________________|
                          |   Broker   |
                          +------------+
                                |
                          +------------+
                          |  Logstash  |
                          |  Indexer   |
                          +------------+
                                |
                        +----------------+
                        |  Elasticsearch |
                        +----------------+
                                |
                          +------------+
                          |   Kibana   |
                          +------------+

Requirements

Usage

  • Add new environment files, eg. env_files/test

When deploy test example within this project, you only need to modify ELASTICSEARCH_DATA_PATH and REDIS_DATA_PATH these two variables in env_files/test/common.env file.

  • Create the .env file used by docker-compose
./env.sh test

Deploy elk stack

  • Create the elk stack compose file
docker-compose config > elk-stack.yml
  • Deploy elk stack to swarm mode cluster
docker stack deploy -c elk-stack.yml $COMPOSE_PROJECT_NAME

Deploy logstash shipper

  • Create the logstash shipper compose file
docker-compose -f logstash-shipper.yml config > elk-logstash-shipper.yml
  • Deploy logstash shipper to every node in swarm mode cluster
docker stack deploy -c elk-logstash-shipper.yml elk-logstash

Deployment test

  • Send some container logs to logstash shipper by gelf logging drivers
docker run --rm --log-driver gelf --log-opt gelf-address=udp://127.0.0.1:9500 -d alpine echo hello world
  • Browser to Kibana pages
http://swarm-node-host:5601
  • Configure an index pattern with logstash-* in Kibana, then select the time range of today

Tips

sudo sysctl -w vm.max_map_count=262144
  • You can use other volume plugins by fork this project then modify the driver and driver_opts under the top-level volumes key

References

License