Skip to content

Commit 0ee9453

Browse files
digikindigikin
committed
Updated docker-compose section
Signed-off-by: Eric Stumbo <ericstumbo@student.purdueglobal.edu>
1 parent c3944c5 commit 0ee9453

File tree

4 files changed

+714
-0
lines changed

4 files changed

+714
-0
lines changed

‎container-deployment/docker-compose.md

Lines changed: 207 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,207 @@
1+
# Docker Logging via EFK (Elasticsearch + Fluentd + Kibana) Stack with Docker Compose
2+
3+
This article explains how to collect [Docker](https://www.docker.com/)
4+
logs to EFK (Elasticsearch + Fluentd + Kibana) stack. The example uses
5+
[Docker Compose](https://docs.docker.com/compose/) for setting up
6+
multiple containers.
7+
8+
![](/images/kibana5-screenshot.png)
9+
10+
11+
[Elasticsearch](https://www.elastic.co/products/elasticsearch) is an
12+
open source search engine known for its ease of use.
13+
[Kibana](https://www.elastic.co/products/kibana) is an open source Web
14+
UI that makes Elasticsearch user friendly for marketers, engineers and
15+
data scientists alike.
16+
17+
By combining these three tools EFK (Elasticsearch + Fluentd + Kibana) we
18+
get a scalable, flexible, easy to use log collection and analytics
19+
pipeline. In this article, we will set up 4 containers, each includes:
20+
21+
- [Apache HTTP Server](https://hub.docker.com/_/httpd/)
22+
- [Fluentd](https://hub.docker.com/r/fluent/fluentd/)
23+
- [Elasticsearch](https://hub.docker.com/_/elasticsearch/)
24+
- [Kibana](https://hub.docker.com/_/kibana/)
25+
26+
All of `httpd`'s logs will be ingested into Elasticsearch + Kibana, via
27+
Fluentd.
28+
29+
30+
## Prerequisites: Docker
31+
32+
Please download and install Docker / Docker Compose. Well, that's it :)
33+
34+
- [Docker Installation](https://docs.docker.com/engine/installation/)
35+
36+
## Step 0: prepare docker-compose.yml
37+
38+
First, please prepare `docker-compose.yml` for [Docker Compose](https://docs.docker.com/compose/overview/). Docker Compose is a
39+
tool for defining and running multi-container Docker applications.
40+
41+
With the YAML file below, you can create and start all the services (in
42+
this case, Apache, Fluentd, Elasticsearch, Kibana) by one command.
43+
44+
``` {.CodeRay}
45+
version: '3'
46+
services:
47+
web:
48+
image: httpd
49+
ports:
50+
- "80:80"
51+
links:
52+
- fluentd
53+
logging:
54+
driver: "fluentd"
55+
options:
56+
fluentd-address: localhost:24224
57+
tag: httpd.access
58+
59+
fluentd:
60+
build: ./fluentd
61+
volumes:
62+
- ./fluentd/conf:/fluentd/etc
63+
links:
64+
- "elasticsearch"
65+
ports:
66+
- "24224:24224"
67+
- "24224:24224/udp"
68+
69+
elasticsearch:
70+
image: docker.elastic.co/elasticsearch/elasticsearch:7.2.0
71+
environment:
72+
- "discovery.type=single-node"
73+
expose:
74+
- "9200"
75+
ports:
76+
- "9200:9200"
77+
78+
kibana:
79+
image: kibana:7.2.0
80+
links:
81+
- "elasticsearch"
82+
ports:
83+
- "5601:5601"
84+
```
85+
86+
`logging` section (check [Docker Compose documentation](https://docs.docker.com/compose/compose-file/#/logging))
87+
of `web` container specifies [Docker Fluentd Logging Driver](https://docs.docker.com/engine/admin/logging/fluentd/) as a
88+
default container logging driver. All of the logs from `web` container
89+
will be automatically forwarded to host:port specified by
90+
`fluentd-address`.
91+
92+
## Step 1: Prepare Fluentd image with your Config + Plugin
93+
94+
Then, please prepare `fluentd/Dockerfile` with the following content, to
95+
use Fluentd's [official Docker image](https://hub.docker.com/r/fluent/fluentd/) and additionally
96+
install Elasticsearch plugin.
97+
98+
``` {.CodeRay}
99+
# fluentd/Dockerfile
100+
FROM fluent/fluentd:v1.6-debian-1
101+
USER root
102+
RUN ["gem", "install", "fluent-plugin-elasticsearch", "--no-document", "--version", "3.5.2"]
103+
USER fluent
104+
```
105+
106+
Then, please prepare Fluentd's configuration file
107+
`fluentd/conf/fluent.conf`. [in\_forward](/plugins/input/forward.md) plugin is used for
108+
receive logs from Docker logging driver, and out\_elasticsearch is for
109+
forwarding logs to Elasticsearch.
110+
111+
``` {.CodeRay}
112+
# fluentd/conf/fluent.conf
113+
<source>
114+
@type forward
115+
port 24224
116+
bind 0.0.0.0
117+
</source>
118+
<match *.**>
119+
@type copy
120+
<store>
121+
@type elasticsearch
122+
host elasticsearch
123+
port 9200
124+
logstash_format true
125+
logstash_prefix fluentd
126+
logstash_dateformat %Y%m%d
127+
include_tag_key true
128+
type_name access_log
129+
tag_key @log_name
130+
flush_interval 1s
131+
</store>
132+
<store>
133+
@type stdout
134+
</store>
135+
</match>
136+
```
137+
138+
## Step 2: Start Containers
139+
140+
Let's start all of the containers, with just one command.
141+
142+
``` {.CodeRay}
143+
$ docker-compose up
144+
```
145+
146+
You can check to see if 4 containers are running by `docker ps` command.
147+
148+
``` {.CodeRay}
149+
$ docker ps
150+
CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES
151+
558fd18fa2d4 httpd "httpd-foreground" 17 seconds ago Up 16 seconds 0.0.0.0:80->80/tcp docker_web_1
152+
bc5bcaedb282 kibana:7.2.0 "/usr/local/bin/kiba…" 18 seconds ago Up 17 seconds 0.0.0.0:5601->5601/tcp docker_kibana_1
153+
9fe2d02cff41 docker.elastic.co/elasticsearch/elasticsearch:7.2.0 "/usr/local/bin/dock…" 20 seconds ago Up 18 seconds 0.0.0.0:9200->9200/tcp, 9300/tcp docker_elasticsearch_1
154+
```
155+
156+
## Step 3: Generate httpd Access Logs
157+
158+
Let's access to `httpd` to generate some access logs. `curl` command is
159+
always your friend.
160+
161+
``` {.CodeRay}
162+
$ repeat 10 curl http://localhost:80/
163+
<html><body><h1>It works!</h1></body></html>
164+
<html><body><h1>It works!</h1></body></html>
165+
<html><body><h1>It works!</h1></body></html>
166+
<html><body><h1>It works!</h1></body></html>
167+
<html><body><h1>It works!</h1></body></html>
168+
<html><body><h1>It works!</h1></body></html>
169+
<html><body><h1>It works!</h1></body></html>
170+
<html><body><h1>It works!</h1></body></html>
171+
<html><body><h1>It works!</h1></body></html>
172+
<html><body><h1>It works!</h1></body></html>
173+
```
174+
175+
## Step 4: Confirm Logs from Kibana
176+
177+
Please go to `http://localhost:5601/` with your browser. Then, you need
178+
to set up the index name pattern for Kibana. Please specify `fluentd-*`
179+
to `Index name or pattern` and press `Create` button.
180+
181+
![](/images/7.2_efk-kibana-index.png)
182+
![](/images/7.2_efk-kibana-timestamp.png)
183+
184+
Then, go to `Discover` tab to seek for the logs. As you can see, logs
185+
are properly collected into Elasticsearch + Kibana, via Fluentd.
186+
187+
![](/images/7.2_efk-kibana-discover.png)
188+
189+
## Conclusion
190+
191+
This article explains how to collect logs from Apache to EFK
192+
(Elasticsearch + Fluentd + Kibana). The example code is available in
193+
this repository.
194+
195+
- <https://github.com/kzk/docker-compose-efk>
196+
197+
## Learn More
198+
199+
- [Fluentd Architecture](https://www.fluentd.org/architecture)
200+
- [Fluentd Get Started](/articles/quickstart.md)
201+
- [Downloading Fluentd](http://www.fluentd.org/download)
202+
203+
204+
------------------------------------------------------------------------
205+
206+
If this article is incorrect or outdated, or omits critical information, please [let us know](https://github.com/fluent/fluentd-docs-gitbook/issues?state=open).
207+
[Fluentd](http://www.fluentd.org/) is a open source project under [Cloud Native Computing Foundation (CNCF)](https://cncf.io/). All components are available under the Apache 2 License.

0 commit comments

Comments
 (0)