Skip to content

Commit e569cf4

Browse files
GuarisGuaris
authored
[NEW] Deploying microservices with docker (#1364)
 * Tech Edit and rewrite done * Tech check and copy edit * backup copy edit to microservices guide, Bob (#1369)
1 parent 0ad3e1e commit e569cf4

File tree

1 file changed

+395
-0
lines changed

1 file changed

+395
-0
lines changed

‎docs/applications/containers/deploying-microservices-with-docker.md‎

Lines changed: 395 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,395 @@
1+
---
2+
author:
3+
name: Bob Strecansky
4+
description: 'This guide describes how to effectively use Docker in production using a sample NGINX/Flask/Gunicorn/Redis/Postgresql Application Stack.'
5+
og_description: 'This guide shows how to deploy a simple microservice using Docker. Best practices for using Docker in production are also demonstrated and explained.'
6+
keywords: ["docker", "nginx", "flask", "gunicorn", "redis", "postgresql", "microservice"]
7+
license: '[CC BY-ND 4.0](https://creativecommons.org/licenses/by-nd/4.0)'
8+
modified: 2018-01-04
9+
modified_by:
10+
name: Bob Strecansky
11+
published: 2018-01-04
12+
title: 'How to Deploy Microservices with Docker'
13+
contributor:
14+
name: Bob Strecansky
15+
link: https://twitter.com/bobstrecansky
16+
external_resources:
17+
- '[Github Repository for Example Microservice](https://github.com/bobstrecansky/flask-microservice)'
18+
- '[Using Containers to Build a Microservices Architecture](https://medium.com/aws-activate-startup-blog/using-containers-to-build-a-microservices-architecture-6e1b8bacb7d1)'
19+
---
20+
21+
## What is a Microservice?
22+
23+
Microservices are an increasingly popular architecture for building large-scale applications. Rather than using a single, monolithic codebase, applications are broken down into a collection of smaller components called microservices. This approach offers several benefits, including the ability to scale individual microservices, keep the codebase easier to understand and test, and enable the use of different programming languages, databases, and other tools for each microservice.
24+
25+
[Docker](https://www.docker.com) is an excellent tool for managing and deploying microservices. Each microservice can be further broken down into processes running in separate Docker containers, which can be specified with Dockerfiles and Docker Compose configuration files. Combined with a provisioning tool such as Kubernetes, each microservice can then be easily deployed, scaled, and collaborated on by a developer team. Specifying an environment in this way also makes it easy to link microservices together to form a larger application.
26+
27+
This guide shows how to build and deploy an example microservice using Docker and Docker Compose.
28+
29+
## Before You Begin
30+
31+
You will need a Linode with Docker and Docker Compose installed to complete this guide.
32+
33+
### Install Docker
34+
35+
{{< section file="/shortguides/docker/install_docker_ce.md" >}}
36+
37+
### Install Docker Compose
38+
39+
{{< section file="/shortguides/docker/install_docker_compose.md" >}}
40+
41+
## Prepare the Environment
42+
43+
This section uses Dockerfiles to configure Docker images. For more information about Dockerfile syntax and best practices, see our [How To Use Dockerfiles guide](/docs/applications/containers/how-to-use-dockerfiles) and Docker's [Dockerfile Best Practices guide](https://docs.docker.com/engine/userguide/eng-image/dockerfile_best-practices/#sort-multi-line-arguments).
44+
45+
1. Create a directory for the microservice:
46+
47+
mkdir flask-microservice
48+
49+
2. Create the directory structure for the microservice components within the new directory:
50+
51+
cd flask-microservice
52+
mkdir nginx postgres web
53+
54+
### NGINX
55+
56+
1. Within the new `nginx` subdirectory, create a Dockerfile for the NGINX image:
57+
58+
{{< file "nginx/Dockerfile" yaml >}}
59+
from nginx:alpine
60+
COPY nginx.conf /etc/nginx/nginx.conf
61+
{{</ file >}}
62+
63+
2. Create the `nginx.conf` referenced in the Dockerfile:
64+
65+
{{< file "/nginx/nginx.conf" conf >}}
66+
user nginx;
67+
worker_processes 1;
68+
error_log /dev/stdout info;
69+
error_log off;
70+
pid /var/run/nginx.pid;
71+
72+
events {
73+
worker_connections 1024;
74+
use epoll;
75+
multi_accept on;
76+
}
77+
78+
http {
79+
include /etc/nginx/mime.types;
80+
default_type application/octet-stream;
81+
82+
log_format main '$remote_addr - $remote_user [$time_local] "$request" '
83+
'$status $body_bytes_sent "$http_referer" '
84+
'"$http_user_agent" "$http_x_forwarded_for"';
85+
86+
access_log /dev/stdout main;
87+
access_log off;
88+
keepalive_timeout 65;
89+
keepalive_requests 100000;
90+
tcp_nopush on;
91+
tcp_nodelay on;
92+
93+
server {
94+
listen 80;
95+
proxy_pass_header Server;
96+
97+
location / {
98+
proxy_set_header Host $host;
99+
proxy_set_header X-Real-IP $remote_addr;
100+
101+
# app comes from /etc/hosts, Docker added it for us!
102+
proxy_pass http://flaskapp:8000/;
103+
}
104+
}
105+
}
106+
{{</ file >}}
107+
108+
### PostgreSQL
109+
110+
The PostgreSQL image for this microservice will use the official `postgresql` image on Docker Hub, so no Dockerfile is necessary.
111+
112+
In the `postgres` subdirectory, create an `init.sql` file:
113+
114+
{{< file "postgres/init.sql">}}
115+
SET statement_timeout = 0;
116+
SET lock_timeout = 0;
117+
SET idle_in_transaction_session_timeout = 0;
118+
SET client_encoding = 'UTF8';
119+
SET standard_conforming_strings = on;
120+
SET check_function_bodies = false;
121+
SET client_min_messages = warning;
122+
SET row_security = off;
123+
CREATE EXTENSION IF NOT EXISTS plpgsql WITH SCHEMA pg_catalog;
124+
COMMENT ON EXTENSION plpgsql IS 'PL/pgSQL procedural language';
125+
SET search_path = public, pg_catalog;
126+
SET default_tablespace = '';
127+
SET default_with_oids = false;
128+
CREATE TABLE visitors (
129+
site_id integer,
130+
site_name text,
131+
visitor_count integer
132+
);
133+
134+
ALTER TABLE visitors OWNER TO postgres;
135+
COPY visitors (site_id, site_name, visitor_count) FROM stdin;
136+
1 linodeexample.com 0
137+
\.
138+
{{</ file >}}
139+
140+
{{< caution >}}
141+
In Line 22 of `init.sql`, make sure your text editor does not convert tabs to spaces. The app will not work without tabs between the entries in this line.
142+
{{< /caution >}}
143+
144+
### Web
145+
146+
The `web` image will hold an example Flask app. Add the following files to the `web` directory to prepare the app:
147+
148+
1. Create a `.python-version` file to specify the use of Python 3.6:
149+
150+
echo "3.6.0" >> web/.python-version
151+
152+
2. Create a Dockerfile for the `web` image:
153+
154+
{{< file "web/Dockerfile" yaml >}}
155+
from python:3.6.2-slim
156+
RUN groupadd flaskgroup && useradd -m -g flaskgroup -s /bin/bash flask
157+
RUN echo "flask ALL=(ALL) NOPASSWD:ALL" >> /etc/sudoers
158+
RUN mkdir -p /home/flask/app/web
159+
WORKDIR /home/flask/app/web
160+
COPY requirements.txt /home/flask/app/web
161+
RUN pip install --no-cache-dir -r requirements.txt
162+
RUN chown -R flask:flaskgroup /home/flask
163+
USER flask
164+
ENTRYPOINT ["/usr/local/bin/gunicorn", "--bind", ":8000", "linode:app", "--reload", "--workers", "16"]
165+
{{</ file >}}
166+
167+
3. Create `web/linode.py` and add the example app script:
168+
169+
{{< file "web/linode.py" >}}
170+
from flask import Flask
171+
import logging
172+
import psycopg2
173+
import redis
174+
import sys
175+
176+
app = Flask(__name__)
177+
cache = redis.StrictRedis(host='redis', port=6379)
178+
179+
# Configure Logging
180+
app.logger.addHandler(logging.StreamHandler(sys.stdout))
181+
app.logger.setLevel(logging.DEBUG)
182+
183+
def PgFetch(query, method):
184+
185+
# Connect to an existing database
186+
conn = psycopg2.connect("host='postgres' dbname='linode' user='postgres' password='linode123'")
187+
188+
# Open a cursor to perform database operations
189+
cur = conn.cursor()
190+
191+
# Query the database and obtain data as Python objects
192+
dbquery = cur.execute(query)
193+
194+
if method == 'GET':
195+
result = cur.fetchone()
196+
else:
197+
result = ""
198+
199+
# Make the changes to the database persistent
200+
conn.commit()
201+
202+
# Close communication with the database
203+
cur.close()
204+
conn.close()
205+
return result
206+
207+
@app.route('/')
208+
def hello_world():
209+
if cache.exists('visitor_count'):
210+
cache.incr('visitor_count')
211+
count = (cache.get('visitor_count')).decode('utf-8')
212+
update = PgFetch("UPDATE visitors set visitor_count = " + count + " where site_id = 1;", "POST")
213+
else:
214+
cache_refresh = PgFetch("SELECT visitor_count FROM visitors where site_id = 1;", "GET")
215+
count = int(cache_refresh[0])
216+
cache.set('visitor_count', count)
217+
cache.incr('visitor_count')
218+
count = (cache.get('visitor_count')).decode('utf-8')
219+
return 'Hello Linode! This page has been viewed %s time(s).' % count
220+
221+
@app.route('/resetcounter')
222+
def resetcounter():
223+
cache.delete('visitor_count')
224+
PgFetch("UPDATE visitors set visitor_count = 0 where site_id = 1;", "POST")
225+
app.logger.debug("reset visitor count")
226+
return "Successfully deleted redis and postgres counters"
227+
228+
{{</ file >}}
229+
230+
4. Add a `requirements.txt` file with the required Python dependencies:
231+
232+
{{< file "web/requirements.txt" text >}}
233+
flask
234+
gunicorn
235+
psycopg2
236+
redis
237+
{{</ file >}}
238+
239+
## Docker Compose
240+
241+
Docker Compose will be used to be define the connections between containers and their configuration settings.
242+
243+
Create a `docker-compose.yml` file in the `flask-microservice` directory and add the following:
244+
245+
{{< file "docker-compose.yml" yaml >}}
246+
version: '3'
247+
services:
248+
# Define the Flask web application
249+
flaskapp:
250+
251+
# Build the Dockerfile that is in the web directory
252+
build: ./web
253+
254+
# Always restart the container regardless of the exit status; try and restart the container indefinitely
255+
restart: always
256+
257+
# Expose port 8000 to other containers (not to the host of the machine)
258+
expose:
259+
- "8000"
260+
261+
# Mount the web directory within the container at /home/flask/app/web
262+
volumes:
263+
- ./web:/home/flask/app/web
264+
265+
# Don't create this container until the redis and postgres containers (below) have been created
266+
depends_on:
267+
- redis
268+
- postgres
269+
270+
# Link the redis and postgres containers together so that they can talk to one another
271+
links:
272+
- redis
273+
- postgres
274+
275+
# Pass environment variables to the flask container (this debug level lets you see more useful information)
276+
environment:
277+
FLASK_DEBUG: 1
278+
279+
# Deploy with 3 replicas in the case of failure of one of the containers (only in Docker Swarm)
280+
deploy:
281+
mode: replicated
282+
replicas: 3
283+
284+
# Define the redis Docker container
285+
redis:
286+
287+
# use the redis:alpine image: https://hub.docker.com/_/redis/
288+
image: redis:alpine
289+
restart: always
290+
deploy:
291+
mode: replicated
292+
replicas: 3
293+
294+
# Define the redis NGINX forward proxy container
295+
nginx:
296+
297+
# build the nginx Dockerfile: http://bit.ly/2kuYaIv
298+
build: nginx/
299+
restart: always
300+
301+
# Expose port 80 to the host machine
302+
ports:
303+
- "80:80"
304+
deploy:
305+
mode: replicated
306+
replicas: 3
307+
308+
# The Flask application needs to be available for NGINX to make successful proxy requests
309+
depends_on:
310+
- flaskapp
311+
312+
# Define the postgres database
313+
postgres:
314+
restart: always
315+
# Use the postgres alpine image: https://hub.docker.com/_/postgres/
316+
image: postgres:alpine
317+
318+
# Mount an initialization script and the persistent postgresql data volume
319+
volumes:
320+
- ./postgres/init.sql:/docker-entrypoint-initdb.d/init.sql
321+
- ./postgres/data:/var/lib/postgresql/data
322+
323+
# Pass postgres environment variables
324+
environment:
325+
POSTGRES_PASSWORD: linode123
326+
POSTGRES_DB: linode
327+
328+
# Expose port 5432 to other Docker containers
329+
expose:
330+
- "5432"
331+
{{</ file >}}
332+
333+
334+
## Test the Microservice
335+
336+
1. Use Docker Compose to build all of the images and start the microservice:
337+
338+
cd flask-microservice/ && docker-compose up
339+
340+
You should see all of the services start in your terminal.
341+
342+
2. Open a new terminal window and make a request to the example application:
343+
344+
curl localhost
345+
346+
{{< output >}}
347+
Hello Linode! This page has been viewed 1 time(s).
348+
{{< /output >}}
349+
350+
3. Reset the page hit counter:
351+
352+
curl localhost/resetcounter
353+
354+
{{< output >}}
355+
Successfully deleted redis and postgres counters
356+
{{< /output >}}
357+
358+
4. Return to the terminal window where Docker Compose was started to view the standard out log:
359+
360+
{{< output >}}
361+
flaskapp_1 | DEBUG in linode [/home/flask/app/web/linode.py:56]:
362+
flaskapp_1 | reset visitor count
363+
{{< /output >}}
364+
365+
## Using Containers in Production: Best Practices
366+
367+
The containers used in the example microservice are intended to demonstrate the following best practices for using containers in production:
368+
369+
Containers should be:
370+
371+
1. **Ephemeral**: It should be easy to stop, destroy, rebuild, and redeploy containers with minimal setup and configuration.
372+
373+
The Flask microservice is an ideal example of this. The entire microservice can be brought up or down using Docker Compose. No additional configuration is necessary after the containers are running, which makes it easy to modify the application.
374+
375+
2. **Disposable**: Ideally, any single container within a larger application should be able to fail without impacting the performance of the application. Using a `restart: on-failure` option in the `docker-compose.yml` file, as well as having a replica count, makes it possible for some containers in the example microservice to fail gracefully while still serving the web application with no degradation to the end user.
376+
377+
{{< note >}}
378+
The replica count directive will only be effective when this configuration is deployed as part of a [Docker Swarm](/docs/applications/containers/how-to-create-a-docker-swarm-manager-and-nodes-on-linode/), which is not covered in this guide.
379+
{{< /note >}}
380+
381+
3. **Quick to start**: Avoiding additional installation steps in the Docker file, removing dependencies that aren't needed, and building a target image that can be reused are three of the most important steps in making a web application that has a quick initialization time within Docker. The example application uses short, concise, prebuilt Dockerfiles in order to minimize initialization time.
382+
383+
4. **Quick to stop**: Validate that a `docker kill --signal=SIGINT {APPNAME}` stops the application gracefully. This, along with a restart condition and a replica condition, will ensure that when containers fail, they will be brought back online efficiently.
384+
385+
5. **Lightweight**: Use the smallest base container that provides all of the utilities that are needed to build and run your application. Many Docker images are based on [Alpine Linux](https://alpinelinux.org/about/), a light and simple Linux distribution that takes up only 5MB in a Docker image. Using a small distro saves network and operational overhead and greatly increases container performance. The example application uses alpine images where applicable (NGINX, Redis, and PostgreSQL), and uses a python-slim base image for the Gunicorn / Flask application.
386+
387+
6. **Stateless**: Since they are ephemeral, containers typically shouldn't maintain state. An application's state should be stored in a separate, persistent data volume, as is the case with the microservice's PostgreSQL data store. The Redis key-value store does maintain data within a container, but this data is not application-critical; the Redis store will fail back gracefully to the database should the container not be able to respond.
388+
389+
7. **Portable**: All of an app's dependencies that are needed for the container runtime should be locally available. All of the example microservice's dependencies and startup scripts are stored in the directory for each component. These can be checked into version control, making it easy to share and deploy the application.
390+
391+
8. **Modular**: Each container should have one responsibility and one process. In this microservice, each of the major processes (NGINX, Python, Redis, and PostgreSQL) is deployed in a separate container.
392+
393+
9. **Logging**: All containers should log to `STDOUT`. This uniformity makes it easy to view the logs for all of the processes in a single stream.
394+
395+
10. **Resilient**: The example application restarts its containers if they are exited for any reason. This helps give your Dockerized application high availability and performance, even during maintenance periods.

0 commit comments

Comments
 (0)