We've used Postgres as our database. We used an sql
script to perform the first-time database related initializations such as creating database and user. Take a look at the postgres_init.sql
to customize per your need.
It's very important to be able to manage database tables migrations and schema updates as our service grows or tables schema changes over time.
We use SQLAlchemy alembic
to manage database migrations. To read more https://alembic.sqlalchemy.org/.
Follow the below steps to enable alembic
migration for your service.
There are different ways to use alembic
library in your service. I've described two different methods.
- Go into a directory and create a python virtual environment, and activate it.
cd /path/to/a/dir
python3 -m venv venv
source venv/bin/activate
- Add
alembic
torequirements.txt
file and install packages:
pip install -r config/requirements.txt
Or install it directly in your environment:
pip install alembic
- Go into the
tutorial04
dir, and update thePYTHONPATH
environment variable:
export PYTHONPATH=$PYTHONPATH:$PWD:$PWD/apps
- Go into the
tutorial04/apps
directory, and initialize alembic, by whichalembic.ini
file andmigrations
directory are created.
alembic init migrations
Important: Once you did the above 4 steps, there are two options for you to enable alembic.
Please make sure that your database is up and accessible from localhost.
- From the
tutorial04/apps
directory, run the following command to generate the first migration version. It will automatically detect changes and create a revision file.
alembic revision --autogenerate -m "create initial tables"
You need to run the above command every time your tables schema changes.
- Finnaly, apply the changes:
alembic upgrade head
This command will reflect the new changes into database table. You need to run this command every time you have new revision files.
Please make sure that your service is up and running
- First build the image and make sure
auth
service is running, then try attaching to the container.
cd tutorial04
docker-compose build auth
docker-compose exec auth bash
Once iside the container, go into the apps
directory and run the below command:
cd apps
alembic revision --autogenerate -m "create initial tables"
You need to run the above command every time your tables schema change.
- Finnaly, apply the changes:
alembic upgrade head
This command will reflect the new changes into database table. You need to run this command every time you have new revision files.
In order to run a set of commands for your service to start up we can use a docker-entrypoint.sh
file. For example, we want to do database migrations before running the service to make sure database schema are up to date.
...
alembic upgrade head
...
We used Redis
in-memory caching app to cache objects from our services.
You often need to have different images for different test, production, or stage environments. Update the TAG
value in the .env
file for different situations.
You can also use shell terminal ENV variables to overwrite the variables inside that file. For example export TAG=local
will overwrite the TAG in the .env file when used by docker-compose.
export TAG=local
docker-compose logs -f <service_name>
e.g,
docker-compose logs -f auth
docker-compose logs -f webserver
You can have as many Dockerfiles as you want, with just making files with .Dockerfile
postfix. For example:
auth.Dockerfile
forauth
servicenginx.Dockerfile
fornginx
service
Similar to .gitignore
file which tells git to ignore files, we can use .dockerignore
file to instruct Docker to ignore files.
Rate limiting can be used to prevent DDoS attacks, or prevent upstream servers from being overwhelmed by too many requests at the same time.
For example, the below config in nginx informs the web server to allow only one request per second from a given ip.
limit_req_zone $binary_remote_addr zone=one:10m rate=1r/s;
- NGINX actually tracks requests at millisecond granularity
- What if we get 2 requests within 100ms of each other?
- For the second request NGINX returns status code 503 to the client. For requests that arrive at the full bucket, NGINX will respond with the 503 Service Unavailable error.
We can buffer any excess requests and service them in a timely manner. This is where we use the burst parameter to limit_req, as in this updated configuration:
limit_req zone=mylimit burst=20;
The burst parameter defines how many requests a client can make in excess of the rate specified by the zone (with our sample mylimit zone, the rate limit is 10 requests per second, or 1 every 100ms). A request that arrives sooner than 100ms after the previous one is put in a queue, and here we are setting the queue size to 20.
You can read more about nginx rate limit.