Skip to content
Browse files

Add support for using Docker for local development (#3002)

* Add Docker support for local dev

* remove those terrible flags from our invoke commands

* Change recursive call in invoke commands and don't expose port for postgres

* Add note about Docker download direct links and how tu run in detached mode

* Add CHOKIDAR_USEPOLLING so that auto rebuild works on windows machines

* Add Docker support for local dev

* remove those terrible flags from our invoke commands

* Change recursive call in invoke commands and don't expose port for postgres

* Add note about Docker download direct links and how tu run in detached mode

* Add CHOKIDAR_USEPOLLING so that auto rebuild works on windows machines

* removing trailing commas

* Move to python 3.7

* add missing lockfile
  • Loading branch information...
patjouk committed Apr 25, 2019
1 parent 6609e25 commit b422098368b9dc452df31b2da5d3cc914ac3508c
@@ -83,6 +83,7 @@ celerybeat-schedule

# dotenv

# virtualenv
@@ -64,6 +64,7 @@ matrix:
- "./travis-scripts/"
- "./travis-scripts/"
- npm run cypress:install
- psql -c 'create database network;' -U postgres
- npm run build
@@ -39,4 +39,4 @@ pyyaml = "==4.2b4"
python-coveralls = "*"

python_version = "3.6"
python_version = "3.7"

Some generated files are not rendered by default. Learn more.

@@ -11,7 +11,11 @@


[Development and tooling](docs/
[Setup with Docker](#how-to-set-up-your-dev-environment-with-docker)

[Local development with invoke and pipenv](docs/

[Local development with Docker](docs/

[Engineer Workflow](docs/

@@ -21,7 +25,7 @@


## Setup
## How to Setup your Dev Environment with Pipenv and Invoke

**Requirements**: [Node](, [npm](, [git](, [python3.6 or later](, [pip](, [pipenv](, [invoke](

@@ -47,6 +51,7 @@ You're done :tada:

To catch up on new dependencies, migrations, etc. after initial setup, you can use the `inv catch-up` command.

For more information on how to run this project, check the [local development with invoke and pipenv](docs/ documentation.

## Testing

@@ -57,7 +62,18 @@ When relevant, we encourage you to write tests. You can run the tests using the
In addition to the code tests there are also visual regression tests, located in the `./cypress/integration` directory. You can run these tests locally by installing [cypress]( using `npm i cypress@3.0.3`, after which the command `npm run cypress` will run these tests locally. However, note that these tests are currently intended for screenshot comparisons across branches, and so will not yield any meaningful results when run for a single branch.

## How to Setup your Dev Environment with Docker

- Install [Docker Desktop]( (macOS and Windows). For Linux users: install [Docker CE]( and [Docker Compose]( If you don't want to create a Docker account, direct links to download can be found [in this issue](,
- [Check your install]( by running `docker run hello-world`,
- If relevant: delete your node_modules directory (`rm -rf node_modules`). It's not necessary, but it speeds up the install.
- Run `invoke docker-setup` ([install invoke]( if you don't have it yet). If you're running on Windows, you need to run `docker-compose --rm pipenv run python network-api/ createsuperuser` when the setup is finished.

This task is copying your `.env` to the new `.docker.env` that is in charge of managing your environment variables while running Docker. The installation will take a few minutes: you need to download images from the Docker Hub, install JS and Python dependencies, create fake data, migrate your database, etc.

When it's done, run `docker-compose up`, wait until the static files to be built, and go to ``. You should have a local working version of the foundation site with fake data. When you want to stop, do `^C` to shut down your containers.

For more information on how to run the project with Docker, check the [local development with Docker](docs/ documentation.

## Security
@@ -0,0 +1,46 @@
version: '3'


context: .
dockerfile: ./dockerfiles/Dockerfile.node
# Need to specify the SHELL env var for chokidar
- SHELL=/bin/sh
# Force polling because inotify doesn't work on Docker Windows
command: npm run watch
- .:/app
- node_dependencies:/app/node_modules/

image: postgres:9.6
- "5432"
- postgres_data:/var/lib/postgresql/data/

context: .
dockerfile: ./dockerfiles/Dockerfile.python
- ".docker.env"
# Prevents pipenv from loading env files (python dotenv and docker-compose compatibility issue).
command: pipenv run python network-api/ runserver
- "8000:8000"
- .:/app
- postgres
- watch-static-files

@@ -0,0 +1,16 @@
FROM node:8-alpine


# Copy package files in the container
COPY package.json package-lock.json ./

RUN apk add --no-cache \
zlib-dev \

RUN npm install

# Install visual testing tools separately with CI true to suppress the hundreds lines of "unziping".
# Might replace it by a silent flag if this PR gets merged:
RUN CI=true npm run cypress:install
@@ -0,0 +1,24 @@
FROM python:3.7-alpine

# We want output
# We don't want *.pyc


# Install dependencies for pillow and psycopg
RUN apk add --no-cache \
build-base \
jpeg-dev \
zlib-dev \

# Install pipenv
RUN pip install pipenv

# Copy Pipfiles in the container
COPY Pipfile Pipfile.lock ./

# Install app deps
RUN pipenv install -d
@@ -0,0 +1,152 @@
# Docker for Local Dev Documentation

This documentation is composed of three main sections:
- [How to install and use Docker for local development](./
- [Docker 101 and how we use it with the foundation site](./ Start here if you're new to Docker
- [FAQ](./

## How to use

To interact with the project, you can use [docker]( and [docker-compose]( CLIs or use shortcuts with invoke.

The general workflow is:
- Install the project with `invoke docker-setup`,
- Run the project with `docker-compose up`,
- Use invoke commands for frequent development tasks (database migrations, dependencies install, run tests, etc),
- After doing a `git pull`, keep your clone up to date by running `invoke docker-catchup`.

### Invoke commands

To get a list of invoke commands available, run `invoke -l`:

docker-catch-up (docker-catchup) Rebuild images and apply migrations
docker-l10n-sync Sync localizable fields in the database
docker-l10n-update Update localizable field data (copies from original unlocalized to default localized field)
docker-makemigrations Creates new migration(s) for apps
docker-manage Shorthand to inv docker.manage "[COMMAND] [ARG]"
docker-migrate Updates database schema
docker-npm Shorthand to npm. inv docker.npm "[COMMAND] [ARG]"
docker-nuke-db Delete your database and create a new one with fake data
docker-pipenv Shorthand to pipenv. inv docker.pipenv "[COMMAND] [ARG]"
docker-setup Prepare your dev environment after a fresh git clone
docker-test-node Run node tests
docker-test-python Run python tests

`docker-` prefixes all Docker commands. Note the double quotes to pass multiples arguments to `docker-manage`, `docker-pipenv` and `docker-npm` commands. There's no `invoke docker-runserver` command: use `docker-compose up` instead.

**A few examples:** `invoke docker-pipenv "install requests""`: add requests to your `Pipfile` and lock it. :rotating_light: The package won't be installed: you need to [rebuild your image](./
- `invoke docker-manage load_fake_data`: add more fake data to your project,
- `invoke docker-npm "install moment"`: install moment, add it to your `package.json` and lock it.

### Docker and docker-compose CLIs

We strongly recommend you to check at least the [docker-compose CLI]( documentation since we're using it a lot. Meanwhile, here are the commands you will use the most:

- [docker-compose up]( start the services and the project. Stop them with `^C`. If you want to rebuild your images, for example after a python dependencies update, add the `--build` flag. If you want to run the services in detached mode, use `--detached`. To get logs, use `docker-compose logs --follow [SERVICE]`,
- [docker-compose down](): stop and remove the services,
- [docker-compose run (--rm) [SERVICE NAME] [COMMAND]]( run a command against a service. `--rm` removes your container when you're done,
- [docker-compose build [SERVICE NAME]]( build a new image for the service. Use `--no-cache` to build the image from scratch again,
- [docker-compose ps]( list the services running.

- [docker image]( interact with images,
- [docker container]( interact with containers,
- [docker volume]( interact with volumes.
- [docker system prune]( delete all unused container, image and network. Add `--volumes` to also remove volume. :rotating_light: It will impact other docker project running on your system! For a more subtle approach, [check this blog post]( on to remove elements selectively.

### How to install or update dependencies?

The significant difference between the two is that python dependencies are "baked" into the image, while JS dependencies are stored in a volume.

#### Python

**Install packages:**

Use `invoke docker-pipenv "install [PACKAGE]"`.

:rotating_light: Important note! :rotating_light: This **only** add the dependency to your `Pipfile` and lock-it: it doesn't install your dependency! After running this command, run `docker-compose build backend` to create a new and updated backend image to use.

**Update packages:**

To update your dependencies, do `invoke docker-pipenv update`, then build the new image with `docker-compose build backend`.

#### JS

**Install packages:**

Use `invoke docker-npm "install [PACKAGE]"`.

**Update packages:**

Use `invoke docker-npm update`.

You don't need to rebuild the `watch-static-files` image.


## Docker vocabulary and overview

Welcome to Docker! Before jumping into Docker installation, take a moment to get familiar with Docker vocabulary:

- Docker: Docker is a platform to develop, deploy and run applications with containers.
- Docker engine: The Docker engine is a service running in the background (daemon). It's managing containers.
- Docker CLI: Command Line Interface to interact with Docker. For example, `Docker image ls` lists the images available on your system.
- Docker hub: Registry containing Docker images.
- Image: An image is a file used to build containers: In our case, it's mostly instructions to install dependencies.
- Container: Containers run an image. In our case, we have a container for the database, another one for building static files and the last one for running Django. A container life is ephemeral: data written there don't persist when you shut down a container.
- Volume: A volume is a special directory on your machine that is used to make data persistent. For example, we use it to store the database: that way, you don't lose your data when you turn down your containers.
- Host: host is used in Docker docs to mean the system on top of which containers run.
- Docker-compose: It's a tool to run multi-container applications: we use it to run our three containers together.
- Docker-compose CLI: Command line interface to interact with docker-compose. It's used to launch your dev environment.
- Docker-compose service: a service is a container and the configuration associated to it.

I would recommend watching [An Intro to Docker for Djangonauts]( by Lacey Williams Henschel (25 min, [repo mentioned in the talk]( it's a great beginner talk to learn Docker and how to use it with Django.

### Project Structure

All our containers run on Linux.

For local development, we have two Dockerfiles that define our images:
- `Dockerfile.node`: use a node8-alpine base image from the Docker Hub and install node dependencies,
- `Dockerfile.python`: use a python3.6-alpine base image, install required build dependencies before installing pipenv and the project dependencies.
We don't have a custom image for running postgres and use one from the Docker Hub.

The `docker-compose.yml` file describes the 3 services that the project needs to run:
- `watch-static-files`: rebuilds static files when they're modified,
- `postgres`: contains a postgres database,
- `backend`: runs Django. Starting this one automatically starts the two other ones.

### Resources about Docker

- [Docker]( and [Docker-compose]( documentations,
- [Intro to Docker]( Lacey wrote a good intro tutorial to Docker and Django, without Harry Potter metaphors this time :),
- [Jérôme Petazzoni's training slides and talks]( presentations and slides if you want to dive into Docker.


## FAQ

### Do I need to build the static files before doing a `docker-compose up`?

Static files are automatically built when starting the `watch-static-files` container.

### Where is Docker fitting in all the tools we're already using?

Let's do a quick overview of all the tools you're currently using to run the foundation site on your computer:

- `npm`: use to manage javascript dependencies (`packages.json`, `packages-lock.json`). Also used to launch commands like `npm run start`.
- `pipenv`: use to manage python dependencies (`pipfile`, `pipfile.lock`). Also manage a python virtual environment, which isolates the foundation site's python packages from the rest of your system.
- `invoke`/`inv`: use as a cli tool to provide shortcuts for most used commands. ex: `inv runserver` is a shortcut for `pipenv run python network-api/ runserver`.

We still use all those tools with Docker. The major difference is that `npm` and `pipenv` is now running inside a container, while invoke continues to run as before.

### Can I use Docker in parallel with the old way of running the foundation site?

Short answer is yes but:
- you will have two different databases
- you will have two files to manage your environments variables (`.env` and `.env.docker`),
- those two environment won't share their dependencies: you will have to maintain and update both of them.
@@ -32,10 +32,7 @@ But it's a bit long. So instead, you can use invoke:
- `inv test`: Run tests
- `inv catch-up`: Install dependencies and apply migrations

For management commands not covered by an invoke tasks, use `inv manage [command]` (example: `inv manage load_fake_data`). You can pass flag and options to management commands using `inv manage [command] -o [positional argument] -f [optional argument]`. For example:
- `inv manage runserver -o 3000`
- `inv manage load_fake_data -f seed=VALUE`
- `inv manage migrate -o news`
For management commands not covered by an invoke tasks, use `inv manage [command]` (example: `inv manage load_fake_data`). You can pass multiple arguments to `inv manage` by using double quotes. Ex: `inv manage "load_fake_data --delete"`.

### Generating a new set of fake model data

@@ -46,15 +43,15 @@ By default, your dev site will use production data (read only!). To load fake mo

You can empty your database and create a full new set of fake model data using the following command

- `inv manage load_fake_data -o --delete`
- `inv manage "load_fake_data --delete"`


- `pipenv run python network-api/ load_fake_data --delete`

You can generate a specific set of fake model data by entering a seed value

- `inv manage load_fake_data -o --delete --seed VALUE`
- `inv manage "load_fake_data --delete --seed VALUE"`


@@ -10,7 +10,7 @@

# network-api environment:


0 comments on commit b422098

Please sign in to comment.
You can’t perform that action at this time.