Skip to content
Permalink
Browse files

Add support for using Docker for local development (#3002)

* Add Docker support for local dev

* remove those terrible flags from our invoke commands

* Change recursive call in invoke commands and don't expose port for postgres

* Add note about Docker download direct links and how tu run in detached mode

* Add CHOKIDAR_USEPOLLING so that auto rebuild works on windows machines

* Add Docker support for local dev

* remove those terrible flags from our invoke commands

* Change recursive call in invoke commands and don't expose port for postgres

* Add note about Docker download direct links and how tu run in detached mode

* Add CHOKIDAR_USEPOLLING so that auto rebuild works on windows machines

* removing trailing commas

* Move to python 3.7

* add missing lockfile
  • Loading branch information...
patjouk committed Apr 25, 2019
1 parent 6609e25 commit b422098368b9dc452df31b2da5d3cc914ac3508c
@@ -83,6 +83,7 @@ celerybeat-schedule

# dotenv
.env
.docker.env

# virtualenv
venv/
@@ -64,6 +64,7 @@ matrix:
install:
- "./travis-scripts/python-install.sh"
- "./travis-scripts/npm-install.sh"
- npm run cypress:install
before_script:
- psql -c 'create database network;' -U postgres
- npm run build
@@ -39,4 +39,4 @@ pyyaml = "==4.2b4"
python-coveralls = "*"

[requires]
python_version = "3.6"
python_version = "3.7"

Some generated files are not rendered by default. Learn more.

@@ -11,7 +11,11 @@

[Setup](#setup)

[Development and tooling](docs/development.md)
[Setup with Docker](#how-to-set-up-your-dev-environment-with-docker)

[Local development with invoke and pipenv](docs/local_development_with_invoke_pipenv.md)

[Local development with Docker](docs/local_development_with_docker.md)

[Engineer Workflow](docs/workflow.md)

@@ -21,7 +25,7 @@

[Stack](docs/stack.md)

## Setup
## How to Setup your Dev Environment with Pipenv and Invoke

**Requirements**: [Node](https://nodejs.org), [npm](https://www.npmjs.com/), [git](https://git-scm.com/), [python3.6 or later](https://www.python.org/), [pip](https://pypi.python.org/pypi), [pipenv](https://docs.pipenv.org/), [invoke](https://www.pyinvoke.org/installing.html).

@@ -47,6 +51,7 @@ You're done :tada:

To catch up on new dependencies, migrations, etc. after initial setup, you can use the `inv catch-up` command.

For more information on how to run this project, check the [local development with invoke and pipenv](docs/local_development_with_invoke_pipenv.md) documentation.

## Testing

@@ -57,7 +62,18 @@ When relevant, we encourage you to write tests. You can run the tests using the
In addition to the code tests there are also visual regression tests, located in the `./cypress/integration` directory. You can run these tests locally by installing [cypress](https://www.cypress.io/) using `npm i cypress@3.0.3`, after which the command `npm run cypress` will run these tests locally. However, note that these tests are currently intended for screenshot comparisons across branches, and so will not yield any meaningful results when run for a single branch.


## How to Setup your Dev Environment with Docker

- Install [Docker Desktop](https://www.docker.com/products/docker-desktop) (macOS and Windows). For Linux users: install [Docker CE](https://docs.docker.com/install/#supported-platforms) and [Docker Compose](https://docs.docker.com/compose/install/). If you don't want to create a Docker account, direct links to download can be found [in this issue](https://github.com/docker/docker.github.io/issues/6910),
- [Check your install](https://docs.docker.com/get-started/#test-docker-version) by running `docker run hello-world`,
- If relevant: delete your node_modules directory (`rm -rf node_modules`). It's not necessary, but it speeds up the install.
- Run `invoke docker-setup` ([install invoke](http://www.pyinvoke.org/installing.html) if you don't have it yet). If you're running on Windows, you need to run `docker-compose --rm pipenv run python network-api/manage.py createsuperuser` when the setup is finished.

This task is copying your `.env` to the new `.docker.env` that is in charge of managing your environment variables while running Docker. The installation will take a few minutes: you need to download images from the Docker Hub, install JS and Python dependencies, create fake data, migrate your database, etc.

When it's done, run `docker-compose up`, wait until the static files to be built, and go to `0.0.0.0:8000`. You should have a local working version of the foundation site with fake data. When you want to stop, do `^C` to shut down your containers.

For more information on how to run the project with Docker, check the [local development with Docker](docs/local_development_with_docker.md) documentation.


## Security
@@ -0,0 +1,46 @@
version: '3'

services:

watch-static-files:
build:
context: .
dockerfile: ./dockerfiles/Dockerfile.node
environment:
# Need to specify the SHELL env var for chokidar
- SHELL=/bin/sh
# Force polling because inotify doesn't work on Docker Windows
- CHOKIDAR_USEPOLLING=1
command: npm run watch
volumes:
- .:/app
- node_dependencies:/app/node_modules/

postgres:
image: postgres:9.6
ports:
- "5432"
volumes:
- postgres_data:/var/lib/postgresql/data/

backend:
build:
context: .
dockerfile: ./dockerfiles/Dockerfile.python
env_file:
- ".docker.env"
environment:
# Prevents pipenv from loading env files (python dotenv and docker-compose compatibility issue).
- PIPENV_DONT_LOAD_ENV=1
command: pipenv run python network-api/manage.py runserver 0.0.0.0:8000
ports:
- "8000:8000"
volumes:
- .:/app
depends_on:
- postgres
- watch-static-files

volumes:
postgres_data:
node_dependencies:
@@ -0,0 +1,16 @@
FROM node:8-alpine

WORKDIR /app/

# Copy package files in the container
COPY package.json package-lock.json ./

RUN apk add --no-cache \
zlib-dev \
build-base

RUN npm install

# Install visual testing tools separately with CI true to suppress the hundreds lines of "unziping".
# Might replace it by a silent flag if this PR gets merged: https://github.com/cypress-io/cypress/pull/2706
RUN CI=true npm run cypress:install
@@ -0,0 +1,24 @@
FROM python:3.7-alpine

# We want output
ENV PYTHONUNBUFFERED 1
# We don't want *.pyc
ENV PYTHONDONTWRITEBYTECODE 1

WORKDIR /app/

# Install dependencies for pillow and psycopg
RUN apk add --no-cache \
build-base \
jpeg-dev \
zlib-dev \
postgresql-dev

# Install pipenv
RUN pip install pipenv

# Copy Pipfiles in the container
COPY Pipfile Pipfile.lock ./

# Install app deps
RUN pipenv install -d
@@ -0,0 +1,152 @@
# Docker for Local Dev Documentation

This documentation is composed of three main sections:
- [How to install and use Docker for local development](./local_development_with_docker.md#how-to-use)
- [Docker 101 and how we use it with the foundation site](./local_development_with_docker.md#docker-vocabulary-and-overview). Start here if you're new to Docker
- [FAQ](./local_development_with_docker.md#faq)

## How to use

To interact with the project, you can use [docker](https://docs.docker.com/engine/reference/commandline/cli/) and [docker-compose](https://docs.docker.com/compose/reference/overview/) CLIs or use shortcuts with invoke.

The general workflow is:
- Install the project with `invoke docker-setup`,
- Run the project with `docker-compose up`,
- Use invoke commands for frequent development tasks (database migrations, dependencies install, run tests, etc),
- After doing a `git pull`, keep your clone up to date by running `invoke docker-catchup`.

### Invoke commands

To get a list of invoke commands available, run `invoke -l`:

```
docker-catch-up (docker-catchup) Rebuild images and apply migrations
docker-l10n-sync Sync localizable fields in the database
docker-l10n-update Update localizable field data (copies from original unlocalized to default localized field)
docker-makemigrations Creates new migration(s) for apps
docker-manage Shorthand to manage.py. inv docker.manage "[COMMAND] [ARG]"
docker-migrate Updates database schema
docker-npm Shorthand to npm. inv docker.npm "[COMMAND] [ARG]"
docker-nuke-db Delete your database and create a new one with fake data
docker-pipenv Shorthand to pipenv. inv docker.pipenv "[COMMAND] [ARG]"
docker-setup Prepare your dev environment after a fresh git clone
docker-test-node Run node tests
docker-test-python Run python tests
```

`docker-` prefixes all Docker commands. Note the double quotes to pass multiples arguments to `docker-manage`, `docker-pipenv` and `docker-npm` commands. There's no `invoke docker-runserver` command: use `docker-compose up` instead.

**A few examples:** `invoke docker-pipenv "install requests""`: add requests to your `Pipfile` and lock it. :rotating_light: The package won't be installed: you need to [rebuild your image](./local_development_with_docker.md#python).
- `invoke docker-manage load_fake_data`: add more fake data to your project,
- `invoke docker-npm "install moment"`: install moment, add it to your `package.json` and lock it.

### Docker and docker-compose CLIs

We strongly recommend you to check at least the [docker-compose CLI](https://docs.docker.com/compose/reference/overview/) documentation since we're using it a lot. Meanwhile, here are the commands you will use the most:

**docker-compose:**
- [docker-compose up](https://docs.docker.com/compose/reference/up/): start the services and the project. Stop them with `^C`. If you want to rebuild your images, for example after a python dependencies update, add the `--build` flag. If you want to run the services in detached mode, use `--detached`. To get logs, use `docker-compose logs --follow [SERVICE]`,
- [docker-compose down](): stop and remove the services,
- [docker-compose run (--rm) [SERVICE NAME] [COMMAND]](https://docs.docker.com/compose/reference/run/): run a command against a service. `--rm` removes your container when you're done,
- [docker-compose build [SERVICE NAME]](https://docs.docker.com/compose/reference/build/): build a new image for the service. Use `--no-cache` to build the image from scratch again,
- [docker-compose ps](https://docs.docker.com/compose/reference/ps/): list the services running.

**docker:**
- [docker image](https://docs.docker.com/engine/reference/commandline/image/): interact with images,
- [docker container](https://docs.docker.com/engine/reference/commandline/container/): interact with containers,
- [docker volume](https://docs.docker.com/engine/reference/commandline/volume_create/): interact with volumes.
- [docker system prune](https://docs.docker.com/engine/reference/commandline/system_prune/): delete all unused container, image and network. Add `--volumes` to also remove volume. :rotating_light: It will impact other docker project running on your system! For a more subtle approach, [check this blog post](https://www.digitalocean.com/community/tutorials/how-to-remove-docker-images-containers-and-volumes) on to remove elements selectively.

### How to install or update dependencies?

The significant difference between the two is that python dependencies are "baked" into the image, while JS dependencies are stored in a volume.

#### Python

**Install packages:**

Use `invoke docker-pipenv "install [PACKAGE]"`.

:rotating_light: Important note! :rotating_light: This **only** add the dependency to your `Pipfile` and lock-it: it doesn't install your dependency! After running this command, run `docker-compose build backend` to create a new and updated backend image to use.

**Update packages:**

To update your dependencies, do `invoke docker-pipenv update`, then build the new image with `docker-compose build backend`.

#### JS

**Install packages:**

Use `invoke docker-npm "install [PACKAGE]"`.

**Update packages:**

Use `invoke docker-npm update`.

You don't need to rebuild the `watch-static-files` image.


---

## Docker vocabulary and overview

Welcome to Docker! Before jumping into Docker installation, take a moment to get familiar with Docker vocabulary:

- Docker: Docker is a platform to develop, deploy and run applications with containers.
- Docker engine: The Docker engine is a service running in the background (daemon). It's managing containers.
- Docker CLI: Command Line Interface to interact with Docker. For example, `Docker image ls` lists the images available on your system.
- Docker hub: Registry containing Docker images.
- Image: An image is a file used to build containers: In our case, it's mostly instructions to install dependencies.
- Container: Containers run an image. In our case, we have a container for the database, another one for building static files and the last one for running Django. A container life is ephemeral: data written there don't persist when you shut down a container.
- Volume: A volume is a special directory on your machine that is used to make data persistent. For example, we use it to store the database: that way, you don't lose your data when you turn down your containers.
- Host: host is used in Docker docs to mean the system on top of which containers run.
- Docker-compose: It's a tool to run multi-container applications: we use it to run our three containers together.
- Docker-compose CLI: Command line interface to interact with docker-compose. It's used to launch your dev environment.
- Docker-compose service: a service is a container and the configuration associated to it.

I would recommend watching [An Intro to Docker for Djangonauts](https://www.youtube.com/watch?v=qsEfVSTZO9Q) by Lacey Williams Henschel (25 min, [repo mentioned in the talk](https://github.com/williln/docker-hogwarts)): it's a great beginner talk to learn Docker and how to use it with Django.

### Project Structure

All our containers run on Linux.

For local development, we have two Dockerfiles that define our images:
- `Dockerfile.node`: use a node8-alpine base image from the Docker Hub and install node dependencies,
- `Dockerfile.python`: use a python3.6-alpine base image, install required build dependencies before installing pipenv and the project dependencies.
We don't have a custom image for running postgres and use one from the Docker Hub.

The `docker-compose.yml` file describes the 3 services that the project needs to run:
- `watch-static-files`: rebuilds static files when they're modified,
- `postgres`: contains a postgres database,
- `backend`: runs Django. Starting this one automatically starts the two other ones.

### Resources about Docker

- [Docker](https://docs.docker.com/) and [Docker-compose](https://docs.docker.com/compose/overview/) documentations,
- [Intro to Docker](https://www.revsys.com/tidbits/brief-intro-docker-djangonauts/): Lacey wrote a good intro tutorial to Docker and Django, without Harry Potter metaphors this time :),
- [Jérôme Petazzoni's training slides and talks](https://container.training/): presentations and slides if you want to dive into Docker.

---

## FAQ

### Do I need to build the static files before doing a `docker-compose up`?

Static files are automatically built when starting the `watch-static-files` container.

### Where is Docker fitting in all the tools we're already using?

Let's do a quick overview of all the tools you're currently using to run the foundation site on your computer:

- `npm`: use to manage javascript dependencies (`packages.json`, `packages-lock.json`). Also used to launch commands like `npm run start`.
- `pipenv`: use to manage python dependencies (`pipfile`, `pipfile.lock`). Also manage a python virtual environment, which isolates the foundation site's python packages from the rest of your system.
- `invoke`/`inv`: use as a cli tool to provide shortcuts for most used commands. ex: `inv runserver` is a shortcut for `pipenv run python network-api/manage.py runserver`.

We still use all those tools with Docker. The major difference is that `npm` and `pipenv` is now running inside a container, while invoke continues to run as before.

### Can I use Docker in parallel with the old way of running the foundation site?

Short answer is yes but:
- you will have two different databases
- you will have two files to manage your environments variables (`.env` and `.env.docker`),
- those two environment won't share their dependencies: you will have to maintain and update both of them.
@@ -32,10 +32,7 @@ But it's a bit long. So instead, you can use invoke:
- `inv test`: Run tests
- `inv catch-up`: Install dependencies and apply migrations

For management commands not covered by an invoke tasks, use `inv manage [command]` (example: `inv manage load_fake_data`). You can pass flag and options to management commands using `inv manage [command] -o [positional argument] -f [optional argument]`. For example:
- `inv manage runserver -o 3000`
- `inv manage load_fake_data -f seed=VALUE`
- `inv manage migrate -o news`
For management commands not covered by an invoke tasks, use `inv manage [command]` (example: `inv manage load_fake_data`). You can pass multiple arguments to `inv manage` by using double quotes. Ex: `inv manage "load_fake_data --delete"`.

### Generating a new set of fake model data

@@ -46,15 +43,15 @@ By default, your dev site will use production data (read only!). To load fake mo

You can empty your database and create a full new set of fake model data using the following command

- `inv manage load_fake_data -o --delete`
- `inv manage "load_fake_data --delete"`

Or

- `pipenv run python network-api/manage.py load_fake_data --delete`

You can generate a specific set of fake model data by entering a seed value

- `inv manage load_fake_data -o --delete --seed VALUE`
- `inv manage "load_fake_data --delete --seed VALUE"`

Or

@@ -10,7 +10,7 @@ TARGET_DOMAINS=foundation.mozilla.org

# network-api environment:

ALLOWED_HOSTS=localhost,127.0.0.1
ALLOWED_HOSTS=*
ASSET_DOMAIN=network.mofoprod.net
CONTENT_TYPE_NO_SNIFF=True
CORS_WHITELIST=*

0 comments on commit b422098

Please sign in to comment.
You can’t perform that action at this time.