Merge pull request #1227 from DefectDojo/dev

Merge from Dev

DOCKER.md

@@ -4,30 +4,105 @@ Docker compose is not intended for production use.
 If you want to deploy a containerized DefectDojo to a production environment,
 use the [Helm and Kubernetes](KUBERNETES.md) approach.
 
-## Setup via Docker Compose
+## Prerequisites
+*  Docker version
+    *  Installing with docker-compose requires at least docker 18.09.4 and docker-compose 1.24.0. See "Checking Docker versions" below for version errors during running docker-compose.
+*  Proxies
+    *  If you're behind a corporate proxy check https://docs.docker.com/network/proxy/ . 
 
-To start your DefectDojo instance on Docker Compose for the first time, just
-run:
+
+## Setup via Docker Compose - introduction
+
+DefectDojo needs several docker images to run. Two of them depend on DefectDojo code:
+
+*  django service - defectdojo/defectdojo-django image 
+*  nginx service - defectdojo/defectdojo-nginx image
+
+The nginx image is build based on the django image.
+
+Before running the application, it's advised to build local images to make sure that you'll be working on images consistent with your current code base.
+When running the application without building images, the application will run based on: 
+*  a previously locally built image if it exists in the docker cache
+*  else the images pulled from dockerhub
+    *  https://hub.docker.com/r/defectdojo/defectdojo-django
+    *  https://hub.docker.com/r/defectdojo/defectdojo-nginx
+
+
+## Setup via Docker Compose - building and running the application
+### Building images
+
+To build images and put them in your local docker cache, run:
 
 ```zsh
-. docker/aliases_release.sh
-docker-compose up
+docker-compose build
 ```
 
+To build a single image, run: 
+
+```zsh
+docker-compose build django
+```
 or
 
+```
+docker-compose build nginx
+```
+
+
+### Run with Docker compose in release mode
+To run the application based on previously built image (or based on dockerhub images if none was locally built), run: 
+
 ```zsh
-docker-compose -f docker-compose_base.yml -f docker-compose_uwsgi-release.yml up
+docker/setEnv.sh release
+docker-compose up
 ```
 
-This command will run the application based on images commited on dockerhub (or the last images built locally). If you need to be more up to date, see "Build images locally" below
+This will run the application based on docker-compose.yml only.
+
+In this setup, you need to rebuild django and/or nginx images after each code change and restart the containers. 
 
-**NOTE:** Installing with docker-compose requires the latest version of docker and docker-compose - at least docker 18.09.4 and docker-compose 1.24.0. See "Checking Docker versions" below for version errors during running docker-compose up.
 
-**NOTE:** Installing with docker-compose requires the latest version of docker and docker-compose - at least docker 18.09.4 and docker-compose 1.24.0. See "Checking Docker versions" below for version errors during running docker-compose up.
+### Run with Docker compose in development mode with hot-reloading
 
+For development, use: 
+
+```zsh
+cp dojo/settings/settings.dist.py dojo/settings/settings.py
+docker/setEnv.sh dev
+docker-compose up
+```
+
+This will run the application based on merged configurations from docker-compose.yml and docker-compose.override.dev.yml.
+
+*  Volumes are mounted to synchronize between the host and the containers :
+    *  static resources (nginx container)
+    *  python code (uwsgi and celeryworker containers). 
+
+*  The `--py-autoreload 1` parameter in entrypoint-uwsgi-dev.sh will make uwsgi handle python hot-reloading for the **uwsgi** container.
+* Hot-reloading for the **celeryworker** container is not yet implemented. When working on deduplication for example, restart the celeryworker container with: 
+
+```
+docker restart django-defectdojo_celeryworker_1
+```
+
+*  The mysql port is forwarded to the host so that you can access your database from outside the container. 
+
+To update changes in static resources, served by nginx, just refresh the browser with ctrl + F5.
+
+
+*Notes about volume permissions*
+
+*The manual copy of settings.py is sometimes required once after cloning the repository, on linux hosts when the host files cannot be modified from within the django container. In that case that copy in entrypoint-uwsgi-dev.sh fails.* 
+
+*Another way to fix this is changing `USER 1001` in Dockerfile.django to match your user uid and then rebuild the images. Get your user id with* 
+
+```
+id -u
+```
+
+### Access the application
 Navigate to <http://localhost:8080> where you can log in with username admin.
-To find out the admin user’s password, check the very beginning of the console
+To find out the admin password, check the very beginning of the console
 output of the initializer container, typically name 'django-defectdojo_initializer_1', or run the following:
 
 ```zsh
@@ -43,45 +118,38 @@ or:
 docker logs django-defectdojo_initializer_1
 ```
 
-If you ran DefectDojo with compose before and you want to prevent the
-initializer container from running again, define an environment variable
-DD_INITIALIZE=false to prevent re-initialization.
-
-### Develop with Docker Compose
-
-For developing the easiset way to make changes is to startup DefectDojo in debug by running:
+Beware that when re-running the application several times, there may be several occurrences of "Admin password". In that case you should use the last occurrence.
 
-```zsh
-. docker/aliases_dev.sh
-docker-compose up
-```
+### Disable the database initialization
+The initializer container can be disabled by exporting: `export DD_INITIALIZE=false`. 
 
-or
+This will ensure that the database remains unchanged when re-running the application, keeping your previous settings and admin password.
 
-```zsh
-docker-compose -f docker-compose_base.yml -f docker-compose_uwsgi-dev.yml up
-```
+### Versioning
+In order to use a specific version when building the images and running the containers, set the environment with 
+*  For the nginx image: `NGINX_VERSION=x.y.z`
+*  For the django image: `DJANGO_VERSION=x.y.z`
 
-This starts the DefectDojo (uwsgi) container with manage.py and shares the local source directory so that changes to the code immediately restart the process.
+Building will tag the images with "x.y.z", then you can run the application based on a specific tagged images.
 
-Navigate to the container directly, <http://localhost:8000>
+*  Tagged images can be seen with: 
 
-The initializer container can be disabled by exporting: `export DD_INITIALIZE=false`
+```
+$ docker images
+REPOSITORY                     TAG                 IMAGE ID            CREATED             SIZE
+defectdojo/defectdojo-nginx    1.0.0               bc9c5f7bb4e5        About an hour ago   191MB
+```
 
-### Build Images Locally
+*  This will show on which tagged images the containers are running:
 
-Build the docker containers locally for testing purposes.
+``` 
+$ docker ps
+CONTAINER ID        IMAGE                                 COMMAND                  CREATED             STATUS              PORTS                                NAMES
+aedc404d6dee        defectdojo/defectdojo-nginx:1.0.0     "/entrypoint-nginx.sh"   2 minutes ago       Up 2 minutes        80/tcp, 0.0.0.0:8080->8080/tcp       django-defectdojo_nginx_1
+```
 
-```zsh
-# Build Dev Compose
-docker-compose build
 
-or:
 
-# Build images
-docker build -t defectdojo/defectdojo-django -f Dockerfile.django .
-docker build -t defectdojo/defectdojo-nginx -f Dockerfile.nginx .
-```
 
 ### Clean up Docker Compose
 
@@ -97,6 +165,43 @@ Removes all containers, networks and the database volume
 docker-compose down --volumes
 ```
 
+### Run the unit-tests with docker
+#### Introduction
+The unit-tests are under `dojo/unittests`
+
+
+
+#### Running the unit-tests 
+This will run all the tests and leave the uwsgi container up: 
+
+```
+cp dojo/settings/settings.dist.py dojo/settings/settings.py
+docker/setEnv.sh unit_tests
+docker-compose up
+```
+Enter the container to run more tests:
+
+```
+docker exec -it django-defectdojo_uwsgi_1 bash
+```
+Rerun all the tests:
+
+```
+python manage.py test dojo.unittests --keepdb
+```
+
+Run all the tests from a python file. Example:
+
+```
+python manage.py test dojo.unittests.test_dependency_check_parser --keepdb
+```
+
+Run a single test. Example:
+
+```
+python manage.py test dojo.unittests.test_dependency_check_parser.TestDependencyCheckParser.test_parse_without_file_has_no_findings --keepdb
+```
+
 ## Checking Docker versions
 
 Run the following to determine the versions for docker and docker-compose:
@@ -129,7 +234,7 @@ OpenSSL version: OpenSSL 1.0.1t  3 May 2016
 
 In this case, both docker (version 17.09.0-ce) and docker-compose (1.18.0) need to be updated.
 
-Follow [Dockers' documentation](https://docs.docker.com/install/) for your OS to get the lastest version of Docker. For the docker command, most OSes have a built-in update mechanism like "apt upgrade".
+Follow [Dockers' documentation](https://docs.docker.com/install/) for your OS to get the latest version of Docker. For the docker command, most OSes have a built-in update mechanism like "apt upgrade".
 
 Docker Compose isn't packaged like Docker and you'll need to manually update an existing install if using Linux. For Linux, either follow the instructions in the [Docker Compose documentation](https://docs.docker.com/compose/install/) or use the shell script below. The script below will update docker-compose to the latest version automatically. You will need to make the script executable and have sudo privileges to upgrade docker-compose:
 
@@ -147,7 +252,7 @@ echo "Note: docker-compose version $VERSION will be downloaded from:"
 echo "https://github.com/docker/compose/releases/download/${VERSION}/docker-compose-$(uname -s)-$(uname -m)"
 echo "Enter sudo password to install docker-compose"
 
-# Download and install lastest docker compose
+# Download and install latest docker compose
 sudo curl -L https://github.com/docker/compose/releases/download/${VERSION}/docker-compose-$(uname -s)-$(uname -m) -o $DESTINATION
 sudo chmod +x $DESTINATION
 

Dockerfile.django

@@ -1,26 +1,62 @@
 
 # code: language=Dockerfile
-FROM python:2
+
+# The code for the build image should be idendical with the code in
+# Dockerfile.nginx to use the caching mechanism of Docker.
+
+FROM python:2 as build
 WORKDIR /app
 RUN \
   apt-get -y update && \
   apt-get -y install \
     dnsutils \
     mysql-client \
     postgresql-client \
+    xmlsec1 \
+    && \
+  apt-get clean && \
+  rm -rf /var/lib/apt/lists && \
+  true
+COPY requirements.txt ./
+RUN pip wheel --wheel-dir=/tmp/wheels -r ./requirements.txt
+
+FROM python:2-slim
+WORKDIR /app
+RUN \
+  apt-get -y update && \
+  # ugly fix to install postgresql-client without errors
+  mkdir -p /usr/share/man/man1 /usr/share/man/man7 && \
+  apt-get -y install --no-install-recommends \
+    # libopenjp2-7 libjpeg62 libtiff5 are required by the pillow package
+    libopenjp2-7 \
+    libjpeg62 \
+    libtiff5 \
+    dnsutils \
+    mysql-client \
+    libmariadbclient18 \
+    xmlsec1 \
+    # only required for the dbshell (used by the initializer job)
+    postgresql-client \
     && \
   apt-get clean && \
   rm -rf /var/lib/apt/lists && \
   true
+RUN pip install --no-cache-dir --upgrade pip
+COPY --from=build /tmp/wheels /tmp/wheels
 COPY requirements.txt ./
-RUN pip install -r ./requirements.txt
+RUN pip install \
+	--no-cache-dir \
+	--no-index \
+  --find-links=/tmp/wheels \
+  -r ./requirements.txt
 COPY \
   docker/entrypoint-celery-beat.sh \
   docker/entrypoint-celery-worker.sh \
   docker/entrypoint-initializer.sh \
   docker/entrypoint-uwsgi.sh \
   docker/entrypoint-uwsgi-dev.sh \
   docker/entrypoint-unit-tests.sh \
+  docker/entrypoint-unit-tests-devDocker.sh \
   docker/wait-for-it.sh \
   /
 COPY wsgi.py manage.py tests/unit-tests.sh ./
@@ -38,7 +74,7 @@ USER 1001
 ENV \
   DD_ADMIN_USER=admin \
   DD_ADMIN_MAIL=admin@defectdojo.local \
-  DD_ADMIN_PASSWORD= \
+  DD_ADMIN_PASSWORD='' \
   DD_ADMIN_FIRST_NAME=Administrator \
   DD_ADMIN_LAST_NAME=User \
   DD_ALLOWED_HOSTS="*" \

Dockerfile.nginx

@@ -1,5 +1,25 @@
 # code: language=Dockerfile
-FROM defectdojo/defectdojo-django:latest AS build
+
+# The code for the build image should be idendical with the code in
+# Dockerfile.django to use the caching mechanism of Docker.
+
+FROM python:2 as build
+WORKDIR /app
+RUN \
+  apt-get -y update && \
+  apt-get -y install \
+    dnsutils \
+    mysql-client \
+    postgresql-client \
+    xmlsec1 \
+    && \
+  apt-get clean && \
+  rm -rf /var/lib/apt/lists && \
+  true
+COPY requirements.txt ./
+RUN pip wheel --wheel-dir=/tmp/wheels -r ./requirements.txt
+
+FROM build AS collectstatic
 
 USER root
 RUN \
@@ -15,7 +35,18 @@ RUN \
   apt-get clean && \
   rm -rf /var/lib/apt/lists && \
   true
+
+RUN pip install \
+	--no-cache-dir \
+	--no-index \
+  --find-links=/tmp/wheels \
+	-r ./requirements.txt
+
 COPY components/ ./components/
+COPY manage.py ./
+COPY dojo/ ./dojo/
+RUN \
+  cp dojo/settings/settings.dist.py dojo/settings/settings.py
 RUN \
   cd components && \
   yarn && \
@@ -24,7 +55,7 @@ RUN \
   true
 
 FROM nginx
-COPY --from=build /app/static/ /usr/share/nginx/html/static/
+COPY --from=collectstatic /app/static/ /usr/share/nginx/html/static/
 COPY wsgi_params nginx/nginx.conf /etc/nginx/
 COPY docker/entrypoint-nginx.sh /
 RUN \

PULL_REQUEST_TEMPLATE.md

@@ -2,7 +2,7 @@ Please submit your pull requests to the 'dev' branch.
 
 When submitting a pull request, please make sure you have completed the following checklist:
 
-- [ ] Your code is flake8 compliant (DefectDojo's code isn't currently flake8 compliant, but we're trying to correct that.)
+- [ ] Your code is flake8 compliant 
 - [ ] If this is a new feature and not a bug fix, you've included the proper documentation in the ReadTheDocs documentation folder. https://github.com/DefectDojo/Documentation/tree/master/docs or provide feature documentation in the PR.
-- [ ] Model changes should include the necessary migrations in the dojo/dd_migrations folder.
+- [ ] Model changes must include the necessary migrations in the dojo/dd_migrations folder.
 - [ ] Add applicable tests to the unit tests.