feature/docker (#433) 🐳

* Updated env-config/config-deploy.py to override
default settings with environment variables
for postgres settings

* Adding the original Dockerfile from
Netflix-Skunkworks/zerotodocker

* Adding original entrypoint scripts from
Netflix-Skunkworks/zerotodocker

* Disable export of the SECURITY_MONKEY_SETTINGS
variable in these entrypoints as this should be
set before these entrypoints exist

* Remove unnecessary commented out lines

* Organize order of operations in api-init

* 1 - Add SECURITY_MONKEY_API_PORT environment
variable to override default settings defined
in SECURITY_MONKEY_SETTINGS

2 - Update entrypoints to use environment
variables

* Add original Dockerfile for nginx from
Netflix-Skunkworks/zerotodocker

* Add original securitymonkey.conf for nginx from
Netflix-Skunkworks/zerotodocker

* Add original insecure certs for tls provided
by Netflix-Skunkworks. These are meant to
act as a placeholder for the example.

* Move nginx resources to docker-nginx for simpler
docker build

* Add items to .gitignore and
.dockerignore to avoid committing
secrets

* Update Dockerfiles to build and run
SecurityMonkey from this directory,
as opposed to checking out the repository
directly, since this code has not yet been
merged. This can be used to build and
develop locally.

* Update FQDN settings to use environment
variables as override to default

* Update email settings to use environment
variable overrides

* Add documentation for SecurityMonkey Docker

* Add a forgotten comma

* Update entrypoints to work without chmod +x

* Add a missing equals sign '='

* Remove baked-in insecure certificates

* Add new entrypoint for nginx, disables ssl if cert:key pair are not found

* Add original nginx.conf for reference

* Turn daemon off in nginx.conf

* Update nginx docker build and entrypoint

* Move NGINX Dockerfile to docker-nginx/

* Refactor for local docker development
  - Removed python-m2crypto from Dockerfile
  - Created env-config/config-docker.py for settings
  - Reverted env-config/config-deploy.py back to v0.7.0 original
  - Moved docker-nginx directory to docker/nginx
  - Entrypoints are executable
  - Added brief documentation in docker folder

* More log changes for Docker

* Improvement for local docker-compose development

* Update securitymonkey.conf to latest from
http://securitymonkey.readthedocs.io/en/latest/quickstart.html#securitymonkey-conf

* Edit ssl crt/key location in securitymonkey.conf

* Make NGINX output logs to stdout/err for SecurityMonkey Docker

* Hardcode smapi NGINX endpoint instead of 127.0.0.1 for Docker

* Updated docker documentation

* Simplify config-docker.py in the use of
environment variables as overrides

* Add support for local insecure development
  - Enables 80 in NGINX
  - Toggle to disable CSRF in settings
  - Toggle to disable ssl in NGINX if
    certs are not provided

* Restore config-deploy.py from upstream

* Set API_PORT back to a String

.dockerignore

@@ -0,0 +1,8 @@
+.git
+secmonkey.env
+boto.cfg
+.travis.yml
+#docs
+supervisor
+config-default.py
+generate-docs.py

.gitignore

@@ -53,3 +53,8 @@ devlog/
 venv/
 .idea/
 
+boto.cfg
+secmonkey.env
+*.crt
+*.key
+

Dockerfile

@@ -0,0 +1,44 @@
+
+# Copyright 2014 Netflix, Inc.
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+#     http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+FROM ubuntu:14.04
+MAINTAINER Netflix Open Source Development <talent@netflix.com>
+
+ENV SECURITY_MONKEY_VERSION=v0.7.0 \
+    SECURITY_MONKEY_SETTINGS=/usr/local/src/security_monkey/env-config/config-docker.py
+
+RUN apt-get update &&\
+  apt-get -y -q install python-software-properties software-properties-common postgresql-9.3 postgresql-client-9.3 postgresql-contrib-9.3 curl &&\
+  apt-get install -y python-pip python-dev python-psycopg2 libffi-dev libpq-dev libyaml-dev libxml2-dev libxmlsec1-dev git sudo swig &&\
+  rm -rf /var/lib/apt/lists/*
+
+RUN cd /usr/local/src &&\
+#   git clone --branch $SECURITY_MONKEY_VERSION https://github.com/Netflix/security_monkey.git
+  /bin/mkdir -p security_monkey
+ADD . /usr/local/src/security_monkey
+
+RUN cd /usr/local/src/security_monkey &&\
+  python setup.py install &&\
+  /bin/mkdir -p /var/log/security_monkey/
+
+RUN chmod +x /usr/local/src/security_monkey/docker/*.sh &&\
+  mkdir -pv /var/log/security_monkey &&\
+  /usr/bin/touch /var/log/security_monkey/securitymonkey.log
+  # ln -s /dev/stdout /var/log/security_monkey/securitymonkey.log
+
+WORKDIR /usr/local/src/security_monkey
+EXPOSE 5000
+
+ENTRYPOINT ["/usr/local/src/security_monkey/docker/api-start.sh"]

docker-compose.yml

@@ -0,0 +1,86 @@
+---
+
+###
+#
+# Documentation: http://securitymonkey.readthedocs.io/en/latest/index.html
+#                http://securitymonkey.readthedocs.io/en/latest/docker.html
+#
+# shortcuts
+# open https://$(docker-machine active | xargs docker-machine ip)
+#
+###
+
+
+version: '2'
+services:
+  postgres:
+    container_name: secmonkey-db
+    image: postgres:9
+    # volumes:
+    #    - ./postgres-data/:/var/lib/postgresql/data
+
+  api:
+    container_name: secmonkey-api
+    image: secmonkey:latest
+    volumes_from:
+      - init
+    depends_on:
+      - postgres
+    env_file: secmonkey.env
+    entrypoint: ["/usr/local/src/security_monkey/docker/api-start.sh"]
+
+  scheduler:
+    container_name: secmonkey-scheduler
+    image: secmonkey:latest
+    volumes_from:
+      - init
+    depends_on:
+      - api
+    env_file: secmonkey.env
+    entrypoint: ["/usr/local/src/security_monkey/docker/scheduler-start.sh"]
+
+  nginx:
+    container_name: secmonkey-nginx
+    build:
+      context: ./
+      dockerfile: ./docker/nginx/Dockerfile
+    image: secmonkey-nginx:latest
+    working_dir: /etc/nginx
+    volumes:
+      - ./docker/nginx/server.crt:/etc/nginx/ssl/server.crt
+      - ./docker/nginx/server.key:/etc/nginx/ssl/server.key
+      - ./docker/nginx/securitymonkey.conf:/etc/nginx/conf.d/securitymonkey.conf
+      - ./docker/nginx/start-nginx.sh:/usr/local/src/security_monkey/docker/nginx/start-nginx.sh
+    depends_on:
+      - api
+    ports:
+      - 80:80
+      - 443:443
+    links:
+      - api:smapi
+
+# volumes:
+#    - postgres-data: {}
+
+###   ###   ###
+   ###   ###   ###
+
+  init:
+    container_name: init
+    build: .
+    image: secmonkey:latest
+    working_dir: /usr/local/src/security_monkey
+    volumes:
+      - ./data/aws_accounts.json:/usr/local/src/security_monkey/data/aws_accounts.json
+      - ./docker:/usr/local/src/security_monkey/docker/
+      - ./env-config/config-docker.py:/usr/local/src/security_monkey/env-config/config-docker.py
+    depends_on:
+      - postgres
+    env_file: secmonkey.env
+    # environment:
+    #   - AWS_ACCESS_KEY_ID=
+    #   - AWS_SECRET_ACCESS_KEY=
+    #   - SECURITY_MONKEY_POSTGRES_HOST=
+    entrypoint: # /usr/local/src/security_monkey/docker/api-init.sh
+      - sleep
+      - 8h

docker/README.rst

@@ -0,0 +1,9 @@
+************************
+Docker local development
+************************
+
+Project resources
+=================
+
+- `Docker documentation <http://securitymonkey.readthedocs.org/docker.html>`_
+- `Development documentation <http://securitymonkey.readthedocs.org/development.html>`_

docker/api-init.sh

@@ -0,0 +1,15 @@
+#!/bin/bash
+
+sudo -u ${SECURITY_MONKEY_POSTGRES_USER:-postgres} psql\
+    -h ${SECURITY_MONKEY_POSTGRES_HOST:-postgres} -p ${SECURITY_MONKEY_POSTGRES_PORT:-5432}\
+    --command "ALTER USER ${SECURITY_MONKEY_POSTGRES_USER:-postgres} with PASSWORD '${SECURITY_MONKEY_POSTGRES_PASSWORD:-securitymonkeypassword}';"
+
+sudo -u ${SECURITY_MONKEY_POSTGRES_USER:-postgres} createdb\
+    -h ${SECURITY_MONKEY_POSTGRES_HOST:-postgres} -p ${SECURITY_MONKEY_POSTGRES_PORT:-5432}\
+    -O ${SECURITY_MONKEY_POSTGRES_USER:-postgres} ${SECURITY_MONKEY_POSTGRES_DATABASE:-secmonkey}
+
+mkdir -p /var/log/security_monkey/
+touch "/var/log/security_monkey/security_monkey-deploy.log"
+
+cd /usr/local/src/security_monkey
+python manage.py db upgrade

docker/api-start.sh

@@ -0,0 +1,4 @@
+#!/bin/bash
+
+cd /usr/local/src/security_monkey
+python manage.py run_api_server -b 0.0.0.0:${SECURITY_MONKEY_API_PORT:-5000}

docker/nginx/Dockerfile

@@ -0,0 +1,50 @@
+# Copyright 2014 Netflix, Inc.
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+#     http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+FROM nginx:1.11.4
+MAINTAINER Netflix Open Source Development <talent@netflix.com>
+
+ENV SECURITY_MONKEY_VERSION=v0.7.0
+RUN apt-get update &&\
+  apt-get install -y curl git sudo apt-transport-https &&\
+  curl https://dl-ssl.google.com/linux/linux_signing_key.pub | apt-key add - &&\
+  curl https://storage.googleapis.com/download.dartlang.org/linux/debian/dart_stable.list > /etc/apt/sources.list.d/dart_stable.list && \
+  apt-get update &&\
+  apt-get install -y -q dart &&\
+  rm -rf /var/lib/apt/lists/*
+
+RUN cd /usr/local/src &&\
+#   git clone -b $SECURITY_MONKEY_VERSION https://github.com/Netflix/security_monkey.git
+  mkdir -p security_monkey
+ADD . /usr/local/src/security_monkey
+
+RUN cd /usr/local/src/security_monkey/dart &&\
+  /usr/lib/dart/bin/pub get &&\
+  /usr/lib/dart/bin/pub build &&\
+  /bin/mkdir -p /usr/local/src/security_monkey/security_monkey/static/ &&\
+  /bin/cp -R /usr/local/src/security_monkey/dart/build/web/* /usr/local/src/security_monkey/security_monkey/static/
+
+RUN /bin/rm /etc/nginx/conf.d/default.conf &&\
+  /bin/mkdir -p /var/log/security_monkey/ /etc/nginx/ssl/ &&\
+  ln -s /dev/stdout /var/log/security_monkey/security_monkey.access.log &&\
+  ln -s /dev/stderr /var/log/security_monkey/security_monkey.error.log
+
+WORKDIR /etc/nginx
+EXPOSE 443
+
+ADD docker/nginx/securitymonkey.conf /etc/nginx/conf.d/securitymonkey.conf
+COPY docker/nginx/nginx.conf /etc/nginx/nginx.conf
+# ADD docker/nginx/server.crt docker/nginx/server.key /etc/nginx/ssl/
+
+ENTRYPOINT ["/usr/local/src/security_monkey/docker/nginx/start-nginx.sh"]

docker/nginx/nginx.conf

@@ -0,0 +1,33 @@
+
+user  nginx;
+worker_processes  1;
+daemon off;
+
+error_log  /var/log/nginx/error.log warn;
+pid        /var/run/nginx.pid;
+
+
+events {
+    worker_connections  1024;
+}
+
+
+http {
+    include       /etc/nginx/mime.types;
+    default_type  application/octet-stream;
+
+    log_format  main  '$remote_addr - $remote_user [$time_local] "$request" '
+                      '$status $body_bytes_sent "$http_referer" '
+                      '"$http_user_agent" "$http_x_forwarded_for"';
+
+    access_log  /var/log/nginx/access.log  main;
+
+    sendfile        on;
+    #tcp_nopush     on;
+
+    keepalive_timeout  65;
+
+    #gzip  on;
+
+    include /etc/nginx/conf.d/*.conf;
+}

docker/nginx/securitymonkey.conf

@@ -0,0 +1,37 @@
+add_header X-Content-Type-Options "nosniff";
+add_header X-XSS-Protection "1; mode=block";
+add_header X-Frame-Options "SAMEORIGIN";
+add_header Strict-Transport-Security "max-age=631138519";
+add_header Content-Security-Policy "default-src 'self'; font-src 'self' https://fonts.gstatic.com; script-src 'self' https://ajax.googleapis.com; style-src 'self' https://fonts.googleapis.com;";
+
+server {
+   listen      0.0.0.0:80;
+   listen      0.0.0.0:443 ssl;
+   ssl_certificate /etc/nginx/ssl/server.crt;
+   ssl_certificate_key /etc/nginx/ssl/server.key;
+   access_log  /var/log/security_monkey/security_monkey.access.log;
+   error_log   /var/log/security_monkey/security_monkey.error.log;
+
+   location ~* ^/(reset|confirm|healthcheck|register|login|logout|api) {
+        proxy_read_timeout 120;
+        proxy_pass  http://smapi:5000;
+        proxy_next_upstream error timeout invalid_header http_500 http_502 http_503 http_504;
+        proxy_redirect off;
+        proxy_buffering off;
+        proxy_set_header        Host            $host;
+        proxy_set_header        X-Real-IP       $remote_addr;
+        proxy_set_header        X-Forwarded-For $proxy_add_x_forwarded_for;
+    }
+
+    location /static {
+        rewrite ^/static/(.*)$ /$1 break;
+        root /usr/local/src/security_monkey/security_monkey/static;
+        index ui.html;
+    }
+
+    location / {
+        root /usr/local/src/security_monkey/security_monkey/static;
+        index ui.html;
+    }
+
+}

docker/nginx/start-nginx.sh

@@ -0,0 +1,14 @@
+#!/usr/bin/env bash
+
+SECURITY_MONKEY_SSL_CERT=${SECURITY_MONKEY_SSL_CERT:-/etc/nginx/ssl/server.crt}
+SECURITY_MONKEY_SSL_KEY=${SECURITY_MONKEY_SSL_KEY:-/etc/nginx/ssl/server.key}
+
+if [ ! -f "$SECURITY_MONKEY_SSL_CERT" ] || [ ! -f "$SECURITY_MONKEY_SSL_KEY" ]; then
+    # Fail if SSL is unavailable
+    echo "$(date) Error: Missing files required for SSL"
+    # exit 1
+    sed -i.bak 's@.*ssl@# &@' /etc/nginx/conf.d/securitymonkey.conf &&\
+    echo "$(date) Warn: Disabled ssl in securitymonkey.conf"
+fi
+
+exec nginx

docker/scheduler-start.sh

@@ -0,0 +1,7 @@
+#!/bin/bash
+
+mkdir -p /var/log/security_monkey
+touch /var/log/security_monkey/security_monkey-deploy.log
+
+cd /usr/local/src/security_monkey
+python manage.py start_scheduler

docs/docker.rst

@@ -0,0 +1,51 @@
+Docker Instructions
+===================
+
+The docker-compose.yml file describes the SecurityMonkey environment. This is intended for local development with the intention of deploying SecurityMonkey containers with a Docker Orchestration tool like Kubernetes.
+
+The Dockerfile builds SecurityMonkey into a container with several different entrypoints. These are for the different responsibilities SecurityMonkey has.
+Also, the docker/nginx/Dockerfile file is used to build an NGINX container that will front the API, serve the static assets, and provide TLS.
+
+Quick Start:
+------------
+  Define your specific settings in **secmonkey.env** file. For example, this file will look like::
+
+    AWS_ACCESS_KEY_ID=
+    AWS_SECRET_ACCESS_KEY=
+    SECURITY_MONKEY_POSTGRES_HOST=postgres
+    SECURITY_MONKEY_FQDN=192.168.99.100
+
+  $ docker-compose build
+  ``this will locally build all the containers necessary``
+
+  $ docker-compose up -d postgres
+  ``this will start the database container``
+
+  $ docker-compose up -d init
+  ``this will start a container in which you canuse to setup the database, create users, and other manual configurations, see the below section for more info``
+
+  $ docker-compose up
+  ``this will bring up the remaining containers (scheduler and nginx)``
+
+Commands:
+---------
+
+  $ docker-compose build ``[api | scheduler | nginx | init]``
+
+  $ docker-compose up -d ``[postgres | api | scheduler | nginx | init]``
+
+More Info:
+----------
+::
+
+  $ docker-compose up -d init
+
+The init container is where the SecurityMonkey code is available for you to run manual configurations such as::
+
+  $ python manage.py create_user admin@example.com Admin
+
+and/or::
+
+  $ python manage.py add_account --number $account --name $name -r SecurityMonkey
+
+The init container provides a sandbox and is useful for local development. It is not required otherwise.

docs/quickstart.rst

@@ -8,6 +8,8 @@ Docker Images
 Before we start, consider following the `docker instructions <https://github.com/Netflix-Skunkworks/zerotodocker/wiki/Security-Monkey>`_
 . Docker helps simplify the process to get up and running.  The docker images are not currently ready for production use, but are good enough to get up and running with an instance of security_monkey.
 
+Local `docker instructions <./docker.html>`_
+
 Not into the docker thing? Keep reading.
 
 Setup IAM Roles