Provision Streamlined, Quick Setup Added, and Multiple Containers Sup…

…port (#535)

* Separate docker containers per service

* Provision Streamlined, Quick Setup Added, and Multiple Containers Support

* The project now includes a number of "Quick Setup" options to ease the installation or startup process of the platform.  The following Quick Setup modes are available:

  *  Direct Installation - Used when directly installing to the system you are on; this is useful when installing on bare metal, an existing VM, or a cloud-based host.

      * `source ./extra/lib.sh`
      * `quick_setup install <dev/prod>`

  * Multi-Server Direct Installation - Used when directly installing the platform with each service on a separate system; this is useful when installing on bare metal systems, existing VMs, or cloud-based hosts.

    * Database Server (MySQL)
      * `source ./extra/lib.sh`
      * `quick_setup install_multi_mysql <dev/prod>`

    * Cache Server (Memcached)
      * `source ./extra/lib.sh`
      * `quick_setup install_multi_nginx <dev/prod>`

    * HHVM Server (HHVM)
      * `source ./extra/lib.sh`
      * `quick_setup install_multi_hhvm <dev/prod> <IP of MySQL Server> <IP of Memcached Server>`

    * Web Server (Nginx)
      * `source ./extra/lib.sh`
      * `quick_setup install_multi_nginx <dev/prod> <IP of HHVM Server>`

  * Standard Docker Startup - Used when running FBCTF as a single docker container.
      * `source ./extra/lib.sh`
      * `quick_setup start_docker <dev/prod>`

  * Multi-Container Docker Startup - Used when running FBCTF on docker with each service hosted in a separate docker container.
      * `source ./extra/lib.sh`
      * `quick_setup start_docker_multi <dev/prod>`

  * Standard Vagrant Startup - Used when running FBCTF as a single vagrant container.
      * `source ./extra/lib.sh`
      * `quick_setup start_docker <dev/prod>`

  * Multi-Container Vagrant Startup - Used when running FBCTF on vagrant with each service hosted in a separate vagrant container.
      * `source ./extra/lib.sh`
      * `quick_setup start_docker_multi <dev/prod>`

* Each installation platform now supports both Production Mode (prod) and Development Mode (dev).

* The `provision.sh` script has been streamlined and organized based on the services being installed.  The installation process now also includes more logging and error handling.  Common and core functionally has been migrated to `lib.sh` where appropriate.  Color coding has been added to the various output to make quick visual monitoring of the process easier.

* Package installation, specifically the check for existing packages has been updated to fix an issue where packages would sometimes not be installed if a similarly named package was already present on the system.

* The `provision.sh` script now supports separate installations for each service using the `--multiple-servers` and `--server-type` options.

* HHVM configuration has been updated to run HHVM as a network-service.

* Nginx configuration is now included in the platform code base and utilized.

* Docker service startup scripts are included for each of the services:
  * `./extra/mysql/mysql_startup.sh`
  * `./extra/hhvm/hhvm_startup.sh`
  * `./extra/nginx/nginx_startup.sh`

* This PR fixes the docker installation dependencies issue #534.

* This PR includes docker-compose configurations for multi-docker containers, fixing issue #440.

* Services on Docker (both single container and multi-container) are now monitored to ensure they do not fail.

* This PR updates HHVM to the latest stable version for Ubuntu 14.04, HHVM Version 3.18.1, fixing issue #496.

* Attachment/Upload permissions have been corrected across the installation environments.  This fixes issues with improper permissions on Docker and Vagrant while still enforcing secure file permissions.  This should resolve issues like #280 going forward.

* Implemented more strict permissions on he CTF PATH (755 verses 777).

* Fixed long-standing, upstream induced, HHVM socket permission issues (like #229), mostly experienced in Docker or after a restart (resulting in a _502 Bad Gateway_):  facebook/hhvm#6336.  Note that this fix is a temporary workaround until the upstream issue is resolved.

* With the introduction of the latest available version of HHVM and the inclusion of multiple-server support, performance increases should be noticeable.  This should help alleviate issues like #456.

* This PR was derived, in part, from PR #530.

* Added Memcached Service Restart to container service script

* Added logging of PHP/HHVM version to provision script.

* Added logging of PHP Alternatives to provision script.

* Composer is now installed with the HHVM binary instead of PHP.

* Composer Install is run with the HHVM binary instead of PHP.

* The Travis trusty Ubuntu image has been downgraded from `sugilite` to `connie`.

* Updated run_tests.sh to have write permissions to settings.ini

* Set run_tests.sh to use localhost for DB and MC.

* HHVM 3.18+ enforces \HH\FormatString - Invariant calls now are of \HH\FormatString type - All `invariant()` calls that are passing in a variable argument have been updated to use literal strings for the format string.  Invariant passes the second (and subsequent) arguments to `sprintf()`.  The second parameter of `invariant()` must be a literal string, containing placeholders when needed.  More information can be found here:  hhvm/user-documentation#448.  This change ensures the code is strict compliant in HHVM versions 3.18

.travis.yml

@@ -4,6 +4,8 @@ language: generic
 sudo: required
 dist: trusty
 
+group: deprecated-2017Q2
+
 install: ./extra/provision.sh -m dev -s $TRAVIS_BUILD_DIR -d $TRAVIS_BUILD_DIR
 
 script: ./extra/run_tests.sh $TRAVIS_BUILD_DIR

Dockerfile

@@ -12,14 +12,7 @@ ARG CRT
 
 WORKDIR $HOME
 COPY . $HOME
-RUN apt-get update \
-  && apt-get install -y \
-  rsync \
-  curl \
-  ca-certificates \
-  && chown www-data:www-data $HOME \
-  && ./extra/provision.sh -m $MODE -c $TYPE -k $KEY -C $CRT -D $DOMAIN -e $EMAIL -s `pwd` --docker \
-  && rm -f /var/run/hhvm/sock \
-  && apt-get clean \
-  && rm -rf /var/lib/apt/lists/* /tmp/* /var/tmp/*
-CMD ["./extra/service_startup.sh"]
+RUN chown www-data:www-data $HOME
+
+RUN ./extra/provision.sh -m $MODE -c $TYPE -k $KEY -C $CRT -D $DOMAIN -e $EMAIL -s `pwd` --docker
+CMD ["./extra/service_startup.sh"]

Vagrantfile

@@ -6,7 +6,7 @@ VAGRANTFILE_API_VERSION = "2"
 Vagrant.configure(VAGRANTFILE_API_VERSION) do |config|
   config.vm.box = "ubuntu/trusty64"
   config.vm.network "private_network", ip: "10.10.10.5"
-  config.vm.hostname = "facebookCTF-Dev"
+  config.vm.hostname = "FacebookCTF-Dev"
   config.ssh.shell = "bash -c 'BASH_ENV=/etc/profile exec bash'"
   config.vm.provision "shell", path: "extra/provision.sh", args: ENV['FBCTF_PROVISION_ARGS'], privileged: false
   config.vm.provider "virtualbox" do |v|

Vagrantfile-multi

@@ -0,0 +1,56 @@
+# -*- mode: ruby -*-
+# vi: set ft=ruby :
+
+VAGRANTFILE_API_VERSION = "2"
+
+Vagrant.configure(VAGRANTFILE_API_VERSION) do |config|
+  config.vm.box = "ubuntu/trusty64"
+  config.ssh.shell = "bash -c 'BASH_ENV=/etc/profile exec bash'"
+
+  # MySQL Server
+  config.vm.define "mysql" do |mysql|
+    mysql.vm.network "private_network", ip: "10.10.10.6"
+    mysql.vm.hostname = "mysql"
+    mysql.vm.provision "shell", path: "extra/provision.sh", args: "ENV['FBCTF_PROVISION_ARGS'] --multiple-servers --server-type mysql", privileged: false
+    mysql.vm.provider "virtualbox" do |v|
+      v.memory = 1024
+      v.cpus = 2
+    end
+  end
+
+  # Cache Server
+  config.vm.define "cache" do |cache|
+    cache.vm.network "private_network", ip: "10.10.10.8"
+    cache.vm.hostname = "cache"
+    cache.vm.provision "shell", path: "extra/provision.sh", args: "ENV['FBCTF_PROVISION_ARGS'] --multiple-servers --server-type cache", privileged: false
+    cache.vm.provider "virtualbox" do |v|
+      v.memory = 1024
+      v.cpus = 2
+    end
+  end
+
+  # HHVM Server
+  config.vm.define "hhvm" do |hhvm|
+    hhvm.vm.network "private_network", ip: "10.10.10.7"
+    hhvm.vm.hostname = "hhvm"
+    hhvm.vm.provision "shell", path: "extra/provision.sh", args: "ENV['FBCTF_PROVISION_ARGS'] --multiple-servers --server-type hhvm --mysql-server 10.10.10.6 --cache-server 10.10.10.8", privileged: false
+    hhvm.vm.provider "virtualbox" do |v|
+      v.memory = 1024
+      v.cpus = 2
+    end
+  end
+
+  # Nginx Server
+  config.vm.define "nginx" do |nginx|
+    nginx.vm.network "private_network", ip: "10.10.10.5"
+    nginx.vm.network "forwarded_port", guest: 80, host: 80
+    nginx.vm.network "forwarded_port", guest: 443, host: 443
+    nginx.vm.hostname = "nginx"
+    nginx.vm.provision "shell", path: "extra/provision.sh", args: "ENV['FBCTF_PROVISION_ARGS'] --multiple-servers --server-type nginx --hhvm-server 10.10.10.7", privileged: false
+    nginx.vm.provider "virtualbox" do |v|
+      v.memory = 1024
+      v.cpus = 2
+    end
+  end
+
+end

Vagrantfile-single

@@ -0,0 +1,16 @@
+# -*- mode: ruby -*-
+# vi: set ft=ruby :
+
+VAGRANTFILE_API_VERSION = "2"
+
+Vagrant.configure(VAGRANTFILE_API_VERSION) do |config|
+  config.vm.box = "ubuntu/trusty64"
+  config.vm.network "private_network", ip: "10.10.10.5"
+  config.vm.hostname = "facebookCTF-Dev"
+  config.ssh.shell = "bash -c 'BASH_ENV=/etc/profile exec bash'"
+  config.vm.provision "shell", path: "extra/provision.sh", args: ENV['FBCTF_PROVISION_ARGS'], privileged: false
+  config.vm.provider "virtualbox" do |v|
+    v.memory = 4096
+    v.cpus = 4
+  end
+end

docker-compose.yml

@@ -0,0 +1,49 @@
+version: '2'
+services:
+  mysql:
+    restart: always
+    build:
+      context: .
+      dockerfile: extra/mysql/Dockerfile
+      #args:
+      #  MODE: prod
+    environment:
+      MYSQL_ROOT_PASSWORD: root
+    expose:
+      - "3306"
+
+  cache:
+    restart: always
+    build:
+      context: .
+      dockerfile: extra/cache/Dockerfile
+      #args:
+      #  MODE: prod
+    expose:
+      - "11211"
+
+  hhvm:
+    restart: always
+    build:
+      context: .
+      dockerfile: extra/hhvm/Dockerfile
+      #args:
+      #  MODE: prod
+    depends_on:
+      - mysql
+      - cache
+    expose:
+      - "9000"
+
+  nginx:
+    restart: always
+    build:
+      context: .
+      dockerfile: extra/nginx/Dockerfile
+      #args:
+      #  MODE: prod
+    depends_on:
+      - hhvm
+    ports:
+      - "80:80"
+      - "443:443"

extra/cache/Dockerfile

@@ -0,0 +1,17 @@
+FROM ubuntu:trusty
+LABEL maintainer="Boik Su <boik@tdohacker.org>"
+
+ENV HOME /root
+
+ARG DOMAIN
+ARG EMAIL
+ARG MODE=dev
+ARG TYPE=self
+ARG KEY
+ARG CRT
+
+WORKDIR $HOME
+COPY . $HOME
+
+RUN ./extra/provision.sh -m $MODE -c $TYPE -k $KEY -C $CRT -D $DOMAIN -e $EMAIL -s `pwd` --docker --multiple-servers --server-type cache
+CMD ["./extra/cache/cache_startup.sh"]

extra/cache/cache_startup.sh

@@ -0,0 +1,11 @@
+#!/bin/bash
+
+set -e
+
+service memcached restart
+
+while true; do
+    sleep 5
+
+    service memcached status
+done

extra/hhvm.conf

@@ -9,6 +9,7 @@ hhvm.enable_xhp = true
 hhvm.force_hh = true
 hhvm.server.type = fastcgi
 hhvm.server.ip = 127.0.0.1
+hhvm.server.port = 9000
 hhvm.server.file_socket = /var/run/hhvm/sock
 hhvm.server.default_document = index.php
 hhvm.server.upload.upload_max_file_size = 25M

extra/hhvm/Dockerfile

@@ -0,0 +1,17 @@
+FROM ubuntu:trusty
+LABEL maintainer="Boik Su <boik@tdohacker.org>"
+
+ENV HOME /root
+
+ARG DOMAIN
+ARG EMAIL
+ARG MODE=dev
+ARG TYPE=self
+ARG KEY
+ARG CRT
+
+WORKDIR $HOME
+COPY . $HOME
+
+RUN ./extra/provision.sh -m $MODE -c $TYPE -k $KEY -C $CRT -D $DOMAIN -e $EMAIL -s `pwd` --docker --multiple-servers --server-type hhvm --mysql-server mysql --cache-server cache
+CMD ["./extra/hhvm/hhvm_startup.sh"]

extra/hhvm/hhvm_startup.sh

@@ -0,0 +1,15 @@
+#!/bin/bash
+
+set -e
+
+service hhvm restart
+
+while true; do
+    if [[ -e /var/run/hhvm/sock ]]; then
+        chown www-data:www-data /var/run/hhvm/sock
+    fi
+
+    sleep 5
+
+    service hhvm status
+done