Provision Streamlined, Quick Setup Added, and Multiple Containers Support #535

justinwray · 2017-07-19T17:05:58Z

The project now includes a number of "Quick Setup" options to ease the installation or startup process of the platform. The following Quick Setup modes are available:
- Direct Installation - Used when directly installing to the system you are on; this is useful when installing on bare metal, an existing VM, or a cloud-based host.
  - source ./extra/lib.sh
  - quick_setup install <dev/prod>
- Multi-Server Direct Installation - Used when directly installing the platform with each service on a separate system; this is useful when installing on bare metal systems, existing VMs, or cloud-based hosts.
  - Database Server (MySQL)
    - source ./extra/lib.sh
    - quick_setup install_multi_mysql <dev/prod>
  - HHVM Server (HHVM)
    - source ./extra/lib.sh
    - quick_setup install_multi_hhvm <dev/prod> <IP of MySQL Server>
  - Web Server (Nginx)
    - source ./extra/lib.sh
    - quick_setup install_multi_nginx <dev/prod> <IP of HHVM Server>
- Standard Docker Startup - Used when running FBCTF as a single docker container.
  - source ./extra/lib.sh
  - quick_setup start_docker <dev/prod>
- Multi-Container Docker Startup - Used when running FBCTF on docker with each service hosted in a separate docker container.
  - source ./extra/lib.sh
  - quick_setup start_docker_multi <dev/prod>
- Standard Vagrant Startup - Used when running FBCTF as a single vagrant container.
  - source ./extra/lib.sh
  - quick_setup start_docker <dev/prod>
- Multi-Container Vagrant Startup - Used when running FBCTF on vagrant with each service hosted in a separate vagrant container.
  - source ./extra/lib.sh
  - quick_setup start_docker_multi <dev/prod>
Each installation platform now supports both Production Mode (prod) and Development Mode (dev).
The provision.sh script has been streamlined and organized based on the services being installed. The installation process now also includes more logging and error handling. Common and core functionally has been migrated to lib.sh where appropriate. Color coding has been added to the various output to make quick visual monitoring of the process easier.
Package installation, specifically the check for existing packages has been updated to fix an issue where packages would sometimes not be installed if a similarly named package was already present on the system.
The provision.sh script now supports separate installations for each service using the --multiple-servers and --server-type options.
HHVM configuration has been updated to run HHVM as a network-service.
Nginx configuration is now included in the platform code base and utilized.
Docker service startup scripts are included for each of the services:
- ./extra/mysql/mysql_startup.sh
- ./extra/hhvm/hhvm_startup.sh
- ./extra/nginx/nginx_startup.sh
This PR fixes the docker installation dependencies issue Problems running development in docker #534.
This PR includes docker-compose configurations for multi-docker containers, fixing issue split services in Dockerfile out into docker-compose.yaml #440.
Services on Docker (both single container and multi-container) are now monitored to ensure they do not fail.
This PR updates HHVM to the latest stable version for Ubuntu 14.04, HHVM Version 3.18.1, fixing issue Upgrade HHVM to version 3.18 #496.
Attachment/Upload permissions have been corrected across the installation environments. This fixes issues with improper permissions on Docker and Vagrant while still enforcing secure file permissions. This should resolve issues like Upload failure #280 going forward.
Implemented more strict permissions on he CTF PATH (755 verses 777).
Fixed long-standing, upstream induced, HHVM socket permission issues (like Docker production deployment fails - rm: cannot remove '/var/run/hhvm/sock': No such file or directory #229), mostly experienced in Docker or after a restart (resulting in a 502 Bad Gateway): Socket Permission facebook/hhvm#6336. Note that this fix is a temporary workaround until the upstream issue is resolved.
With the introduction of the latest available version of HHVM and the inclusion of multiple-server support, performance increases should be noticeable. This should help alleviate issues like Performance Issue(s) #456.
NOTE: HHVM and Memcached currently always reside on the same server/container, this is due to an unidentified bug when connecting from HHVM to a remote Memcached service, resulting in a core dump. As more testing is completed on this issue, a bug report can be filed upstream.
This PR was derived, in part, from PR Separate docker containers per service #530.

…port * The project now includes a number of "Quick Setup" options to ease the installation or startup process of the platform. The following Quick Setup modes are available: * Direct Installation - Used when directly installing to the system you are on; this is useful when installing on bare metal, an existing VM, or a cloud-based host. * `source ./extra/lib.sh` * `quick_setup install <dev/prod>` * Multi-Server Direct Installation - Used when directly installing the platform with each service on a separate system; this is useful when installing on bare metal systems, existing VMs, or cloud-based hosts. * Database Server (MySQL) * `source ./extra/lib.sh` * `quick_setup install_multi_mysql <dev/prod>` * HHVM Server (HHVM) * `source ./extra/lib.sh` * `quick_setup install_multi_hhvm <dev/prod> <IP of MySQL Server>` * Web Server (Nginx) * `source ./extra/lib.sh` * `quick_setup install_multi_nginx <dev/prod> <IP of HHVM Server>` * Standard Docker Startup - Used when running FBCTF as a single docker container. * `source ./extra/lib.sh` * `quick_setup start_docker <dev/prod>` * Multi-Container Docker Startup - Used when running FBCTF on docker with each service hosted in a separate docker container. * `source ./extra/lib.sh` * `quick_setup start_docker_multi <dev/prod>` * Standard Vagrant Startup - Used when running FBCTF as a single vagrant container. * `source ./extra/lib.sh` * `quick_setup start_docker <dev/prod>` * Multi-Container Vagrant Startup - Used when running FBCTF on vagrant with each service hosted in a separate vagrant container. * `source ./extra/lib.sh` * `quick_setup start_docker_multi <dev/prod>` * Each installation platform now supports both Production Mode (prod) and Development Mode (dev). * The `provision.sh` script has been streamlined and organized based on the services being installed. The installation process now also includes more logging and error handling. Common and core functionally has been migrated to `lib.sh` where appropriate. Color coding has been added to the various output to make quick visual monitoring of the process easier. * Package installation, specifically the check for existing packages has been updated to fix an issue where packages would sometimes not be installed if a similarly named package was already present on the system. * The `provision.sh` script now supports separate installations for each service using the `--multiple-servers` and `--server-type` options. * HHVM configuration has been updated to run HHVM as a network-service. * Nginx configuration is now included in the platform code base and utilized. * Docker service startup scripts are included for each of the services: * `./extra/mysql/mysql_startup.sh` * `./extra/hhvm/hhvm_startup.sh` * `./extra/nginx/nginx_startup.sh` * This PR fixes the docker installation dependencies issue #534. * This PR includes docker-compose configurations for multi-docker containers, fixing issue #440. * Services on Docker (both single container and multi-container) are now monitored to ensure they do not fail. * This PR updates HHVM to the latest stable version for Ubuntu 14.04, HHVM Version 3.18.1, fixing issue #496. * Attachment/Upload permissions have been corrected across the installation environments. This fixes issues with improper permissions on Docker and Vagrant while still enforcing secure file permissions. This should resolve issues like #280 going forward. * Implemented more strict permissions on he CTF PATH (755 verses 777). * Fixed long-standing, upstream induced, HHVM socket permission issues (like #229), mostly experienced in Docker or after a restart (resulting in a _502 Bad Gateway_): facebook/hhvm#6336. Note that this fix is a temporary workaround until the upstream issue is resolved. * With the introduction of the latest available version of HHVM and the inclusion of multiple-server support, performance increases should be noticeable. This should help alleviate issues like #456. * NOTE: HHVM and Memcached currently always reside on the same server/container, this is due to an unidentified bug when connecting from HHVM to a remote Memcached service, resulting in a core dump. As more testing is completed on this issue, a bug report can be filed upstream. * This PR was derived, in part, from PR #530.

…entation

* Memcached can now be run in a separate container. * "cache" server-type has been added to the provision script. * The --cache-server parameter has been added to provision, to supply the IP address of the Memcached server. * Docker and Vagrant multi-container installations have been updated to support the new Memcached container. * The Quick Install options have been updated to support the new Memcached container.

* Invariant passes the second (and subsequent) arguments to `sprintf()`. The second parameter of `invariant()` must be a literal string, containing placeholders when needed. * More information can be found here: hhvm/user-documentation#448 * All `invariant()` calls that are passing in a variable argument have been updated to use literal strings for the format string. * This change ensures the code is strict compliant in HHVM versions 3.18

fredemmott · 2017-07-24T16:27:07Z

src/Utils.php

@@ -9,6 +9,8 @@ function must_have_idx<Tk, Tv>(?KeyedContainer<Tk, Tv> $arr, Tk $idx): Tv {
  $result = idx($arr, $idx);
  invariant(
    $result !== null,
+    /* HH_IGNORE_ERROR[4110] - HHVM 3.18+ enforces \HH\FormatString, ignoring pending upstream documentation. */
+    /* HH_IGNORE_ERROR[4027] - HHVM 3.18+ enforces \HH\FormatString, ignoring pending upstream documentation. */


what you want is:

invariant( $result !== null, 'Index %s not found in container', $idx, );

Given that Tk isn't constrained, the code as-is could error at runtime (including on < 3.18) if $idx contains %s

fredemmott · 2017-07-24T16:27:28Z

src/Utils.php

@@ -19,6 +21,8 @@ function must_have_string<Tk as string, Tv>(
  Tk $idx,
 ): string {
  $result = must_have_idx($arr, $idx);
+  /* HH_IGNORE_ERROR[4110] - HHVM 3.18+ enforces \HH\FormatString, ignoring pending upstream documentation. */
+  /* HH_IGNORE_ERROR[4027] - HHVM 3.18+ enforces \HH\FormatString, ignoring pending upstream documentation. */
  invariant(is_string($result), "Expected $idx to be a string");


ditto, and please don't use interpolation

fredemmott · 2017-07-24T16:28:16Z

Comments addressed by your update :)

Merge of `dev` into `master` Commits: * Registration enforcing strong passwords (#442) (ac64f55) * Custom branding for icon and text (#448) (081062c) * Merge of /master into /dev - Baseline for Development (#509) (25c1748) * Updated Language Translations (#511) (b9f031e) * Auto Announcements and Activity Log Expansion (#513) (323ba05) * Level Import Fix (#514) (dc7c87c) * Announcements Controls Rename (#515) (c5da9f7) * Set Default Scoring Cache Values (#516) (ec996a5) * Unique Logos Per Team # (#517) (6d4f919) * Custom Branding Update (#518) (ea78f6a) * Backup and Restore settings.ini on Tests (#519) (eb4a5b5) * Maintain Team Protection on Database Reset (#520) (5d91ae9) * Fixed Login Form JS Bug (Fixes: #521) (#523) (2b1474b) * Level Deletion Confirmation and Bug Fixes (#512) (4a7b5b5) * Provision Streamlined, Quick Setup Added, and Multiple Containers Support (#535) (b487fc1) * Merge branch 'dev' into WraySec/fbctf/merge@7f8c281

George Evans and others added 2 commits July 19, 2017 12:13

Separate docker containers per service

fdcd596

facebook-github-bot added the CLA Signed label Jul 19, 2017

Added Memcached Service Restart to HHVM container service script

c168d1a

justinwray mentioned this pull request Jul 19, 2017

Separate docker containers per service #530

Closed

justinwray added 10 commits July 19, 2017 14:17

Added logging of PHP/HHVM version.

21bfeb3

Fixed typo in previous commit.

0f039e9

Travis PHP Alternatives debugging

8248419

Install Composer wih HHVM.

06666e4

Use HHVM for Composer Install

d64b32f

Downgrading the Travis trusty image

42bf4d0

Updated run_tests.sh to have write permissions to settings.ini

dda50bb

Set run_tests.sh to use localhost for DB and MC.

90d0190

Fixed typo in run_tests.sh

af44f47

HHVM 3.18+ enforces \HH\FormatString, ignoring pending upstream docum…

c02fb17

…entation

justinwray mentioned this pull request Jul 23, 2017

HH\FormatString Typing Errors in sprintf() and invariant() hhvm/user-documentation#448

Closed

justinwray added 2 commits July 24, 2017 11:32

fredemmott reviewed Jul 24, 2017

View reviewed changes

justinwray merged commit b487fc1 into facebookarchive:dev Aug 4, 2017

This was referenced Aug 4, 2017

Docker file updated #534 #538

Closed

Merge /dev to /master #542

Merged

stevcoll mentioned this pull request Aug 11, 2017

Upload failure #280

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Provision Streamlined, Quick Setup Added, and Multiple Containers Support #535

Provision Streamlined, Quick Setup Added, and Multiple Containers Support #535

justinwray commented Jul 19, 2017

fredemmott Jul 24, 2017

fredemmott Jul 24, 2017

fredemmott commented Jul 24, 2017

Provision Streamlined, Quick Setup Added, and Multiple Containers Support #535

Provision Streamlined, Quick Setup Added, and Multiple Containers Support #535

Conversation

justinwray commented Jul 19, 2017

fredemmott Jul 24, 2017

Choose a reason for hiding this comment

fredemmott Jul 24, 2017

Choose a reason for hiding this comment

fredemmott commented Jul 24, 2017