SAK-41868 Add Docker foundation (sakaiproject#6967)

* SAK-41868 Add Docker examples

* SAK-41868 Implement codacy suggestions

* SAK-41868 Implement codacy suggestions for automated

* SAK-41868 Use secrets module for Sakai + Mysql example

* SAK-41868 Fix a port mapping

* SAK-41868 Add development examples, and restructure exmaples

* SAK-41868 Remove dnagling Vagrantfile

* SAK-41868 Fix css for codacy

* SAK-41868 Use generic local image in example

* SAK-41868 Tweaks from codacy

* SAK-41868 Tweaks from codacy

Codacy> Expressions don't expand in single quotes, use double quotes for that.
Me> That's because I do not want them to do that.. fine,, I'll escape it... better?

* SAK-41868 Tweaks from codacy, when all else fails.. escape all the things..

* Update README.md

* Update README.md

* SAK-41868 Remove static jars

* SAK-41868 Add an image builder example to dev context, fix a port mapping

* SAK-41868 Codacy tweaks

* SAK-41868 Clean up deployment example Sakai configuration

* SAK-41868 Use example Docker Secrets configuration for minimal stack

docker/README.md

@@ -0,0 +1,38 @@
+# Sakai Docker Deployment Examples
+
+A collection of tools and examples to demonstrate developing, building and deploying Sakai using Docker Swarm.
+
+### Features
+Features of the files you will find here:
+
+* Development stack examples
+* Docker image building examples
+* Docker Swarm deployment examples
+
+# Quick install Docker (Linux, new server/workstation)
+Docker provides an installation script for most Linux distributions, With Ubuntu and RHEL/CentOS being the most used. 
+This script is located at https://get.docker.com/ and has the following instructions at the top of the file:
+
+    This script is meant for quick & easy install via:
+    $ curl -fsSL https://get.docker.com -o get-docker.sh
+    $ sh get-docker.sh
+
+This will install and prepare everything needed to build images and run them in standalone mode.
+Additionally, docker has a built-in container orchestration platform called "Docker Swarm". Swarm needs to be enabled to use any of the deployment example scripts in `deploy/`, this can be enabled by simply running:
+
+     $ docker swarm init 
+
+# Development
+In the `dev/` folder you will find an example staged Swarm stack that included everything needed to build and test sakai in development.
+
+GOTO [Development README](dev/)
+
+# Building
+In the `build/` folder you will find examples for building a docker image for Sakai. Before deployment, you need to build the docker image you will deploy, examples are provided for building from source or a binary release.
+
+GOTO [Building README](build/)
+
+# Deployment
+Deploying Sakai in Docker Swarm is easy, even with a full stack of supporting services.
+
+GOTO [Deployment README](deploy/)

docker/build/Dockerfile.binary

@@ -0,0 +1,29 @@
+FROM tomcat:9.0.20-jre8
+
+ARG release=19.1
+
+COPY lib/server.xml /usr/local/tomcat/conf/server.xml
+COPY lib/context.xml /usr/local/tomcat/conf/context.xml
+COPY lib/entrypoint.sh /entrypoint.sh
+
+RUN mkdir /sakaibin && cd /sakaibin && wget http://source.sakaiproject.org/release/${release}/artifacts/sakai-bin-${release}.tar.gz && tar -zxvf sakai-bin-${release}.tar.gz && rm -fr /usr/local/tomcat/components && rm -fr /usr/local/tomcat/sakai-lib && rm -fr /usr/local/tomcat/webapps && cp -R /sakaibin/components /usr/local/tomcat/components/ && cp -R /sakaibin/lib /usr/local/tomcat/sakai-lib/ && cp -R /sakaibin/webapps /usr/local/tomcat/webapps/ && cd / && rm -fr /sakaibin && mkdir -p /usr/local/sakai/properties && sed -i '/^common.loader\=/ s/$/,"\$\{catalina.base\}\/sakai-lib\/*.jar"/' /usr/local/tomcat/conf/catalina.properties && curl -L -o /usr/local/tomcat/lib/mysql-connector-java-5.1.47.jar https://repo1.maven.org/maven2/mysql/mysql-connector-java/5.1.47/mysql-connector-java-5.1.47.jar && mkdir -p /usr/local/tomcat/sakai && chmod +x /entrypoint.sh
+
+ENV CATALINA_OPTS_MEMORY -Xms2000m -Xmx2000m
+ENV CATALINA_OPTS \
+-server \
+-verbose:gc -XX:+PrintGCDetails -XX:+PrintGCTimeStamps -XX:+UseConcMarkSweepGC -XX:+UseParNewGC \
+-XX:+CMSParallelRemarkEnabled -XX:+UseCompressedOops -XX:+UseCMSInitiatingOccupancyOnly -XX:CMSInitiatingOccupancyFraction=80 -XX:TargetSurvivorRatio=90 \
+-Djava.awt.headless=true \
+-Dsun.net.inetaddr.ttl=0 \
+-Dsakai.component.shutdownonerror=true \
+-Duser.language=en -Duser.country=US \
+-Dsakai.home=/usr/local/sakai/properties -Dsakai.security=/usr/local/tomcat/sakai \
+-Duser.timezone=US/Eastern \
+-Dsun.net.client.defaultConnectTimeout=300000 \
+-Dsun.net.client.defaultReadTimeout=1800000 \
+-Dorg.apache.jasper.compiler.Parser.STRICT_QUOTE_ESCAPING=false \
+-Dsun.lang.ClassLoader.allowArraySyntax=true \
+-Dhttp.agent=Sakai \
+-Djava.util.Arrays.useLegacyMergeSort=true 
+
+ENTRYPOINT ["/entrypoint.sh"]

docker/build/Dockerfile.source

@@ -0,0 +1,49 @@
+FROM maven:3.6.1-jdk-8 as build
+
+ARG release=master
+
+COPY lib/settings.xml /usr/share/maven/conf/settings.xml
+RUN mkdir /deploy
+WORKDIR /deploy
+RUN git clone https://github.com/sakaiproject/sakai.git
+WORKDIR /deploy/sakai
+RUN git checkout ${release} && mvn clean install sakai:deploy -Dmaven.test.skip=true
+
+
+FROM tomcat:9.0.20-jre8
+
+COPY lib/server.xml /usr/local/tomcat/conf/server.xml
+COPY lib/context.xml /usr/local/tomcat/conf/context.xml
+COPY --from=build /deploy/components /usr/local/tomcat/components/
+COPY --from=build /deploy/lib /usr/local/tomcat/sakai-lib/
+COPY --from=build /deploy/webapps /usr/local/tomcat/webapps/
+
+RUN mkdir -p /usr/local/sakai/properties
+
+ENV CATALINA_OPTS_MEMORY -Xms2000m -Xmx2000m
+ENV CATALINA_OPTS \
+-server \
+-verbose:gc -XX:+PrintGCDetails -XX:+PrintGCTimeStamps -XX:+UseConcMarkSweepGC -XX:+UseParNewGC \
+-XX:+CMSParallelRemarkEnabled -XX:+UseCompressedOops -XX:+UseCMSInitiatingOccupancyOnly -XX:CMSInitiatingOccupancyFraction=80 -XX:TargetSurvivorRatio=90 \
+-Djava.awt.headless=true \
+-Dsun.net.inetaddr.ttl=0 \
+-Dsakai.component.shutdownonerror=true \
+-Duser.language=en -Duser.country=US \
+-Dsakai.home=/usr/local/sakai/properties -Dsakai.security=/usr/local/tomcat/sakai \
+-Duser.timezone=US/Eastern \
+-Dsun.net.client.defaultConnectTimeout=300000 \
+-Dsun.net.client.defaultReadTimeout=1800000 \
+-Dorg.apache.jasper.compiler.Parser.STRICT_QUOTE_ESCAPING=false \
+-Dsun.lang.ClassLoader.allowArraySyntax=true \
+-Dhttp.agent=Sakai \
+-Djava.util.Arrays.useLegacyMergeSort=true 
+
+RUN sed -i '/^common.loader\=/ s/$/,"\$\{catalina.base\}\/sakai-lib\/*.jar"/' /usr/local/tomcat/conf/catalina.properties
+
+RUN curl -L -o /usr/local/tomcat/lib/mysql-connector-java-5.1.47.jar https://repo1.maven.org/maven2/mysql/mysql-connector-java/5.1.47/mysql-connector-java-5.1.47.jar
+
+RUN mkdir -p /usr/local/tomcat/sakai
+COPY lib/entrypoint.sh /entrypoint.sh
+RUN chmod +x /entrypoint.sh
+ENTRYPOINT ["/entrypoint.sh"]
+

docker/build/README.md

@@ -0,0 +1,54 @@
+# Sakai Docker Deployment Examples
+
+A collection of tools and examples to demonstrate building and deploying Sakai using Docker and/or Docker Swarm.
+
+### Features
+Features of the files you will find here:
+
+* Multiple build types: 
+  * From source 
+  * From binary release.
+  * Automated via DockerHub
+
+#### Table of Contents
+- [Quick install Docker (Linux, new server/workstation)](#quick-install-docker--linux--new-server-workstation-)
+- [Building the Image](#building-the-image)
+  * [From a binary release](#from-a-binary-release)
+  * [From source (github tag/branch)](#from-source--github-tag-branch-)
+
+
+# Quick install Docker (Linux, new server/workstation)
+Docker provides an installation script for most Linux distributions, With Ubuntu and RHEL/CentOS being the most used. 
+This script is located at https://get.docker.com/ and has the following instructions at the top of the file:
+
+    This script is meant for quick & easy install via:
+    $ curl -fsSL https://get.docker.com -o get-docker.sh
+    $ sh get-docker.sh
+
+# Building the Image
+There are two variants of this docker build, depending on your preference to build from source code or from a binary release. As well there is an example automated build to be used with docker hub.
+
+## From a binary release
+To build from a binary release use these steps:
+ 1. From this folder
+ 1. Execute `docker build --build-arg release=19.1 -t sakai -f ./Dockerfile.binary .` substituting "19.1" for the release you wish to build
+ 1. Upon completion you can execute `docker image ls sakai` to verify it's creation
+    *     $ docker image ls sakai
+          REPOSITORY          TAG                 IMAGE ID            CREATED             SIZE
+          sakai               latest              369fde564591        5 seconds ago       2.55GB
+
+## From source (github tag/branch)
+The source build uses a multi-stage build, building an intermediate image with JDK and Maven in which to build Sakai, then building a Tomcat image using only the binary artifacts from the build container. This creates a smaller Sakai image that does not include Maven and all the build time libraries, source, etc.
+
+To build from source use these steps:
+ 1. From this folder
+ 1. Execute `docker build --build-arg release=master -t sakai -f ./Dockerfile.source .` substituting "master" for the branch/tag you wish to build
+ 1. Upon completion you can execute `docker image ls sakai` to verify it's creation
+    *     $ docker image ls sakai
+          REPOSITORY          TAG                 IMAGE ID            CREATED             SIZE
+          sakai               latest              78b32fe87fda        8 seconds ago       2.55GB
+
+## Automated from DockerHub
+The Dockerfiles here are configured to work with the build hook in the hooks folder.
+
+See https://docs.docker.com/docker-hub/builds/ for detailed information on setting up automated builds

docker/build/hooks/build

@@ -0,0 +1,3 @@
+#!/bin/bash
+RELEASE=$(echo $DOCKER_TAG | sed s/release-//g)
+docker build --build-arg release=$RELEASE -f $DOCKERFILE_PATH -t $IMAGE_NAME .

docker/build/lib/context.xml

@@ -0,0 +1,7 @@
+<Context>
+    <JarScanner>
+        <!-- This is to speedup startup so that tomcat doesn't scan as much -->
+        <JarScanFilter defaultPluggabilityScan="false" defaultTldScan="false" tldScan="jsf2-widgets-*.jar,javax.faces-*.jar,jsf-impl-*.jar,jsf-widgets-*.jar,myfaces-impl-*.jar,pluto-taglib-*.jar,sakai-sections-app-util-*.jar,spring-webmvc-*.jar,standard-*.jar,tomahawk*.jar,tomahawk-*.jar"/>
+    </JarScanner>
+</Context>
+

docker/build/lib/entrypoint.sh

@@ -0,0 +1,6 @@
+#!/bin/bash
+# Append any secret properties from /run/secrets/security.properties
+cat /run/secrets/security.properties >> /usr/local/tomcat/sakai/security.properties
+
+# Start tomcat
+catalina.sh run

docker/build/lib/server.xml

@@ -0,0 +1,16 @@
+<?xml version='1.0' encoding='utf-8'?>
+<Server port="8005" shutdown="SHUTDOWN" address="0.0.0.0">
+  <Listener className="org.apache.catalina.core.AprLifecycleListener" SSLEngine="on" />
+  <Listener className="org.apache.catalina.core.JreMemoryLeakPreventionListener" />
+  <Listener className="org.apache.catalina.mbeans.GlobalResourcesLifecycleListener" />
+  <Listener className="org.apache.catalina.core.ThreadLocalLeakPreventionListener" />
+  <Service name="Catalina">
+    <Connector port="8080" protocol="HTTP/1.1" connectionTimeout="20000" redirectPort="8443" URIEncoding="UTF-8" address="0.0.0.0"/>
+    <Connector port="8009" protocol="AJP/1.3" redirectPort="8443" URIEncoding="UTF-8" tomcatAuthentication="false" maxThreads="500" connectionTimeout="60000" enableLookups="false" address="0.0.0.0"/>
+
+    <Engine name="Catalina" defaultHost="localhost">
+      <Host name="localhost" startStopThreads="4" appBase="webapps" unpackWARs="true" autoDeploy="true">
+      </Host>
+    </Engine>
+  </Service>
+</Server>

docker/build/lib/settings.xml

@@ -0,0 +1,21 @@
+<settings xmlns="http://maven.apache.org/POM/4.0.0"
+   xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
+   xsi:schemaLocation="http://maven.apache.org/POM/4.0.0
+                      http://maven.apache.org/xsd/settings-1.0.0.xsd">
+  <profiles>
+    <profile>
+      <id>tomcat</id>
+      <activation>
+        <activeByDefault>true</activeByDefault>
+      </activation>
+      <properties>
+        <appserver.id>tomcat</appserver.id>
+	<appserver.home>/deploy</appserver.home>
+	<maven.tomcat.home>/deploy</maven.tomcat.home>
+	<sakai.appserver.home>/deploy</sakai.appserver.home>
+        <surefire.reportFormat>plain</surefire.reportFormat>
+        <surefire.useFile>false</surefire.useFile>
+      </properties>
+    </profile>
+  </profiles>
+</settings>

docker/deploy/README.md

@@ -0,0 +1,112 @@
+# Sakai Docker Deployment Examples
+
+A collection of tools and examples to demonstrate building and deploying Sakai using Docker and/or Docker Swarm.
+
+### Features
+Features of the files you will find here:
+
+* Example Swarm stacks
+  * Sakai + Mysql 
+  * Sakai + Mysql + Elasticsearch + Cerebro + Mailcatcher
+
+#### Table of Contents
+- [Quick install Docker (Linux, new server/workstation)](#quick-install-docker--linux--new-server-workstation-)
+- [Running Sakai](#running-sakai)
+  * [Swarm enabled Docker](#swarm-enabled-docker)
+    + [Sakai+Mysql](#sakai-mysql)
+    + [Sakai+Mysql+Elasticsearch+Mailcatcher](#sakai-mysql-elasticsearch-mailcatcher)
+    + [Monitoring Examples](#monitoring-examples)
+    + [Management Examples](#management-examples)
+  * [Standalone Docker](#standalone-docker)
+    + [Monitoring Examples](#monitoring-examples-1)
+    + [Management Examples](#management-examples-1)
+
+
+# Quick install Docker (Linux, new server/workstation)
+Docker provides an installation script for most Linux distributions, With Ubuntu and RHEL/CentOS being the most used. 
+This script is located at https://get.docker.com/ and has the following instructions at the top of the file:
+
+    This script is meant for quick & easy install via:
+    $ curl -fsSL https://get.docker.com -o get-docker.sh
+    $ sh get-docker.sh
+
+This will install and prepare everything needed to build images and run them in standalone mode.
+Additionally, docker has a built-in container orchestration platform called "Docker Swarm". Swarm needs to be enabled to use any of these deployment example scripts, this can be enabled by simply running:
+
+     $ docker swarm init 
+
+# Running Sakai
+You will need to provide at a minimum a `sakai.properties` configuration containing a minimal database configuration. In the examples here, a MySQL configuration is provided.
+
+## Swarm enabled Docker
+These are basic examples, that do not include mounting storage volumes to persist data.
+
+### Sakai+Mysql
+To start the example Sakai and Mysql stack follow these steps:
+ 1. From this folder.
+ 1. Execute `docker stack deploy -c sakai_docker.yml sakai`
+ 	* This creates a stack named `sakai` using the compose file `sakai_docker.yml`
+ 	* Services in the stack are named \<StackName\>_\<ServiceName\> (e.g. sakai_mysql and sakai_sakai)
+ 	* The configuration file `conf/sakai.properties` is mounted inside the container in /usr/local/sakai/properties
+ 	* Sakai will be located at `http://<dockerhost>:8080/portal`
+
+### Sakai+Mysql+Elasticsearch+Cerebro+Mailcatcher
+To start the example Sakai, Mysql, Elasticsearch, Cerebro, and Mailcatcher stack follow these steps:
+ 1. From this folder.
+ 1. Execute `docker stack deploy -c sakai_es_docker.yml sakai`
+ 	* This creates a stack named `sakai` using the compose file `sakai_es_docker.yml`
+ 	* Services in the stack are named \<StackName\>_\<ServiceName\> (e.g. sakai_mysql and sakai_sakai)
+ 	* The configuration file `conf/sakai.properties` is mounted inside the container in /usr/local/sakai/properties
+    * The secrets file `secrets/security.properties` is deployed using Docker Secrets and placed in /usr/local/tomcat/sakai/
+    * Sakai will be located at `http://<dockerhost>:8080/portal`
+    * Mailcatcher will be located at `http://<dockerhost>:8081/`
+    * Cerebro (Elasticsearch Management) will be located at `http://<dockerhost>:8082/`
+    * Use ES Node address: `http://elasticsearch:9200/`
+
+
+### Monitoring Examples
+Now that the stack has started you can monitor with the Swarm commands:
+ 1. List running stacks with `docker stack ls`
+ 1. List services running in the sakai stack with `docker stack services sakai`
+ 1. List containers running in the sakai service with `docker service ps sakai_sakai`
+ 1. View (tail) the logs from the Sakai container with `docker service logs -f sakai_sakai`
+
+### Management Examples
+You can manage using Swarm commands:
+ 1. Restart the sakai service with `docker service update --force sakai_sakai`
+ 1. Stop and remove the stack with `docker stack rm sakai`
+
+## Standalone Docker 
+In this example you will start a standalone MySQL container and a linked standalone sakai container.
+ 1. From this folder.
+ 1. Start a MySQL 5.5 container with:
+    *     docker run -d \
+          --name mysql \
+          -e MYSQL_ROOT_PASSWORD=examplerootpassword \
+          -e MYSQL_DATABASE=sakai -e MYSQL_USER=sakai \
+          -e MYSQL_PASSWORD=examplepassword \
+          mysql:5.5 \
+          --character-set-server=utf8 \
+          --collation-server=utf8_general_ci
+ 1. Monitor MySQL startup with `docker logs -f mysql` and hit Ctrl-C once MySQL has finished starting.
+ 1. Start a Sakai Container with the MySQL container linked to it:
+    *     docker run -d \
+          --name sakai \
+          -p 8080:8080 \
+          -v $(pwd)/config/sakai.properties:/usr/local/sakai/properties/sakai.properties \
+          --link mysql:mysql \         
+          sakai
+ 1. Monitor Sakai startup with `docker logs -f sakai` and hit Ctrl-C once Sakai has finished starting.
+ 1. Sakai should be available on port 8080 of the Dockerhost.
+
+### Monitoring Examples
+You may need to monitor the containers
+ 1. View logs of the sakai container with `docker logs -f sakai`
+ 1. Inspect the sakai container configuration with `docker inspect sakai`
+
+### Management Examples
+You may need to manage the containers
+ 1. Stop the running sakai container with `docker kill sakai`
+ 2. Remove the sakai container with `docker rm sakai`
+
+

docker/deploy/config/elasticsearch.yml

@@ -0,0 +1,14 @@
+# cluster name
+cluster.name: sakai_cluster
+
+# Prevent memory swapping
+bootstrap.memory_lock: true
+
+# Connect to sakai nodes using DNS discovery
+discovery.zen.ping.unicast.hosts: tasks.sakai:9300
+
+# Require 2 master Names
+discovery.zen.minimum_master_nodes: 2 
+
+# This is a master node
+node.master: true

docker/deploy/config/es_entry.sh

@@ -0,0 +1,18 @@
+#!/bin/bash
+
+# Find the correct interface for elasticsearch using the subnet defined in the compose file
+ES_INT=$(ip addr | grep 10.99.99 | cut -d " " -f 11)
+
+# Log the interface found
+echo "Using interface $ES_INT for elasticsearch"
+
+# Create the updated Sakai configuration
+sed s/#interface#/\_$ES_INT:ipv4\_/g /usr/local/sakai/es.properties > /usr/local/sakai/properties/sakai.properties
+
+rm -fr /usr/local/tomcat/webapps/ROOT
+mkdir /usr/local/tomcat/webapps/ROOT
+cd /usr/local/tomcat/webapps/ROOT
+tar zxvf /demopage.tgz
+
+# Execute the real entrypoint script
+/entrypoint.sh