[WIP] s2i and general extendability support (#45)

* Implement source-to-image support

This is very similar to the MongoDB PoC: sclorg/mongodb-container#239

It adds extending support using [source-to-image](https://github.com/openshift/source-to-image).

For example to build customized MariaDB database image `my-mariadb-centos7` with configuration in `~/image-configuration/` run:

```
$ s2i build ~/image-configuration/ centos/mariadb-101-centos7 my-mariadb-centos7
```

The directory passed to `s2i build` can contain these directories:
- `mysql-cfg/`
  - when starting the container, files from this directory will be used as a configuration for the `mysqld` daemon
    - `envsubst` command is run on this file to still allow customization of the image using environmental variables

- `mysql-pre-init/`
  - shell scripts (`*.sh`) available in this directory are sourced before `mysqld` daemon is started

- `mysql-init/`
  - shell scripts (`*.sh`) available in this directory are sourced when `mysqld` daemon is started locally
    - in this phase, use `${mysql_flags}` to connect to the locally running daemon, for example `mysql $mysql_flags < dump.sql`

Variables that can be used in the scripts provided to s2i:

- `$mysql_flags` -- arguments for the `mysql` tool that will connect to the locally running `mysqld` during initialization
- `$MYSQL_RUNNING_AS_MASTER` -- variable defined when the container is run with `run-mysqld-master` command
- `$MYSQL_RUNNING_AS_SLAVE` -- variable defined when the container is run with `run-mysqld-slave` command
- `$MYSQL_DATADIR_FIRST_INIT` -- variable defined when the container was initialized from the empty data dir

During `s2i build` all provided files are copied into `/opt/app-root/src` directory into the resulting image. If some configuration files are present in the destination directory, files with the same name are overwritten. Also only one file with the same name can be used for customization and user provided files are preferred over default files in `/usr/share/container-scripts/mysql/`- so it is possible to overwrite them.

Same configuration directory structure can be used to customize the image every time the image is started using `docker run`. The directory has to be mounted into `/opt/app-root/src/` in the image (`-v ./image-configuration/:/opt/app-root/src/`). This overwrites customization built into the image.

* Implement source-to-image support

This is very similar to the MongoDB PoC: sclorg/mongodb-container#239

It adds extending support using [source-to-image](https://github.com/openshift/source-to-image).

For example to build customized MariaDB database image `my-mariadb-centos7` with configuration in `~/image-configuration/` run:

```
$ s2i build ~/image-configuration/ centos/mariadb-101-centos7 my-mariadb-centos7
```

The directory passed to `s2i build` can contain these directories:
- `mysql-cfg/`
  - when starting the container, files from this directory will be used as a configuration for the `mysqld` daemon
    - `envsubst` command is run on this file to still allow customization of the image using environmental variables

- `mysql-pre-init/`
  - shell scripts (`*.sh`) available in this directory are sourced before `mysqld` daemon is started

- `mysql-init/`
  - shell scripts (`*.sh`) available in this directory are sourced when `mysqld` daemon is started locally
    - in this phase, use `${mysql_flags}` to connect to the locally running daemon, for example `mysql $mysql_flags < dump.sql`

Variables that can be used in the scripts provided to s2i:

- `$mysql_flags` -- arguments for the `mysql` tool that will connect to the locally running `mysqld` during initialization
- `$MYSQL_RUNNING_AS_MASTER` -- variable defined when the container is run with `run-mysqld-master` command
- `$MYSQL_RUNNING_AS_SLAVE` -- variable defined when the container is run with `run-mysqld-slave` command
- `$MYSQL_DATADIR_FIRST_INIT` -- variable defined when the container was initialized from the empty data dir

During `s2i build` all provided files are copied into `/opt/app-root/src` directory into the resulting image. If some configuration files are present in the destination directory, files with the same name are overwritten. Also only one file with the same name can be used for customization and user provided files are preferred over default files in `/usr/share/container-scripts/mysql/`- so it is possible to overwrite them.

Same configuration directory structure can be used to customize the image every time the image is started using `docker run`. The directory has to be mounted into `/opt/app-root/src/` in the image (`-v ./image-configuration/:/opt/app-root/src/`). This overwrites customization built into the image.

* Report tests result properly

* Add OpenShift tests

File test-lib-openshift.sh is a candidate for container-common-scripts.

* Fix test for s2i mount

* Update common submodule and use test-lib-openshift.sh from that submodule

* Add SSL connection testing

* Add self-signed-ssl example

10.0/Dockerfile

@@ -1,4 +1,4 @@
-FROM centos:centos7
+FROM centos/s2i-core-centos7
 
 # MariaDB image for OpenShift.
 #
@@ -11,6 +11,7 @@ FROM centos:centos7
 #  * $MYSQL_ROOT_PASSWORD (Optional) - Password for the 'root' MySQL account
 
 ENV MYSQL_VERSION=10.0 \
+    APP_DATA=/opt/app-root/src \
     HOME=/var/lib/mysql
 
 ENV SUMMARY="MariaDB 10.0 SQL database server" \
@@ -36,7 +37,7 @@ EXPOSE 3306
 # safe in the future. This should *never* change, the last test is there
 # to make sure of that.
 RUN yum install -y centos-release-scl-rh && \
-    INSTALL_PKGS="rsync tar gettext hostname bind-utils rh-mariadb100" && \
+    INSTALL_PKGS="rsync tar gettext hostname bind-utils groff-base shadow-utils rh-mariadb100" && \
     yum install -y --setopt=tsflags=nodocs $INSTALL_PKGS && \
     rpm -V $INSTALL_PKGS && \
     yum clean all && \
@@ -56,6 +57,7 @@ ENV BASH_ENV=${CONTAINER_SCRIPTS_PATH}/scl_enable \
     PROMPT_COMMAND=". ${CONTAINER_SCRIPTS_PATH}/scl_enable"
 
 COPY 10.0/root-common /
+COPY 10.0/s2i-common/bin/ $STI_SCRIPTS_PATH
 COPY 10.0/root /
 
 # this is needed due to issues with squash

10.0/Dockerfile.rhel7

@@ -1,4 +1,4 @@
-FROM rhel7
+FROM rhscl/s2i-core-rhel7
 
 # MariaDB image for OpenShift.
 #
@@ -11,6 +11,7 @@ FROM rhel7
 #  * $MYSQL_ROOT_PASSWORD (Optional) - Password for the 'root' MySQL account
 
 ENV MYSQL_VERSION=10.0 \
+    APP_DATA=/opt/app-root/src \
     HOME=/var/lib/mysql
 
 ENV SUMMARY="MariaDB 10.0 SQL database server" \
@@ -43,7 +44,7 @@ RUN yum repolist > /dev/null && \
     yum-config-manager --enable rhel-7-server-rpms && \
     yum-config-manager --enable rhel-7-server-optional-rpms && \
     yum-config-manager --enable rhel-server-rhscl-7-rpms && \
-    INSTALL_PKGS="rsync tar gettext hostname bind-utils rh-mariadb100" && \
+    INSTALL_PKGS="rsync tar gettext hostname bind-utils groff-base shadow-utils rh-mariadb100" && \
     yum install -y --setopt=tsflags=nodocs $INSTALL_PKGS && \
     rpm -V $INSTALL_PKGS && \
     yum clean all && \
@@ -63,6 +64,7 @@ ENV BASH_ENV=${CONTAINER_SCRIPTS_PATH}/scl_enable \
     PROMPT_COMMAND=". ${CONTAINER_SCRIPTS_PATH}/scl_enable"
 
 COPY 10.0/root-common /
+COPY 10.0/s2i-common/bin/ $STI_SCRIPTS_PATH
 COPY 10.0/root /
 
 # this is needed due to issues with squash

10.0/root/usr/share/container-scripts/mysql/README.md

@@ -140,6 +140,81 @@ location is `/etc/my.cnf` but you can change it to `/etc/mysql/my.cnf` by settin
  `MYSQL_DEFAULTS_FILE=/etc/mysql/my.cnf`
 
 
+Extending image
+---------------
+This image can be extended using [source-to-image](https://github.com/openshift/source-to-image).
+
+For example, to build a customized MariaDB database image `my-mariadb-rhel7`
+with a configuration in `~/image-configuration/` run:
+
+```
+$ s2i build ~/image-configuration/ rhscl/mariadb-100-rhel7 my-mariadb-rhel7
+```
+
+The directory passed to `s2i build` can contain these directories:
+
+`mysql-cfg/`
+    When starting the container, files from this directory will be used as
+    a configuration for the `mysqld` daemon.
+    `envsubst` command is run on this file to still allow customization of
+    the image using environmental variables
+
+`mysql-pre-init/`
+    Shell scripts (`*.sh`) available in this directory are sourced before
+    `mysqld` daemon is started.
+
+`mysql-init/`
+    Shell scripts (`*.sh`) available in this directory are sourced when
+    `mysqld` daemon is started locally. In this phase, use `${mysql_flags}`
+    to connect to the locally running daemon, for example `mysql $mysql_flags < dump.sql`
+
+Variables that can be used in the scripts provided to s2i:
+
+`$mysql_flags`
+    arguments for the `mysql` tool that will connect to the locally running `mysqld` during initialization
+
+`$MYSQL_RUNNING_AS_MASTER`
+    variable defined when the container is run with `run-mysqld-master` command
+
+`$MYSQL_RUNNING_AS_SLAVE`
+    variable defined when the container is run with `run-mysqld-slave` command
+
+`$MYSQL_DATADIR_FIRST_INIT`
+    variable defined when the container was initialized from the empty data dir
+
+During `s2i build` all provided files are copied into `/opt/app-root/src`
+directory into the resulting image. If some configuration files are present
+in the destination directory, files with the same name are overwritten.
+Also only one file with the same name can be used for customization and user
+provided files are preferred over default files in
+`/usr/share/container-scripts/mysql/`- so it is possible to overwrite them.
+
+Same configuration directory structure can be used to customize the image
+every time the image is started using `docker run`. The directory has to be
+mounted into `/opt/app-root/src/` in the image
+(`-v ./image-configuration/:/opt/app-root/src/`).
+This overwrites customization built into the image.
+
+
+Securing the connection with SSL
+--------------------------------
+In order to secure the connection with SSL, use the extending feature described
+above. In particular, put the SSL certificates into a separate directory:
+
+    sslapp/mysql-certs/server-cert-selfsigned.pem
+    sslapp/mysql-certs/server-key.pem
+
+And then put a separate configuration file into mysql-cfg:
+
+    $> cat sslapp/mysql-cfg/ssl.cnf
+    [mysqld]
+    ssl-key=${APP_DATA}/mysql-certs/server-key.pem
+    ssl-cert=${APP_DATA}/mysql-certs/server-cert-selfsigned.pem
+
+Such a directory `sslapp` can then be mounted into the container with -v,
+or a new container image can be built using s2i.
+
+
 Changing the replication binlog_format
 --------------------------------------
 Some applications may wish to use `row` binlog_formats (for example, those built

10.0/s2i-common

@@ -0,0 +1 @@
+../s2i-common

10.1/Dockerfile

@@ -1,4 +1,4 @@
-FROM centos:centos7
+FROM centos/s2i-core-centos7
 
 # MariaDB image for OpenShift.
 #
@@ -11,6 +11,7 @@ FROM centos:centos7
 #  * $MYSQL_ROOT_PASSWORD (Optional) - Password for the 'root' MySQL account
 
 ENV MYSQL_VERSION=10.1 \
+    APP_DATA=/opt/app-root/src \
     HOME=/var/lib/mysql
 
 ENV SUMMARY="MariaDB 10.1 SQL database server" \
@@ -36,7 +37,7 @@ EXPOSE 3306
 # safe in the future. This should *never* change, the last test is there
 # to make sure of that.
 RUN yum install -y centos-release-scl-rh && \
-    INSTALL_PKGS="rsync tar gettext hostname bind-utils rh-mariadb101" && \
+    INSTALL_PKGS="rsync tar gettext hostname bind-utils groff-base shadow-utils rh-mariadb101" && \
     yum install -y --setopt=tsflags=nodocs $INSTALL_PKGS && \
     rpm -V $INSTALL_PKGS && \
     yum clean all && \
@@ -56,6 +57,7 @@ ENV BASH_ENV=${CONTAINER_SCRIPTS_PATH}/scl_enable \
     PROMPT_COMMAND=". ${CONTAINER_SCRIPTS_PATH}/scl_enable"
 
 COPY 10.1/root-common /
+COPY 10.1/s2i-common/bin/ $STI_SCRIPTS_PATH
 COPY 10.1/root /
 
 # this is needed due to issues with squash

10.1/Dockerfile.fedora

@@ -1,4 +1,4 @@
-FROM fedora:25
+FROM registry.fedoraproject.org/f26/s2i-core:latest
 
 # MariaDB image for OpenShift.
 #
@@ -11,6 +11,7 @@ FROM fedora:25
 #  * $MYSQL_ROOT_PASSWORD (Optional) - Password for the 'root' MySQL account
 
 ENV MYSQL_VERSION=10.1 \
+    APP_DATA=/opt/app-root/src \
     HOME=/var/lib/mysql \
     NAME=mariadb \
     VERSION=10.1 \
@@ -34,7 +35,7 @@ EXPOSE 3306
 # This image must forever use UID 27 for mysql user so our volumes are
 # safe in the future. This should *never* change, the last test is there
 # to make sure of that.
-RUN INSTALL_PKGS="rsync tar gettext hostname bind-utils mariadb-server policycoreutils" && \
+RUN INSTALL_PKGS="rsync tar gettext hostname bind-utils groff-base shadow-utils mariadb-server policycoreutils" && \
     dnf install -y --setopt=tsflags=nodocs $INSTALL_PKGS && \
     rpm -V $INSTALL_PKGS && \
     dnf clean all && \
@@ -50,6 +51,7 @@ ENV CONTAINER_SCRIPTS_PATH=/usr/share/container-scripts/mysql \
     MYSQL_PREFIX=/usr
 
 COPY 10.1/root-common /
+COPY 10.1/s2i-common/bin/ $STI_SCRIPTS_PATH
 COPY 10.1/root /
 
 # this is needed due to issues with squash

10.1/Dockerfile.rhel7

@@ -1,4 +1,4 @@
-FROM rhel7
+FROM rhscl/s2i-core-rhel7
 
 # MariaDB image for OpenShift.
 #
@@ -11,6 +11,7 @@ FROM rhel7
 #  * $MYSQL_ROOT_PASSWORD (Optional) - Password for the 'root' MySQL account
 
 ENV MYSQL_VERSION=10.1 \
+    APP_DATA=/opt/app-root/src \
     HOME=/var/lib/mysql
 
 ENV SUMMARY="MariaDB 10.1 SQL database server" \
@@ -43,7 +44,7 @@ RUN yum repolist > /dev/null && \
     yum-config-manager --enable rhel-7-server-rpms && \
     yum-config-manager --enable rhel-7-server-optional-rpms && \
     yum-config-manager --enable rhel-server-rhscl-7-rpms && \
-    INSTALL_PKGS="rsync tar gettext hostname bind-utils rh-mariadb101" && \
+    INSTALL_PKGS="rsync tar gettext hostname bind-utils groff-base shadow-utils rh-mariadb101" && \
     yum install -y --setopt=tsflags=nodocs $INSTALL_PKGS && \
     rpm -V $INSTALL_PKGS && \
     yum clean all && \
@@ -63,6 +64,7 @@ ENV BASH_ENV=${CONTAINER_SCRIPTS_PATH}/scl_enable \
     PROMPT_COMMAND=". ${CONTAINER_SCRIPTS_PATH}/scl_enable"
 
 COPY 10.1/root-common /
+COPY 10.1/s2i-common/bin/ $STI_SCRIPTS_PATH
 COPY 10.1/root /
 
 # this is needed due to issues with squash

10.1/root/usr/share/container-scripts/mysql/README.md

@@ -140,6 +140,81 @@ location is `/etc/my.cnf` but you can change it to `/etc/mysql/my.cnf` by settin
  `MYSQL_DEFAULTS_FILE=/etc/mysql/my.cnf`
 
 
+Extending image
+---------------
+This image can be extended using [source-to-image](https://github.com/openshift/source-to-image).
+
+For example, to build a customized MariaDB database image `my-mariadb-rhel7`
+with a configuration in `~/image-configuration/` run:
+
+```
+$ s2i build ~/image-configuration/ rhscl/mariadb-101-rhel7 my-mariadb-rhel7
+```
+
+The directory passed to `s2i build` can contain these directories:
+
+`mysql-cfg/`
+    When starting the container, files from this directory will be used as
+    a configuration for the `mysqld` daemon.
+    `envsubst` command is run on this file to still allow customization of
+    the image using environmental variables
+
+`mysql-pre-init/`
+    Shell scripts (`*.sh`) available in this directory are sourced before
+    `mysqld` daemon is started.
+
+`mysql-init/`
+    Shell scripts (`*.sh`) available in this directory are sourced when
+    `mysqld` daemon is started locally. In this phase, use `${mysql_flags}`
+    to connect to the locally running daemon, for example `mysql $mysql_flags < dump.sql`
+
+Variables that can be used in the scripts provided to s2i:
+
+`$mysql_flags`
+    arguments for the `mysql` tool that will connect to the locally running `mysqld` during initialization
+
+`$MYSQL_RUNNING_AS_MASTER`
+    variable defined when the container is run with `run-mysqld-master` command
+
+`$MYSQL_RUNNING_AS_SLAVE`
+    variable defined when the container is run with `run-mysqld-slave` command
+
+`$MYSQL_DATADIR_FIRST_INIT`
+    variable defined when the container was initialized from the empty data dir
+
+During `s2i build` all provided files are copied into `/opt/app-root/src`
+directory into the resulting image. If some configuration files are present
+in the destination directory, files with the same name are overwritten.
+Also only one file with the same name can be used for customization and user
+provided files are preferred over default files in
+`/usr/share/container-scripts/mysql/`- so it is possible to overwrite them.
+
+Same configuration directory structure can be used to customize the image
+every time the image is started using `docker run`. The directory has to be
+mounted into `/opt/app-root/src/` in the image
+(`-v ./image-configuration/:/opt/app-root/src/`).
+This overwrites customization built into the image.
+
+
+Securing the connection with SSL
+--------------------------------
+In order to secure the connection with SSL, use the extending feature described
+above. In particular, put the SSL certificates into a separate directory:
+
+    sslapp/mysql-certs/server-cert-selfsigned.pem
+    sslapp/mysql-certs/server-key.pem
+
+And then put a separate configuration file into mysql-cfg:
+
+    $> cat sslapp/mysql-cfg/ssl.cnf
+    [mysqld]
+    ssl-key=${APP_DATA}/mysql-certs/server-key.pem
+    ssl-cert=${APP_DATA}/mysql-certs/server-cert-selfsigned.pem
+
+Such a directory `sslapp` can then be mounted into the container with -v,
+or a new container image can be built using s2i.
+
+
 Changing the replication binlog_format
 --------------------------------------
 Some applications may wish to use `row` binlog_formats (for example, those built

10.1/s2i-common

@@ -0,0 +1 @@
+../s2i-common

examples/extend-image/mysql-cfg/myconfig.cnf

@@ -0,0 +1,3 @@
+[mysqld]
+query-cache-limit=262144
+

examples/extend-image/mysql-data/init.sql

@@ -0,0 +1,4 @@
+CREATE TABLE products (id INTEGER, name VARCHAR(256), price FLOAT, variant INTEGER);
+CREATE TABLE products_variant (id INTEGER, name VARCHAR(256));
+INSERT INTO products_variant (id, name) VALUES ('1', 'blue'), ('2', 'green');
+

examples/extend-image/mysql-init/80-add-arbitrary-users.sh

@@ -0,0 +1,17 @@
+create_arbitrary_users() {
+  # Do not care what option is compulsory here, just create what is specified
+  log_info "Creating user specified by MYSQL_OPERATIONS_USER (${MYSQL_OPERATIONS_USER}) ..."
+mysql $mysql_flags <<EOSQL
+    CREATE USER '${MYSQL_OPERATIONS_USER}'@'%' IDENTIFIED BY '${MYSQL_OPERATIONS_PASSWORD}';
+EOSQL
+
+  log_info "Granting privileges to user ${MYSQL_OPERATIONS_USER} for ${MYSQL_DATABASE} ..."
+mysql $mysql_flags <<EOSQL
+      GRANT ALL ON \`${MYSQL_DATABASE}\`.* TO '${MYSQL_OPERATIONS_USER}'@'%' ;
+      FLUSH PRIVILEGES ;
+EOSQL
+}
+
+if ! [ -v MYSQL_RUNNING_AS_SLAVE ]; then
+  create_arbitrary_users
+fi

examples/extend-image/mysql-init/90-init-db.sh

@@ -0,0 +1,12 @@
+init_arbitrary_database() {
+  local thisdir
+  local init_data_file
+  thisdir=$(dirname ${BASH_SOURCE[0]})
+  init_data_file=$(readlink -f ${thisdir}/../mysql-data/init.sql)
+  log_info "Initializing the arbitrary database from file ${init_data_file}..."
+  mysql $mysql_flags ${MYSQL_DATABASE} < ${init_data_file}
+}
+
+if ! [ -v MYSQL_RUNNING_AS_SLAVE ] && $MYSQL_DATADIR_FIRST_INIT ; then
+  init_arbitrary_database
+fi

examples/extend-image/mysql-pre-init/80-check-arbitrary-users.sh

@@ -0,0 +1,10 @@
+check_arbitrary_users() {
+  if ! [[ -v MYSQL_OPERATIONS_USER && -v MYSQL_OPERATIONS_PASSWORD && -v MYSQL_DATABASE ]]; then
+    echo "You need to specify all these variables: MYSQL_OPERATIONS_USER, MYSQL_OPERATIONS_PASSWORD, and MYSQL_DATABASE"
+    return 1
+  fi
+}
+
+if ! [ -v MYSQL_RUNNING_AS_SLAVE ]; then
+  check_arbitrary_users
+fi