Updated container deployment doc and added container upgrade doc.

Change-Id: Iac2e8e44474f0eb77ce080ed5f85b890c5396d0c
(cherry picked from commit 303503b)

STS-CONTAINER-DEPLOYMENT.md

@@ -1,81 +1,109 @@
-Secure Token Server Container Deployment
-========================================
+# Build
+To build the Lightwave STS Container, follow the instructions in
+BUILD.md. The build process generates a saved Docker container in the
+file vmware-lightwave-sts.tar.
 
-
+# Overview
+There are many disadvantages in having the application and the
+persistent data to co-exist in a single container. Co-existing the
+persistent data with the application causes issues with upgrades,
+portability, backup and restore. To overcome these disadvantages,
+store the persistent data in in volumes created in a data-only
+container.
 
-Build
------
+# Deploy Lightwave using a data-only container
+The following steps show how to deploy the container image on a Photon
+host.
 
-To build the Lightwave STS Container, follow the instructions in BUILD.md. The
-build process generates a saved Docker container in the file
-vmware-lightwave-sts.tar.
+Enable Docker on Photon machine
 
-
+    systemctl status docker
+    systemctl start docker
 
-Deployment
-----------
+### Transfer the lightwave container image onto your docker host
 
-The following steps show how to deploy the container image on a Photon host.
+    scp <lightwave-build-machine>:/root/lightwave/stage/vmware-lightwave-sts.tar .
 
- 
+### Load the image
 
-### 1. Make sure that the docker daemon is running.
+     docker load < vmware-lightwave-sts.tar
 
-~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
-bash-4.3# systemctl status docker
-~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+### Check image list
 
-If the daemon is not active, start it with the command:
+    # docker images
+    REPOSITORY             TAG                 IMAGE ID            CREATED             SIZE
+    vmware/lightwave-sts   latest              1a712667c72d        About an hour ago   656.5 MB
 
-~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
-bash-4.3# systemctl start docker
-~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+### Create the lightwave data container
 
- 
+This creates a container with the needed volumes for the data. 
 
-### 2. Load the container image:
+Note: Volumes are separate entities from containers and persist beyond
+the life of a container. Application containers can use these volumes
+by running with --volumes-from <data-container> commandline argument.
 
-~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
-bash-4.3# docker load < vmware-lightwave-sts.tar
-~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+    # docker create -v /var/lib/vmware -v /var/lib/likewise -v /etc/likewise -v /etc/vmware-sso  --name lw_data_container vmware/lightwave-sts /bin/true
+    b6c1f9206b5bcb2011bf97eb63e52c2d15923f6ebfc2f10b5513eb07be987c61
 
- 
+### Check the lightwave data container is created
 
-### 3. Create a configuration file
+    # docker ps -a 
+
+    CONTAINER ID        IMAGE                  COMMAND                  CREATED             STATUS              PORTS               NAMES
+    b6c1f9206b5b        vmware/lightwave-sts   "/usr/sbin/init /bin/"   37 seconds ago      Created                                 lw_data_container
 
-Example data to be placed in /var/lib/vmware/config/lightwave-server.cfg:
+### Create the config file
 
-~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
-deployment=standalone
-domain=vsphere.local
-admin=Administrator
-password=<Administrator password>
-site-name=Default-first-site
-first-instance=true
-hostname=<Host IP address>
-~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+The location of the /var/lib/vmware/config directory on the host 
 
- 
+Name of the file is lightwave-server.cfg
 
-### 4. Run a container
+*For the first node:*
+
+    # cat /var/lib/vmware/config/lightwave-server.cfg
+    deployment=standalone
+    domain=vsphere.local
+    admin=Administrator
+    password=<administrator-password>
+    site-name=Default-first-site
+    first-instance=true
+    hostname=<ip>
 
-~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
-bash-4.3# docker run -d --name <name> --privileged --net=host \
-                     -v /sys/fs/cgroup:/sys/fs/cgroup:ro \
-                     -v /var/lib/vmware/config:/var/lib/vmware/config \
-                     vmware/lightwave-sts
-~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+*For subsequent nodes that will be joined to existing node:*
 
-Notes:
+    # cat /var/lib/vmware/config/lightwave-server.cfg
+    deployment=partner
+    domain=vsphere.local
+    admin=Administrator
+    password=<administrator-password>
+    site-name=Default-first-site
+    hostname=<ip>
+    replication-partner-hostname=<partner hostname or ip>
+
+### Start the Application container
 
--   Choose a unique name for the container
+This will spin up the Lightwave application container. The
+--volumes-from argument has this container use the data volumes in
+data container. 
+
+    # docker run -d --name lw-sts --privileged --net=host -v /sys/fs/cgroup:/sys/fs/cgroup:ro -v /var/lib/vmware/config:/var/lib/vmware/config --volumes-from lw_data_container vmware/lightwave-sts
+
+### Verify deployment was successful
+    #  docker exec <container-id> journalctl | grep configure-lightwave-server
 
--   This starts the container in host networking mode, meaning that it shares
-    the networking configuration with the container host. Only one container can
-    be running on the host in this mode.
+### Remove Lightwave configuration file
+This file contains administrator credentials and should be deleted
+after container is started.
 
--   The directory /var/lib/vmware/config will be mounted from the host to the
-    container and the lighwave-server.cfg file created in step 3 will be used to
-    automatically configure Lightwave the first time the container is run.
 
-
+    # rm /var/lib/vmware/config/lightwave-server.cfg 
+
+Notes:
+-   Choose a unique name for the container
+-   This starts the container in host networking mode, meaning that it
+    shares the networking configuration with the container host. Only
+    one container can be running on the host in this mode.
+-   The directory /var/lib/vmware/config will be mounted from the host
+    to the container and the lighwave-server.cfg file created in step
+    3 will be used to automatically configure Lightwave the first time
+    the container is run.

STS-CONTAINER-UPGRADE-BACKUP-RESTORE.md

@@ -0,0 +1,144 @@
+# How to perform container upgrade
+
+The following steps require that Lightwave has been configured to use
+data volumes. (See STS-CONTAINER-DEPLOYMENT.md)
+
+For multiple replicas, backup/upgrade/rollback can be performed on
+individual nodes one at a time, only temporarily bringing down the
+node currently being operated on. In the case of an upgrade failure,
+only the failed node will need to be rolled back. Having nodes with
+different versions in a domain is supported.
+
+**Tag images**
+
+New images may have same name as the old image. To keep them
+organized, tag the image before loading a new image:
+
+    # docker tag vmware/lightwave-sts vmware/lightwave-sts-old
+
+## Create a backup of data from volumes
+
+Before performing upgrade, backup all data for use in rollback in the
+case of upgrade failure.
+
+#### Create a backup directory on the host
+    # mkdir backup-6-6-0
+
+#### Create a new 'backup' container
+
+This container will map a volume to a backup directory on the host,
+and can be used for backup and restore operations.
+
+**Create backup container:**
+
+    # docker run -d --name backup --volumes-from lw_data -v $(pwd)/backup-6-6-0:/backup vmware/lightwave-sts
+
+**Create tars of all data volumes**
+
+    # docker exec backup tar cvfP /backup/lib-vmware.tar /var/lib/vmware
+    # docker exec backup tar cvfP /backup/lib-likewise.tar /var/lib/likewise
+    # docker exec backup tar cvfP /backup/etc-likewise.tar /etc/likewise/
+    # docker exec backup tar cvfP /backup/etc-vmware-sso.tar /etc/vmware-sso
+
+**Remove backup container**
+
+    # docker stop <backup container id>
+    # docker rm <backup container id>
+
+## Deploy Upgraded Container
+
+**Stop running Lightwave container**
+
+    # docker stop <lw container name>
+
+**Create/edit config file so that domain, deployment and vmdir
+  password in the lightwave-server config**
+
+    # cat /var/lib/vmware/config/lightwave-server.cfg
+    deployment=standalone
+    domain=vsphere.local   <<<<
+    admin=Administrator    <<<<
+    password=Admin!23
+    first-instance=true
+    site-name=Default-first-site
+    hostname=10.118.97.160
+
+**Load new container:**
+
+    # docker load < vmware-lightwave-sts.tar
+
+**Deploy a container with the new image using same data volumes as
+  previous installation**
+
+Giving the container a distinguishing name may help with managing the
+many containers that may get used during the upgrade process.
+
+During initialization, the container will detect if upgrade logic is
+to executed, put the directory into non-replication state and perform
+any data patching needed.
+
+Example:
+
+    # docker run -d --name lightwave-1-1-0 --privileged --net=host -v /sys/fs/cgroup:/sys/fs/cgroup:ro  -v /var/lib/vmware/config:/var/lib/vmware/config --volumes-from lw_data_container vmware/lightwave-sts
+    a689a33d718ff41692d230f4c39b3422e759f68d764c0a4a0638aca5af9af80f
+
+If upgrade is successful, the directory will be taken out of
+non-replication state.
+
+### Verify that upgrade was successful
+
+**Check journalctl for upgrade completion:**
+
+    # docker exec lightwave-1-1-0 journalctl | grep configure-lightwave-server
+    <snip>
+    Sep 10 00:12:10 photon-ga.eng.vmware.com configure-lightwave-server[64]: Running vdcupgrade
+    Sep 10 00:12:10 photon-ga.eng.vmware.com configure-lightwave-server[64]: Directory upgrade success.
+    Sep 10 00:12:10 photon-ga.eng.vmware.com configure-lightwave-server[64]: Upgrade complete.
+
+**Remove lightwave config file**
+
+This file contains administrator credentials and should be deleted
+when finished with upgrade.
+
+    # rm /var/lib/vmware/config/lightwave-server.cfg
+
+## Restore due to failed upgrade
+
+If journalctl indicates an upgrade failure, a rollback may be
+performed to revert Lightwave container back to known good state.
+
+**Create a new 'rollback' container**
+
+As with backup, this container will map a volume to the backup
+directory on the host.
+
+*Note: If a data volumes container was created during installation,
+ the failed container can be killed at this point, and the data
+ container can provide the data volumes. Otherwise, the running failed
+ container can provide the volumes since it is in a non-replication
+ state.*
+
+**Create rollback container using volumes from the container to be
+  restored:**
+
+    # docker run -d --name rollback --volumes-from lw_data -v $(pwd)/backup-6-6-0:/backup vmware/lightwave-sts
+
+**Extract tars of all data volumes**
+
+    # docker exec rollback tar xvfP backup/lib-vmware.tar
+    # docker exec rollback tar xvfP backup/lib-likewise.tar
+    # docker exec rollback tar xvfP backup/etc-likewise.tar
+    # docker exec rollback tar xvfP backup/etc-vmware-sso.tar
+
+**Remove rollback container**
+
+    # docker stop <rollback container id>
+    # docker rm <rollback container id>
+
+*If the failed upgrade container has not been stopped, do so before
+ starting a container using the old image.*
+
+**Restart old version container or use docker run to start a new
+  instance of the old Lightwave container using the restored data:**
+
+    # docker run -d --name vmsts-restored --privileged --net=host -v /sys/fs/cgroup:/sys/fs/cgroup:ro --volumes-from <backup image id>  <original LW image id>