Procedure for setting up a new OpenStack VM configured to serve as either a Test or Production VM.

[tmcphillips]

This will allow developers to set up their own development VMs identically to the Test and Production VMs. The procedure will include configuring user accounts, mounting NFS volumes, installing and configuring Docker and its dependencies, and maintaining system security.

[jterstriep]

I have some Ansible tools to do this but I don't think we need it. The current approach everything would be running in containers and you never need to customize the host VM (other than to install docker). Checkout docker-machine for a cool mechanism to create a docker development VM and launch containers. We could have some development images that inherit from the production image and load some development data using the ENTRYPOINT. The NFS mount from the storage condo is problematic requiring an email to help to get the export going, so non-admins (i.e. me) shouldn't have to worry about it. This shouldn't be an issue for developers or testers or other people doing a deployment if we do it right.

…

On Wed, Dec 13, 2017 at 7:58 PM Timothy McPhillips ***@***.***> wrote: This will allow developers to set up their own development VMs identically to the Test and Production VMs. The procedure will include configuring user accounts, mounting NFS volumes, installing and configuring Docker and its dependencies, and maintaining system security. — You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub <#21>, or mute the thread <https://github.com/notifications/unsubscribe-auth/ACdK7BJKneBNlHDoyPZNtiD6NrAwhOVbks5tAIDhgaJpZM4RBcFZ> .

[tmcphillips]

I was thinking that this can be short text or markdown document, just a few steps, ideally with links to existing detailed instructions for doing what's required.

I imagine our document would include what initial VM image on OpenStack to start from. A link to instructions for installing the current version of docker from Docker's own apt package repository rather that simply using apt-get to install the docker.io or docker-engine packages from Ubuntu's default repositories. Any customization of docker once its installed. Instructions for automating security updates if that's what we're going to do. Also what user account(s) and groups to create along with the UIDs and GIDs to use. Setting the hostname properly is also something that is easy to get wrong (or forget to do) which then causes havoc with some network services. And part of the procedure would be to ask you to request authorization for the NFS mount to the new host. (I'd also include contact info for the person you contact to get this done in case you are away and can't do it).

I can draft the document and you can correct it to match how the production and test VMs are actually configured if that works for you. The document doesn't have to publicly accessible.

[tmcphillips]

When running the elasticsearch:6.1.1 docker image on Ubuntu 16.04 for the first time I ran into an issue (it sounds like others have run into this issue as well) that should be addressed in the instructions for setting up an Ubuntu VM to run the SKOPE application. The elasticsearch:6.1.1 container exits after reporting this error:

max virtual memory areas vm.max_map_count [65530] is too low, increase to at least [262144]

The above is a suggestion to change the Linux kernel parameter vm.max_map_count. I did so by adding this line to /etc/sysctl.conf on the Ubuntu host and rebooting:

vm.max_map_count = 320000

Following the reboot the elasticsearch:6.1.1 image runs fine.

Note that because Docker containers use the kernel provided by the host, it is the kernel of the host machine (for example, any new VM on Nebula that will be running elasticsearch for SKOPE) that needs to be reconfigured to satisfy the kernel requirements of this Docker image.

(I'm assuming we're not considering running Docker in privileged mode, which might allow containers to reconfigure the host kernel at run time.)

[tmcphillips]

Just learned that Whole Tale is automating deployment on OpenStack using Terraform:
https://github.com/whole-tale/terraform_deployment

With the process documented in the repo above one can "launch almost exact copy of WT on
any OpenStack," according to Kacper. They have tested this both on Nebula and XSEDE Jetstream.

For SKOPE it's good to know from this that (a) once our deployment becomes complex enough there are ways to deploy multiple VMs, a Docker swarm cluster, and numerous services nearly from scratch in a reproducible way; and (b) doing so is practical enough that another project at NCSA is actually doing this.