⚠️ WARNING (12/3/2021) ⚠️ – This project is not set up to work with the maple release of edx-platform, and possibly some earlier releases. If no one has the time to add support for later releases to this project, it's best to set up your local devstack the old-fashioned way (running a shell in LMS/Studio, then changing config settings and installing packages in there)

ODL Devstack Tools

This repo provides some helpful tools for configuring and running devstack (edX's Docker-based solution for running Open edX services).

Features

• Create a custom docker image (which is just a layer on top of edX's devstack image) with some helpful packages already installed (e.g.: pdbpp)
• Automatically apply changes to the JSON config files in devstack when running the containers (lms.yml or studio.yml for Juniper or later; lms.env.json, cms.env.json, lms.auth.json, or cms.auth.json for every release before Juniper).
• Via docker-compose configuration, specify local repos that you want mounted into the devstack containers and installed. This is useful for testing changes in repos that devstack depends on (e.g.: XBlock, xblock-utils, edx-sga). If the packages use setuptools, they'll be installed via pip with the -e flag, so the server will restart automatically when you make changes to those packages locally.
• Create any number of docker-compose files to take advantage of the features above for different purposes (e.g.: one compose file for testing changes to XBlock, another one for testing changes to edx-sga, another one for testing some LMS feature that requires a bunch of config changes). You can also combine them as desired.
• If desired, sets up your container to enable exporting courses to Github (specifically the github.mit.edu private repos) so you don't need to create SSH keys and add them to Github every single time you restart your containers.

Usage

1) Set environment variables

These environment variables will need to be set in your host machine (they can be added to ~/.bash_profile, et. al.):

# The name of the custom devstack image that will be built as a layer on top of the devstack image.
CUSTOM_DEVSTACK_IMG_NAME="edxops/edxapp:odlcustom"
# The name of the existing devstack image on top of which your custom image will be based. Defaults to 'edxops/edxapp:latest'
CUSTOM_DEVSTACK_BASE_IMG="edxops/edxapp:latest"
# Path to this repo on your machine.
CUSTOM_DEVSTACK_PATH="/path/to/odl_devstack_tools"

# The path to helper files in the container. ***Do not change this value***
DEVSTACK_CONTAINER_HELPER_DIR="/edx/app/edxapp/helper"
# The path to the directory in the container where local repos will be mounted. ***Do not change this value***
DEVSTACK_CONTAINER_MOUNT_DIR="/edx/app/edxapp/venvs/edxapp/src"

2) Build custom image

When you have the latest images from edX (via make dev.pull - more details here), you can run the following command to create the new image based on edX's image.

docker build $CUSTOM_DEVSTACK_PATH -t $CUSTOM_DEVSTACK_IMG_NAME --build-arg BASE_IMG=$CUSTOM_DEVSTACK_BASE_IMG --no-cache

3) Run devstack with the added compose file(s)

To spin up the containers using the custom image and take advantage of the features described above, run docker-compose up like the commands below. For the sake of familiarity and consistency, these commands mimic the 'make dev.up' command in devstack.

# Run LMS with the basic custom docker-compose file
docker-compose -f docker-compose.yml -f docker-compose-host.yml -f $CUSTOM_DEVSTACK_PATH/docker-compose-custom.yml up -d lms
# Run LMS with the basic custom docker-compose file and an additional one that you created
docker-compose -f docker-compose.yml -f docker-compose-host.yml \
  -f $CUSTOM_DEVSTACK_PATH/docker-compose-custom.yml -f $CUSTOM_DEVSTACK_PATH/docker-compose-mine.yml up -d lms

Applying config changes

Summary

Changes to YML/JSON config values can be automatically applied when the container starts by doing the following:

• Create a JSON patch file in this repo's ./configpatch directory (NOTE: The files are written as JSON patches, but they will work as expected with YAML config files)
• In your custom docker-compose file, mount the JSON patch file into the ${DEVSTACK_CONTAINER_HELPER_DIR}/configpatch/ directory in the container

Detail

You can automatically apply changes to any of the devstack JSON config files (lms.yml or studio.yml for Juniper or later; lms.env.json, cms.env.json, lms.auth.json, or cms.auth.json for every release before Juniper) by creating a patch file in the ./configpatch directory. These patch files should be .json type and in jsonpatch format. The file name doesn't matter. See the .example files in the ./configpatch/ directory for example usage.

After the patch file is created, the patch file needs to be mounted into the container by adding to the volume section of your docker-compose file.

services:
  lms:
    volumes:
      - ${CUSTOM_DEVSTACK_PATH}/configpatch/my_patch.json:${DEVSTACK_CONTAINER_HELPER_DIR}/configpatch/my_patch.json

Specifying local repos to be mounted and installed automatically

Summary

Local repos can be mounted and installed in devstack containers by doing the following:

• In your docker-compose file, add a environment variable with a name that starts with ADDED_REQ_, with the value being the package name (e.g.: ADDED_REQ_XBLOCK=XBlock)
• In your docker-compose file, mount your local repo directory alongside the edx-platform virtualenv packages (e.g.: /path/to/repo/XBlock:${DEVSTACK_CONTAINER_MOUNT_DIR}/XBlock)

Detail

When hacking on or testing changes to some package that edX depends on, it's very helpful to mount and install your local repo into the devstack containers. Among the benefits: (1) changes you make to those repos trigger a server restart automatically for LMS/Studio (if your package can be installed as "editable"); (2) when you create a migration for one of those repos in the devstack container, the migration files are immediately available in your local repo.

Running docker-compose up with the example docker-compose file below would automatically mount the XBlock repo and install it as a local package in the lms container.

services:
  lms:
    environment:
      - ADDED_REQ_XBLOCK=XBlock
    volumes:
      - /path/to/repo/XBlock:${DEVSTACK_CONTAINER_MOUNT_DIR}/XBlock

Enabling course content exporting to Github in Studio

If you want to be able to export course content to github.mit.edu, follow these steps:

1. On your host machine, generate an SSH key (Github guide) and move the keys (id_rsa and id_rsa.pub) to path/to/odl_devstack_tools/ssh. NOTE: Use your github.mit.edu email address and NO PASSPHRASE.
2. Add that SSH key to your Github account (guide).
3. Apply all the necessary JSON settings values and advanced course settings related to course content exporting.

Extras

It's helpful to have a shorthand for running the containers using certain compose files explicitly or by default. The variables and bash function below provide that.

export USE_CUSTOM_DEVSTACK=true
export DEFAULT_CUSTOM_DEVSTACK_COMPOSE_FILE="docker-compose-custom.yml"

function dedxup() {
  local compose_file_args=()
  
  # Add custom compose file(s) to docker-compose params if it's set and the file exists
  if [ "$USE_CUSTOM_DEVSTACK" = true ]; then
    compose_file_args+=( '-f' "$CUSTOM_DEVSTACK_PATH/$DEFAULT_CUSTOM_DEVSTACK_COMPOSE_FILE" )
    if [ ! -z $ADDED_DEVSTACK_COMPOSE_FILE ]; then
      compose_file_args+=( '-f' "$CUSTOM_DEVSTACK_PATH/$ADDED_DEVSTACK_COMPOSE_FILE" )
    fi
    for compose_file_arg in "${compose_file_args[@]}"
    do
      if [ $compose_file_arg != '-f' ]; then
        echo -e "Using additional compose file (\033[1;92m$compose_file_arg\e[0m) ..."
      fi
    done
    echo ''
  fi
  
  docker-compose -f docker-compose.yml -f docker-compose-host.yml ${compose_file_args[@]} up -d $@
}

Example usages:

# Run LMS using the default custom compose file (docker-compose-custom.yml)
dedxup lms
# Run LMS without any custom compose files
USE_CUSTOM_DEVSTACK=false dedxup lms
# Run LMS using the default custom compose file AND an additional custom compose file
ADDED_DEVSTACK_COMPOSE_FILE='docker-compose-xblock-dev.yml' dedxup lms