Docker updates for WW 2.17: ADDITIONAL_BASE_IMAGE_PACKAGES and updated OPL metadata download approach

[taniwallach]

1. Modify the approach Docker uses to get the OPL metadata - to use the new approach.
2. Add a new feature to make it possible to use a new build time argument to add additional packages to the stage 1 (OS) image so which is more efficient that adding them each time a container is started.

docker build --tag webwork-base:forWW217 --build-arg ADDITIONAL_BASE_IMAGE_PACKAGES="nano less" -f DockerfileStage1 .

[taniwallach]

@mgage - The idea of the ADDITIONAL_BASE_IMAGE_PACKAGES is intended to make it easy to have persistent local additional packages in the image itself. I did not find a manner where this can be done without the extra packages being set on the docker build command line. I do not like the startup delay caused when the packages are installed at container startup time using ADD_APT_PACKAGES.

[drgrice1]

I haven't tested this yet, but the general concept of the things done here looks good. I will try to do some testing this weekend.

[taniwallach]

For testing the OPL metadata feature without building a new image:

1. Mount the updated docker-entrypoint.sh and bin/OPL-update into the container via docker-compose.yml
2. Use the steps below to force the process to run on the next startup.

Forcing an existing container to download the metadata from GitHub (on the next startup) can be done by starting a shell in the running container docker container exec -it webwork2_app_1 bash and then running:

touch $APP_ROOT/libraries/Restore_or_build_OPL_tables
rm $APP_ROOT/libraries/webwork-open-problem-library/TABLE-DUMP/OPL-tables.sql

so that the next docker-compose up will need to download the metadata.

When htdocs/DATA is not persistent, on a second restart it should not download the metadata again from GitHub but just reload it and copy the JSON file back to where they are needed.

[taniwallach]

I brought up my development server (docker based) with a branch on this PR and what was then an up-to-date WW 2.17 using the branch https://github.com/taniwallach/webwork2/tree/WW217-with-PR-1722 .

---

Updating an existing OPL volume

I found it convenient to update the OPL git settings before building the Docker images, etc. as I have an existing storage volume with the OPL:

docker-compose up -d
docker container exec -it webwork2_app_1 bash
cd libraries/webwork-open-problem-library/
git branch -m master main
git fetch origin

# Hit an error
# edited .git/config to replace the "fetch" line in the
# [remote "origin"] section:
# OLD:   fetch = +refs/heads/master:refs/remotes/origin/master
# NEW:   fetch = +refs/heads/main:refs/remotes/origin/main

git fetch origin
git branch -u origin/main main
git remote set-head origin -a

It would be a cleaner test to use a different name for the OPL volume in docker-compose.yml and get it to be newly created.

---

Building a WW 2.17 image with the changes in this PR

Something like the following should probably work (sorry, my Docker setup for production has lots of customization in it, SSL settings, a special location for the docker-compose.yml so a build-context, etc.).

Pick what extra packages to install below. I used vim and some others, as find it inconvenient not to have a nice editor in the running container.

docker pull ubuntu:20.04

cd /your_path_to_webwork2
git add remote tani https://github.com/taniwallach/webwork2.git 
git fetch tani
git checkout WW217-with-PR-1722
docker build --tag webwork-base:forWW217 --build-arg ADDITIONAL_BASE_IMAGE_PACKAGES="vim telnet mc file less" -f DockerfileStage1 .
vim docker-compose.yml
# Change
#      WEBWORK2_GIT_URL=https://github.com/taniwallach/webwork2.git
#      WEBWORK2_BRANCH=WW217-with-PR-1722
#      PG_BRANCH=PG-2.17

docker-compose build

docker-compose up -d

docker container exec -it webwork2_app_1 bash

• Next few steps relevant if there are existing courses mounted to your Docker container.

docker container exec -it webwork2_app_1 bash
webwork2/bin/upgrade_admin_db.pl

• Log in to admin course
• Upgrade courses

---

Testing the OPL-update changes

Next steps relevant if there was an exiting OPL volume which prevented docker-entrypoint.sh from doing the OPL-update.
If a new OPL volume was in use - it should just do it automatically.

docker container exec -it webwork2_app_1 bash
export SKIP_UPLOAD_OPL_statistics=1
webwork2/bin/OPL-update

[drgrice1]

This looks good. One minor detail with a couple of unneeded lines.

[mgage]

I successfully rebuilt my course using tani's branch WW217-with-PR-1722
I was able to install vim using the build-arg ADDITIONAL_BASE_IMAGE_PACKAGES

I couldn't find the script
touch $APP_ROOT/libraries/Restore_or_build_OPL_tables ()
I deleted the OPL tables anyway from inside the container.
now the library browser doesn't work -- with error
DBD::MariaDB::db selectall_arrayref failed: Unknown column 'pgf.libraryroot' in 'where clause' at /opt/webwork/webwork2/lib/WeBWorK/Utils/ListingDB.pm line 522.

Perhaps I was supposed to rebuild one of the volumes as well?

[mgage]

I was able to build the webwork2 container from scratch including adding vim using the build-arg.

After a couple of tries I was able to rebuild the library tables. I found that starting from docker-compose build and then using docker-compose up was the easiest way for me to make sure that I was using the latest OPL-update script

[drgrice1]

I will merge this now.