Join GitHub today
GitHub is home to over 36 million developers working together to host and review code, manage projects, and build software together.Sign up
Docker newbie instructions
-- by Mike Gage
The docker files for webwork were constructed by Pan Luo (@xcompass) at UBC. Thanks Pan!!
This explanation is by and for newbies and may well be simplified, clarified, corrected in later editions. (Jan 6, 2018, July 21, 2018)
The purpose of using docker is to allow nearly everyone who has worked with webwork using the command line to download and test new patches or pull requests to WeBWorK. The initial setup takes a bit more than an hour. After that you can pull a new experimental version of WeBWorK in five minutes for review without interfering with any of the existing work on the computer. Faster testing will allow us to proceed more quickly with new WeBWorK features.
The ways of setting up Docker are evolving. What follows is a relatively simple (if slightly time consuming way) of installing WeBWorK inside docker. We are experimenting with installing webwork2, pg, the mysql database and OPL separately as docker components and then connecting them. This will make it easier to modify and save files in each of these components and to rebuild without having to rebuild everything. (Rebuilding the library database in particular takes some time.) Those with some experience using Docker and webwork may want to look at https://github.com/openwebwork/webwork2/wiki/Docker---things-to-get-used-to-(and-avoid-surprises) and review and contribute to experiments described in the develop branch and on the forum
I have run docker on the MacOS high sierra and am now running it on Mojave. You need at least El Capitan MacOS in order to run Docker.
Docker CE has been successfully installed on Linux machines.
Docker CE is available for Windows machines as well, but getting WeBWorK running under Docker CE for Windows is a bit more complicated than doing so on MacOS X or Linux.
- Docker CE for Windows depends on Hyper-V, so only works on Windows Professional, Education, or Enterprise, but not on Windows Home.
- There is an older Docker Toolbox for Windows which works differently, and which may work on on machines whose version of Windows does not support Docker CE for Windows.
- When Hyper-V is activated during the installation of Docker, it disables VirtualBox and other VM hypervisiors. (They cannot operate in parallel.)
- In order to run VirtualBox again you have to disable Hyper-V. To do so: uncheck Hyper-V in Control Panel -> Programs -> Programs and Features -> Turn Windows features on or off. Then reboot.
- Getting WeBWorK to work in Docker for Windows also requires:
- getting symbolic links to work properly in Windows (possible on recent versions of Windows when Developer Mode is enabled)
- and sharing the Windows drive on which you have your
Please report on your experiences. This wiki is versioned so don't worry about making mistakes when editing this document -- we can always restore an older version.
All commands are issued from the command line so you will need a terminal window. You will need some experience typing unix commands in order to follow these instructions.
Install Docker on your computer. (There are versions for Mac, Windows and Linux)
Docker download home Get the community edition. (CE)
The setup of docker on the Mac is standard and quick, see the Mac Instructions. For Windows see the Windows Instructions and https://servingmath.org/wp/2018/08/installing-webwork-on-docker-on-windows/
Testing the Docker installation
Test that it is installed by opening up a terminal and entering
$ docker --version Docker version 18.06.0-ce, build 0ffa825 $ docker-compose --version docker-compose version 1.22.0, build f46880f $ docker-machine --version docker-machine version 0.15.0, build b48dc28d
Do the "explore the application and run examples" section
mkdir webwork_docker # (create a directory anywhere on your disk) cd webwork_docker # all your work will be in this directory docker run -d -p 80:80 --name webserver nginx # start a local webserver
after a few minutes an image of the nginx image will have been pulled from the cloud and a container running the webserver will have been created from the image.
docker ps # see what containers are running, the first column is container ID gage-imac-461:webwork_docker mgage$ docker ps CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES 017118b37fef nginx "nginx -g 'daemon ofâ¦" 2 minutes ago Up 2 minutes 0.0.0.0:80->80/tcp webserver docker logs CONTAINER_ID # print log/stdout of the container with CONTAINER_ID gage-imac-461:webwork_docker mgage$ docker stop webserver webserver docker stop webserver # stop the server docker ps # no container is running (they are all stopped and therefore "hidden"). docker ps -a # includes all of the containers including ones that are stopped. gage-imac-461:webwork_docker mgage$ docker ps -a CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES 017118b37fef nginx "nginx -g 'daemon ofâ¦" 5 minutes ago Exited (0) About a minute ago webserver docker start webserver # restart the server docker rm -f webserver # remove the webserver (the -f stands for "force" ) # the image which you downloaded from the cloud and from which the container was created is still on your machine docker images gage-imac-461:webwork_docker mgage$ docker images REPOSITORY TAG IMAGE ID CREATED SIZE nginx latest 7042885a156a 44 hours ago 109MB docker rmi nginx # will remove the image completely
At this point you can read/skim the rest of the Mac Instructions, Windows Instructions, etc pages at your leisure. There are no changes that need to be made, but skimming to see what is available on this page can be helpful.
There is a second getting started page that is useful to read in order to get oriented. For example it explains the difference between a container and an image but first let's actually set up a webwork installation using docker.
Download and install a WeBWorK container:
Github and git activity
- The two Git config settings included below
-c core.autocrlf=false -c core.symlinks=trueshould really only be needed on Windows (as these are the default values on Linux and MacOS X), however forcing them should not interfere with anything on Linux and MacOS X.
- It is not necessary to install the
pgrepository below. You can skip that
git cloneinstruction unless you plan to make changes to the core
pgcode, in which case you will also need to edit
webwork2/docker-compose.ymlto include your local
pgtree at run-time. ( See https://github.com/openwebwork/webwork2/wiki/Docker---things-to-get-used-to-(and-avoid-surprises))
cd webwork_docker ls # it should be empty -- running the demo above left no trace in this directory git clone https://github.com/openwebwork/webwork2 -c core.autocrlf=false -c core.symlinks=true # grab the webwork2 repo #optional -- not needed until you start modifying pg # git clone https://github.com/openwebwork/pg -c core.autocrlf=false -c core.symlinks=true # grab and install the pg repo cd webwork2 # Change to the branch needed. (Docker is now part of the master branch so no change is needed. # If you are doing advanced work you may want to switch to the develop branch. git branch * master # should show "* master" highlighted.
- Depending on how fast and how accurately you type these git commands will take 5 minutes or less.
- The revised files will run
OPL-updateautomatically when the container is first run.
- this takes 20 minutes to complete.
- The revised files will run
Before proceeding, shut everything down in Docker and clear images.
docker ps -a should be clear.
We are still in the webwork2 directory:
If everything is successful the
ls listing will show the usual webwork files but also the files
Now while still in the
webwork2 directory type
docker build . docker-compose up
You will now have to wait for a while. The terminal window will scroll through lots of lines to let you know that something is running.
Next time you will probably prefer to run
docker-compose up -d # detaches the terminal so you don't see all the output # and so that you can quit the terminal without stopping the container.
You can see that it is installing a lot of things for the ubuntu version of the linux operating system.
Eventually it installs TeX and Apache, various CPAN modules (you'll see a lot of
Building and testing lines) and then MathJax. Then there are lines about creating the
mysql (dabase) application db_1 and finally the webwork application app_1. Then the admin
course is created and then it starts creating the OPL database.
IMPORTANT: Don't close the terminal window since that will stop the container running the website.
Towards the end you should see
........ app_1 | Admin course is created. app_1 | create modelCourse subdirectory app_1 | defaultClasslist.lst is being created app_1 | Library version is 2.5; using OPLtables! app_1 | Mysql database reinitialized. app_1 | Reading in textbook data from Textbooks in the library /opt/webwork/libraries/webwork-open-problem-library/OpenProblemLibrary. app_1 | Reading in OPL taxonomy from Taxonomy2 in the library /opt/webwork/libraries/webwork-open-problem-library/OpenProblemLibrary. app_1 | Saved taxonomy to /opt/webwork/webwork2/htdocs/DATA/tagging-taxonomy.json. app_1 | Converting data from tagged pgfiles into mysql. app_1 | Number of files processed: app_1 | 100 200 300 400 500 600 700 800 900 1000 .....
as docker-compose completes the installation of webwork (app_1), the database (db_1) and R (r_1) and it starts to build the OPL database. Creating the OPL database takes the most time. There are about 35K files that have to be processed. Go get some coffee. After the files are processed there are 126 textbooks that need to be processed to line up problems relevant to those text books.
On my iMac the whole process took about 35 minutes from typing
docker build . ; docker-compose up; to the final result
app_1 | Updating Library Statistics. app_1 | Importing statistics for 2 courses. app_1 | 1 2 app_1 | app_1 | mysql: [Warning] Using a password on the command line interface can be insecure. app_1 | app_1 | Done. app_1 | Fixing ownership and permissions (just in case it is needed) app_1 | AH00558: apache2: Could not reliably determine the server's fully qualified domain name, using 172.18.0.4. Set the 'ServerName' directive globally to suppress this message app_1 | webwork.apache2-config: WeBWorK server is starting app_1 | webwork.apache2-config: WeBWorK root directory set to /opt/webwork/webwork2 in webwork2/conf/webwork.apache2-config app_1 | webwork.apache2-config: The following locations and urls are set in webwork2/conf/site.conf app_1 | webwork.apache2-config: PG root directory set to /opt/webwork/pg app_1 | webwork.apache2-config: WeBWorK server userID is www-data app_1 | webwork.apache2-config: WeBWorK server groupID is root app_1 | webwork.apache2-config: The webwork url on this site is http://localhost/webwork2 app_1 | webwork.apache2-config: The webwork smtp server address is localhost app_1 | webwork.apache2-config: The webwork smtp server port is app_1 | webwork.apache2-config: The webwork smtp server protocol is 'not ssl' app_1 | WebworkSOAP::WSDL: webwork_directory set to /opt/webwork/webwork2 via $WeBWorK::Constants::WEBWORK_DIRECTORY set in webwork.apache2-config app_1 | WebworkSOAP::WSDL: rpc_url set to http://localhost/webwork2_rpc app_1 | WebworkWebservice: webwork_directory set to /opt/webwork/webwork2 via $WeBWorK::Constants::WEBWORK_DIRECTORY set in webwork.apache2-config
DO NOT CLOSE THIS WINDOW!!!! The window must be kept open while running the webserver. If you close the window it will shut down the webwork server.
Testing out the locally running WeBWorK container
Once the lines have stopped scrolling you can type
http://localhost:8080/webwork2 into your browser and you have a webwork site running.
The only course is "course administration" and it has login "admin" with password "admin".
You can login and add yourself as another admin in the admin course. (First add yourself as a student, then promote yourself to admin.)
You can check the library browser to make sure that it works and that the OPL database has been updated properly.
Create a new course 'test_course', add some guest "students" by importing from demoCourse.lst and add a few homework problems to the course using the LibraryBrowser.
What could possibly go wrong?
I forgot to select a target set from the course in the library browser. Usually the blank for the target set would read "Select a Set from this Course" but because no homework sets had been set up at all this field was set to "no sets in this course yet". This lead to an "instructorXMLRPC: 403 Forbidden" error.
Usually "instructorXMLRPC: 403 Forbidden" error is triggered by some glitch in setting up the site URL in the site.conf file so that it works properly with AJAX, but not in this case. Making sure that I had created a new set in the course and then choosing that set as a target was enough to clear the error.
I found a message in the error log at /var/log/apache2/error.log indicating that no homework set had been created for the course (see more information below on reading this log) but that message was not transmitted to the screen. It's likely that this particular error can only occur with a brand new, empty course.
On some builds I got an "app_1 exited with error 1" at the end of the "docker build ." command or the "docker-compose up". This is obviously not good. In some cases I had to go back to the beginning and start the reinstall from the beginning. This involved removing all prebuilt docker packages so
docker container ps -aand
docker imagesshould all be empty. If they are not use
docker container rm <id>and
docker rmi <id>to remove them. You only need to type the first couple of numbers/letters of the SHA identifier -- not the whole thing.
- The docker set up is still fragile. We hope to break it into pieces so that webwork2, pg, mysql and OPL are all handled separately. This way if one download fails you don't have to start the entire process over. In particular you can update changes to webwork2 and pg without necessarily rebuilding the OPL database.
In a new terminal window go to webwork_docker/webwork2. Don't close the first terminal window.
Where is the data being stored? You won't be able to see much of the data that you usually expect to see stored in webwork on the disk, at least not from "outside" the docker environment. One thing you can see is the courses directory.
A new directory has been created to hold persistent webwork2 data
cd webwork_docker ls ww-webwork-data courses ls ww-docker-data/courses admin/ adminClasslist.lst defaultClasslist.lst modelCourse/ test_course/
You'll see courses, which is where persistent data for courses run in the Docker container will be saved.
IMPORTANT: You can add (upload) archived courses to the courses directory and then unarchive them using the admin course in the browser. This is the quickest way to populate your docker webwork with courses you have been using on your production site. Even if you don't have command line access to your production site you can archive and download your own courses using the commands on the FileManager page of your course.
With older versions of the Docker config files, there will also be a
db directory which is the location for the persistent storage for the database. More recent versions of the Docker config files put the databases into a named Docker volume. It's wise not to touch the db directory unless you are a MariaDB expert.
Read more about the persistent and transient storage at https://github.com/openwebwork/webwork2/wiki/Docker---things-to-get-used-to-(and-avoid-surprises). It's one of the things that takes a while to get used to. The exact balance of which data is persistent and which is transient will continue to be tweaked as we work on creating the best experience of using docker with WeBWorK.
In your second terminal window (in directory
webwork_docker/webwork2) -- you may have to cd to
webwork_docker/webwork2 -- type
docker-compose down Stopping webwork2_app_1 ... done Stopping webwork2_db_1 ... done Stopping webwork2_r_1 ... done Removing webwork2_app_1 ... done Removing webwork2_db_1 ... done Removing webwork2_r_1 ... done
If you go back to your first window you will see that the build process has exited and your browser can no longer access the webwork site.
congratulations you've completed your first use of webwork on your laptop using docker.
Next try -- things will be quicker now:
In the first window type
docker-compose up -d Creating network webwork2_default with the default driver Creating webwork2_db_1 ... done Creating webwork2_r_1 ... done Creating webwork2_app_1 ... done
Notice that things are much quicker this time as the images are already built. And the webserver has started up right where you left off.
Close the first terminal window. And check that the website is still running. (This is what the -d accomplished.)
In the second terminal (webwork_docker/webwork2 directory) type
docker container ls # (or docker ps will also work)
Observe the container id for the
webwork container. Some commands will require at least the first 4 or 5 characters of that
WeBWorK_id to identify the container. Sometimes the name
webwork2_app_1 alone will work.
You will need this in the command below.
docker container exec -it WeBWorK_id bash root@WeBWorK_id:/opt/webwork#
A terminal window has just been opened to the inside of the "remote" machine at the directory
This command is worth creating an alias for. In my
.bashrc account I have added
alias dockerbash="docker container exec -it webwork2_app_1 bash"
so that I don't have to remember the whole string. This command lets you "login via a terminal window" to the webwork2 site running in the docker container. What you see should be similar to what
you see when you login via a command line to your production server or a development server. One difference is that you also have an "outside" view via the desktop of the permanent files. Changes made on the outside, e.g. copying a course from your desktop into the
ww-docker-data/courses folder can be seen (after a brief pause) via the terminal at
Anything that you would do by logging in to your local WeBWorK server on the command line you can do by "logging" in to the container using the approach above.
Warning: Recall that changes made in the command line of the Docker container will only be persistent if they are making changes to files on the persistent storage. See https://github.com/openwebwork/webwork2/wiki/Docker---things-to-get-used-to-(and-avoid-surprises). Courses are on permanent storage, changes to macro files in pg or in the OPL are not. Changes to a macro file in a course template/macros file will be preserved.
When you are done exploring type
exit and you will close the terminal window "connection" to the container (the remote machine)
Installing an archived course:
If you have a copy of an archived course that you've downloaded from your usual webwork site (a .gz file) we can place that in the container as follows.
- In webwork_docker/webwork2
docker-compose down cp ../path/to/course/archiveFile.tgz ww-docker-data/courses docker-compose up
- Check the web browser and the admin course. You should be able to "unarchive" your course using the "unarchive" tab.
- If it is an old course you may have to "update courses" to fix the database on the course you have imported.
- In the web browser open the library browser and make sure that it works.
- You can now add more problems to that course. Changes will be preserved.
Double check that you are running the versions of webwork and pg that you want. At the bash prompt (which you opened inside the Docker container) type:
cd pg git fetch origin # will refresh you copy of the index # of the branches available at origin (aka github.com/openwebwork) git branch -a git checkout -b develop origin/develop # if you want to test the develop branch apachectl graceful # restarts the web server.
You do not need to rebuild the entire docker container to change from one branch of of webwork to another. (whew!)
Adding a new course
- Now try adding a new course from the browser using the Course Administration web page.
- You should see that you can build it using modelCourse as a template.
Shutting down (and restarting) the Docker container:
- When you are done using your Docker container type
docker-compose downand docker will stop using CPU's on your computer.
- It will still take up some disk memory space.
- It can be restarted with
docker-compose up -d.
- see Docker for newbies part 2 for adding daemon_course (still under development)
That's enough for now. Play with the WeBWorK site you have created and go back to skimming Docker's getting started docs to try to figure out what you have just done. We've only used the first two sections of that document -- there is more that you can do to create a "production site" that can handle many students.
Warning With what we have so far you DO NOT want to use this as a production WeBWorK site -- this is a local installation on your machine for your use only. None of the databases are using passwords for example, so there would not be much security if you start letting people do homework on a network connected to your computer.
To repeat, the purpose of using Docker (so far) is to allow nearly everyone who has worked with WeBWorK using the command line to download and test new patches or pull requests to WeBWorK. Faster testing will allow us to proceed more quickly with new WeBWorK features. You can for example create homework questions that use the latest features in the develop branch without endangering the behavior of your school's production WeBWorK site.
The UBC people are eventually planning to use docker on their production machines so that it is super simple to install WeBWorK.
Also watch out, this installation is temporary. If you create a WeBWorK course in your local site it will stay around if you do docker-compose down, which just makes the container dormant. It will also stay around if you quit the Docker app. If you remove the container however (a command with a "rm" or "rmi" in it) the site will disappear. You could first save the course using the "archive" button in the FileManager in WeBWorK and download the archived file. (You could also do this for multiple courses from Course Administration course.)
All the persistent files (database and course files) are stored in
webwork_docker/ww-docker-data directory. In order to do a completely clean uninstall of this "webwork in a box", remove the
ww-docker-data directory or
webwork_docker directory (and the named
mysql storage volume).