This repository contains the developer playground/workstation environment for the MIT Probabilistic Computing Project probcomp/notebook image. It contains the full probcomp software stack in addition to enabling active development on the various repositories including bayeslite, cgpm, crosscat, iventure and venturecxx.
In addition to running the software locally as detailed below, you also have the option of using the hosted service:
- Access is controlled by membership in the probcomp GitHub org.
- 96 CPU cores, 360GB of RAM
- 10GB persistent disk
- One hour browser-based inactivity timeout
- 10 max concurrent instances
A JupyterLab mode is also available which supports additional functionality like system shell terminals. To use lab, change the tree in the URL to lab after your notebook starts. ex: https://notebook.gcloud.probcomp.net/user/avinson/tree -> https://notebook.gcloud.probcomp.net/user/avinson/lab.
The docker image and commands in this documentation are tested with docker version 17.12.0-ce and docker-compose version 1.8.0. We recommend using these minimum versions.
The playground depends on local copies of various probcomp repos in a relative path to this directory. See the docker-compose.yml file for the full list or run the make bootstrap command:
$ make bootstrap
bayeslite repo not found at ../bayeslite. cloning...
Cloning into '../bayeslite'...
cgpm repo not found at ../cgpm. cloning...
Cloning into '../cgpm'...
crosscat repo not found at ../crosscat. cloning...
Cloning into '../crosscat'
.
.
Start your environment with the make up command. Other functionality in this documentation depends on the notebook container being running. Paste the URL from the output into your browser to access the jupyter environment and the tutorial notebooks.
$ make up
notebook_1 | Enabling develop on: bayeslite
notebook_1 | Uninstalling bayeslite conda package
notebook_1 | Creating bayeslite.egg-link
notebook_1 | Building bayeslite
notebook_1 | Set username to: jovyan
notebook_1 | usermod: no changes
notebook_1 | Set jovyan UID to: 501
notebook_1 | Granting jovyan sudo access and appending /opt/conda/bin to sudo PATH
notebook_1 | Execute the command: jupyter notebook
notebook_1 | [I 07:20:25.408 NotebookApp] Writing notebook server cookie secret to /home/jovyan/.local/share/jupyter/runtime/notebook_cookie_secret
notebook_1 | [W 07:20:25.630 NotebookApp] WARNING: The notebook server is listening on all IP addresses and not using encryption. This is not recommended.
notebook_1 | [I 07:20:25.636 NotebookApp] Serving notebooks from local directory: /home/jovyan
notebook_1 | [I 07:20:25.636 NotebookApp] 0 active kernels
notebook_1 | [I 07:20:25.636 NotebookApp] The Jupyter Notebook is running at:
notebook_1 | [I 07:20:25.636 NotebookApp] http://[all ip addresses on your system]:8888/?token=cd76eccc16179d46ef93ff866f318940a20b41a9883c7106
notebook_1 | [I 07:20:25.636 NotebookApp] Use Control-C to stop this server and shut down all kernels (twice to skip confirmation).
notebook_1 | [C 07:20:25.637 NotebookApp]
notebook_1 |
notebook_1 | Copy/paste this URL into your browser when you connect for the first time,
notebook_1 | to login with a token:
notebook_1 | http://localhost:8888/?token=cd76eccc16179d46ef93ff866f318940a20b41a9883c7106
The instructions below assume active development on the bayeslite repo but this playground environment supports all of the other probcomp public repos as well.
Development is enabled by default for the bayeslite repo. If you wish to add other repos, pass the DEVELOP_REPOS variable to your make up command. You can, of course, alternately edit your docker-compose.yml file directly to pass in different environment vairables. If you've already started your environment and want to add repos, run make down first.
$ make up
notebook_1 | Enabling develop on: bayeslite
.
.
$ DEVELOP_REPOS="bayeslite iventure" make up
notebook_1 | Enabling develop on: bayeslite iventure
.
.
Build local sources with make <REPO_NAME> to make your changes take effect:
make bayeslite
Run a repo's test suite with make <REPO_NAME>-test:
make bayeslite-test
Travis test runs can be run automatically by opening a PR against any of the probcomp repos. Additionally, tests automatically run every 24 hours against the master branch to give insight into test stability.
Releasing software to the probcomp anaconda.org channel occurs automatically when any of the following occurs. The conda label of the package is determined by the event type:
- Adding a tag to master (releases a
mainpackage) - Committing or merging a PR to master (releases an
edgepackage) - Adding a tag to a non-master branch (releases a
devpackage)
main packages are typically incorporated into the Dockerfile while edge packages ensure integration tests against other repos pass and dev packages are used for arbitrary additional functionality or testing.
The probcomp/notebook:latest image is automatically built on any commit to the probcomp/notebook repo's master branch. Tagged images are automatically created by pushing a tag to the master branch. For non-trivial changes, you should open a pull request which will trigger a jenkins job to run tests. Note that these tests require substantial computing resources and may take up to an hour to complete. Please use with care.
Additionally, the probcomp/notebook:edge image is built daily by a jenkins job to ensure that the latest edge packages are available in said image.
make up
make bash
make ipython
make julia
make pull
Reset your environment by reinstalling the packages. Effectively the same thing as removing and recreating the container. You can optionally install edge versions of packages:
make reinstall
make down
Run docker system prune to clean up your docker environment. You'll need to run this periodically as your D4M virtual machine or Linux system gets low on disk space. Otherwise, the environment may fail to start or you may see strange behavior (e.g. processes unexpectedly exiting).
The default D4M resource limits are too low for the developer playground. It's recommended that you allocate at least 8GB of RAM and all CPU cores to D4M. Any unused resources will still be available to OSX.
If you have sufficient system resources, allocate 32GB of RAM to D4M for optimal developer environment performance.
Since the docker-compose.yml maps port 8888 to the host system, you won't be able to run multiple copies unless you change the port in docker-compose.yml.
Additionally, the Makefile automatically prepends your username to container names to avoid namespace collisions.
To make additional host directories available from inside the container, first cp docker-compose.override.yml.example docker-compose.override.yml and then add additional entries to the YAML list in docker-compose.override.yml at the keypath services notebook volumes. For more information see the official Docker documentation for the volumes key.
If you are using a remote server as a Docker host you will need to ensure that your user is a part of the docker group. See the post-install steps for Linux from the official Docker documentation for details.
If you are using a remote server as a Docker host and will be using the Jupyter notebook interface you will need to make whichever port is mapped to the remote host system (8888 by default) accessible by the machine on which you will be running your web browser. The easiest way to do this is via local port forwarding.