ENH? Add Dockerfile for development inside a container

[AustinRochford]

Not sure if this will be useful for others, but I like to be able to use my development code in a notebook running inside a Linux container. Here is the Dockerfile I use to do so. Even if we don't want to merge this, it could be useful for others who like that style of development.

To build the container image

docker build -t pymc3 .

To launch a container

docker run -d -p 8888:8888 -v $PYMC3_DIR:/home/jovyan/pymc3 -v $NOTEBOOK_DIR:/home/jovyan/work/ --name pymc3 pymc3

where $PYMC3_DIR is the root of your PyMC3 source directory and $NOTEBOOK_DIR is the directory where notebooks will be persisted. After this command, there will be a Jupyter notebook server running available at localhost:8888.

[ColCarroll]

This is great -- I'm playing around with it locally.

One thought is that it would be quite helpful to have a docker image that mirrors the travis build for troubleshooting. Does the scipy base image do any optimizations on the install? If not, we might use an ubuntu base image, then use the travis install scripts (and maybe a requirements-dev.txt for convenience).

[AustinRochford]

@ColCarroll that's a great idea

[ColCarroll]

Just to be more portable, can this be ENV PYTHONPATH $PYTHONPATH:"$HOME"/pymc3 ?

[ColCarroll]

Using the notebook works great -- we may want to add a startup script to the /scripts folder, so I'm not looking up this PR every time.

One problem I got after running docker exec -it pymc3 bash and then opening an ipython shell was that pyqt was not installed. Looks like pyqt was removed from this base image (https://github.com/jupyter/docker-stacks/blob/master/scipy-notebook/Dockerfile), and installing it does some wacky things if you try to work inside the container (this is the issue I ran into https://groups.google.com/forum/#!topic/etetoolkit/4SrZk5fRtCM)

This should probably be a new pymc3 issue (I should be able to import pymc3 without an X server).

In [1]: import pymc3
enabling matplotlib
/opt/conda/lib/python3.5/site-packages/matplotlib/font_manager.py:273: UserWarning: Matplotlib is building the font cache using fc-list. This may take a moment.
  warnings.warn('Matplotlib is building the font cache using fc-list. This may take a moment.')
/opt/conda/lib/python3.5/site-packages/matplotlib/font_manager.py:273: UserWarning: Matplotlib is building the font cache using fc-list. This may take a moment.
  warnings.warn('Matplotlib is building the font cache using fc-list. This may take a moment.')

In [2]:  : cannot connect to X server  # exits ipython

[AustinRochford]

@ColCarroll that is odd, I have no such problem importing pymc3 from ipython in a terminal.

[ColCarroll]

Are you running on linux? It may be an osx thing... I'm also surprised you did not need to add pyqt=4.11.4 to the conda install line

[AustinRochford]

I am running on OS X; weird.

[ColCarroll]

Looks fine to me, if another OS X user can have it run.

I'm actually getting a few test failures running the tests inside the container:
a bunch are because we use scipy.stats.skewnorm, so scipy has to be version 0.18.1 (the image has 0.17.1). The rest are RuntimeError: Invalid DISPLAY variable, which I'd guess is related to my pyqt woes (but it'd be helpful if the test suite passed for someone before landing).

I wonder if instead of wget-ing requirements.txt, you wget create_testenv.sh, and we can have that script accept a --global flag to install everything globally?

[twiecki]

Just noting (without having reviewed it) that I generally support adding a Dockerfile to the repo and publishing it on dockerhub.

[springcoil]

I think a dockerfile is a good addition. Maybe add some documentation for
this.

On Sun, Oct 23, 2016 at 2:13 PM, Austin Rochford notifications@github.com
wrote:

Not sure if this will be useful for others, but I like to be able to use
my development code in a notebook running inside a Linux container. Here is
the Dockerfile I use to do so. Even if we don't want to merge this, it
could be useful for others who like that style of development.

To build the container image

docker build -t pymc3 .

To launch a container

docker run -d -p 8888:8888 -v $PYMC3_DIR:/home/jovyan/pymc3 -v $NOTEBOOK_DIR:/home/jovyan/work/ --name pymc3 pymc3

where $PYMC3_DIR is the root of your PyMC3 source directory and
$NOTEBOOK_DIR is the directory where notebooks will be persisted. After
this command, there will be a Jupyter notebook server running available at

localhost:8888.

You can view, comment on, or merge this pull request online at:

#1473
Commit Summary

• Add Dockerfile for development inside a container

File Changes

• A Dockerfile
  https://github.com/pymc-devs/pymc3/pull/1473/files#diff-0 (11)

Patch Links:

• https://github.com/pymc-devs/pymc3/pull/1473.patch
• https://github.com/pymc-devs/pymc3/pull/1473.diff

—
You are receiving this because you are subscribed to this thread.
Reply to this email directly, view it on GitHub
#1473, or mute the thread
https://github.com/notifications/unsubscribe-auth/AA8DiOSUw6jaUu9OnV31k-vr2MeXiI4-ks5q211igaJpZM4KeIM7
.

Peadar Coyle
Skype: springcoilarch
www.twitter.com/springcoil
peadarcoyle.wordpress.com

[AustinRochford]

@ColCarroll I think that's a good idea, and the tests should definitely pass in the container before merging this.

Do you have a suggestion for a base image that is close to what Travis uses? (I am quite a Docker novice, and don't really know exactly where to look)

[ColCarroll]

What if you used continuumio/minconda3 as the base image, then ran scripts/create_testenv.sh instead of requirements.txt? I'll make a PR that updates create_testenv.sh to install globally.

As an aside, it looks like there are two copies of the actual requirements -- one in create_testenv.sh, one in requirements.txt (and a few more in setup.py). Would be nice to streamline that, but I don't know much about conda environments.

[twiecki]

This probably needs a rebase.

[ColCarroll]

It looks like the tests were getting messed up by that same 'theano' problem that has been fixed (this should not effect the build).

@AustinRochford I grabbed the changes from your branch and got tests passing inside the container. Ended up using the jupyter/minimal-notebook base image, and doing some environment variable magic with the matplotlib backend to get my tests to pass. Also updated CONTRIBUTING.md to give some instructions.

Up to you whether you want to update this PR to incorporate some of those changes, or I can make a new PR from my branch.

[AustinRochford]

@ColCarroll thanks for picking this up as I've been busy elsewhere the last few weeks. If you've got the tests working, I'd say go ahead an merge your work and I will close this.