Impossible to build notebooks with Scipy requirements on readthedocs.io

[timstaley]

Hi there,

I've been trying to use nbsphinx to build some code-example notebooks on http://readthedocs.io/, but have been running into an issue for codebases which rely on scipy.

You can see a test-case here:
https://github.com/timstaley/nbsphinx-rtd-test
http://readthedocs.org/projects/nbsphinx-rtd-test/builds/

NBSphinx and RTD play well together if your package can be installed into a pure (no-site-packages) virtualenv, but if you try to install Scipy that way you hit an error of error: no lapack/blas resources found. Usually this is no problem - just tick the 'Use system packages' option in RTD settings. The build then goes ahead and almost completes, but fails at the last hurdle; we get to

writing output... [ 50%] demo_notebook
...
File "/home/docs/checkouts/readthedocs.org/user_builds/nbsphinx-rtd-test/envs/latest/local/lib/python2.7/site-packages/pygments/lexers/__init__.py", line 90, in get_lexer_by_name
...
VersionConflict: (setuptools 3.3 (/usr/lib/python2.7/dist-packages),     Requirement.parse('setuptools>=18.5'))

(Full traceback attached:
nbsphinx-rtd-traceback.txt)

So it seems that RTFD has a system-wide ancient version of setuptools which is causing issues for Pygements, when the virtualenv has access to system-packages.

I really wasn't sure where to post this - to your issue tracker or RTFD's - so you may want to bounce it. But it seems like a fairly serious failure case, so I'm hoping you might have a work-around or some suggestions - perhaps trying to build full-dependency notebooks on RTD is a foolish idea from the get-go - should I be trying to pre-generate locally with NBSphinx and just upload the RST's, do you think? Is that something that is already do-able, or I could contribute towards implementing?

I should add, I've actually got this sort of working in an older project - I hacked together a messy conversion script in conf.py, simply shelling out to jupyter nbconvert - which oddly enough seems to work, which is why I was surprised this didn't:
https://github.com/timstaley/voeventdb.remote/blob/master/docs/source/conf.py#L118
Not sure why shelling out works when direct-calls don't, I guess the subshell runs fully in the virtualenv? Probably not of much help for nbsphinx though.

Anyway, many thanks! NBSphinx is great :)

[mgeier]

I don't know why they have such an ancient version of setuptools on RTD, it seems strange. I've asked at readthedocs/readthedocs.org#2429.

You could try switching to Python 3, there they have a newer version: http://packages.readthedocs.io/en/latest/python/python3/.

What happens if you add setuptools>=18.5 to your requirements.txt?

Another idea would be to try using conda. I've not tried that myself, but it's supposed to work: http://read-the-docs.readthedocs.io/en/latest/conda.html.
That's also what the Jupyter people use, e.g. https://github.com/jupyter/notebook/blob/master/docs/environment.yml.

perhaps trying to build full-dependency notebooks on RTD is a foolish idea from the get-go

I would hope not!

The only limit should be the execution time on the server, which I think is currently limited to 15 minutes.

[timstaley]

Funnily enough this seems to have been somehow related to #24, it's certainly fixed by the same workaround of adding 'IPython.sphinxext.ipython_console_highlighting' to the sphinx extensions, cf this commit.

[mgeier]

This is very strange. Thanks for finding a work-around!