Support conda for builds

[mrocklin]

Read the Docs Conda Support

This will add the ability to generate documentation with conda environments on Read the Docs. This is mainly useful for libraries with large C dependencies, including many packages in the Scentific Python ecosystem.

Task List

• Add Project option to use conda
• Abstract the "environment path" in the builders to respect conda & virtualenv
• Ensure cleanup of conda environment in wiping & other env delete scenarios
• Ensure Python 2 & 3 support
• Write tests for proper conda environment setup
• Document conda support
• Write a topical guide for Choosing a Build Environment
• Optional: Investigate pruning

Abilities

You will be able to specify a conda environment.yml file,
and Read the Docs will install these dependencies in your build environment.

Considerations

Read the Docs will keep seperate virtualenv & conda directories:

• /user_builds//env for pip
• /user_builds//conda for conda

Users will be able to define a way to install packages for a project:

• None
• Pip
• Conda

Read the Docs will need to change it's build code so that we don't hard-code virtualenv paths. We'll need to vary our environment creation, as well as bin path's for executables, based on the backend environment.

The other main thing is that we'll also need to install Sphinx & other build dependencies into the conda environment. We will continue to use pip for this, and it should be transparent, other than using the pip executable in the conda environment instead of the virtualenv.

It should also be noted that miniconda has a different install process from Python 2 and 3 -- also they recommend installing it from their bash scripts instead of pip. I hope that we will be able to use pip, as it will simplify our installation, and won't require an update to a bash script on version upgrades. We will have to see if we hit issues in testing.

Cleanup

Read the Docs will manage conda environment deletion on the removal of a project or version.

Documentation

We will need to add information about conda support to our documentation. We might want to add a topic guide around installing requirements, along with adding a specific reference for how to use & enable conda support.

Sponsorship

This work is being funded by Clinical Graphics -- many thanks for their support of Open Source.

[ericholscher]

Is there a reason you can't install conda-like packages with pip? I believe pip is the default installer for Python, and is likely the only thing we will support.

[mrocklin]

Pip tends not to work well with packages that have non-Python dependencies. This includes a lot of the Numeric/Scientific Python stack (e.g. Pandas.) This community is a decently sized chunk of the Python ecosystem, and would definitely appreciate better support from RTD (which, btw, I <3, thanks!)

In regards to default installers you may find it interesting that conda was recently blessed and brought in under the Python Packaging Authority (cc @ncoghlan).

[ncoghlan]

conda is actually still its own org (since it isn't Python specific), but
we do recommend it when Python folks need a cross platform package manager
that can handle the Python runtime and arbitrary external dependencies. By
contrast, the PyPA toolset focuses specifically on Python packages
(including C extensions), including playing nice with redistributors like
conda and the Linux distros.

[ericholscher]

Interesting. So it would be a way to install binaries onto the build server? Does it work with virtualenvs, or is it installing things system wide?

[mrocklin]

Exactly.

It has its own system for virtual environments that relies on linking many packages into environments. Interestingly, because it packages binaries Python is itself just a package, so it's easy to do things like quickly spin up two environments in Python 2 and 3 for simultaneous testing. Many who use it (myself included) vastly prefer it to virtualenv, but that's subjective.

@asmeurer might have more information.

[ncoghlan]

pip/virtualenv & conda work at different levels.

The PyPA tools are designed to work within a larger system that provides the Python runtime and any external dependencies. That may be a Linux distro package manager, something like homebrew on Mac OS X, or just downloading and running a binary installer from python.org.

By contrast, conda is such a larger system - rather than being Python specific, it's a full "cross platform platform", designed to manage arbitrary binaries, including Python runtimes and external dependencies. This means it doesn't integrate with other environments the way pip can, but it also means it can be used to manage components where pip will fail completely.

[ncoghlan]

Oh, and to answer the "is conda system wide" question, no it isn't. It's designed to be run as an ordinary user, creating installation environments in their own directory space without needing root access.

[shoyer]

👍 this would be a fantastic addition.

I figured out how to make the baseline scientific packages (numpy, scipy) + pip to work together to build everything I need for my docs, but it was a real guessing game to figure out which combinations of pip packages could safely install together.

I actually tagged and released v0.1 of my package before I realized that I would not be able to build it from scratch on RTD, because it only worked when I changed the requirements.txt file incrementally. This is obviously somewhat unfortunate and is actually pretty typical for the problems that arise when using pip to install scientific python libraries.

[tritemio]

I agree it would be a great addition. As a matter of fact I'm struggling with installing numpy to build the docs for my project. @shoyer how did you solved the problem?

[shoyer]

@tritemio The trick is to give the virtualenv in which you build your docs access to the global site-packages directory -- see Advanced Settings > Use system packages. RTD has numpy 1.8, scipy and matplotlib installed system wide. I setup my conf.py to print out the versions when building the docs: https://github.com/xray/xray/blob/v0.2/doc/conf.py

As for testing, to ensure that you can build your docs from scratch in a new virtualenv (each version of the docs gets its own virtualenv), try deleting the build environment:
http://read-the-docs.readthedocs.org/en/latest/builds.html#deleting-a-stale-or-broken-build-environment

[tritemio]

@shoyer thanks! Your suggestions are narrowing down my problems, hope to fix them soon ...

[faph]

+1

I'd love to have conda support in RTD. In the same way that Travis CI does it. http://conda.pydata.org/docs/travis.html

[tswicegood]

FWIW, if RTD used buildpacks like Heroku and Cloud Foundry, there's a conda-buildpack that can detect a Conda environment.yml file and spin that up. That spec supports creating environments that have both Conda and pip packages in them. If nothing else, its conda_compile script might serve as a good reference is someone wants to take a stab at implementing this in RTD.

[zgrnk]

I am currently working on a project which uses numba and am trying to upload the project onto RTD. This cannot currently be done as it requires the llvm compiler. Is it possibilt for RTD either to install llvm and include it in the system packages under the Advanced Settings, or add support for conda?

[ncoghlan]

Using buildpacks or a container tech like Docker to do builds would require fundamentally redesigning the way ReadTheDocs works. On the other hand, that might not be a bad idea at some point, especially as Docker based public cloud services with a grants program for open source projects come online

Disclosure: I work for Red Hat, OpenShift Online has a grants program that includes open source projects in its scope, our next generation architecture is based on Docker & Kubernetes, and I personally believe that hosting a valuable service like RTFD would be a great way for us to support the community. So while porting to the current OpenShift architecture likely wouldn't make sense, porting to Docker/Kubernetes would open up both Google Container Engine and a future version of OpenShift Online as hosting options.

[astrojuanlu]

👍 for the addition of conda, I am interested in hosting a package depending on numba too.

[pitrou]

While conda isn't supported, I've tried at least to disable the setup.py running and I'm having this weird error:
#1240

[cdeil]

👍 I know a few packages that need a more recent scipy or matplotlib for their docs build and pip install on readthedocs fails ... conda would be a very nice solution!

[chebee7i]

Practically, won't this be necessary if you want to build older versions of your documentation that require older versions of NumPy, matplotlib, possibly with different APIs from whatever version of NumPy is installed system-wide?

[shoyer]

@chebee7i practically speaking, matplotlib and numpy have strong backwards compatibility guarantees, so I'm not too worried about API changes for them. Though matplotlib has been talking about a 2.0 release with a new default colormap...

[chebee7i]

@shoyer but functionality does change between versions and this can cause the documentation to be wrong, especially if you rely on buildtime-generated documentation through the sphinxext IPython directive. And I am thinking much more generally than matplotlib and numpy, but even extending to just pandas reveals very recent backwards compatibility changes.

[jakirkham]

+1 for conda support.

[Carreau]

+1, i'm currently trying to update old docs that use both Python, R and rpy2, I can almost trivially have everything working fine in a conda 2.7 env.

[ericholscher]

We aren't running conda in an interactive session, so I'm not sure how we could run activate. If there are environment variables that should be supported, we could set them, but we're running these commands independently, so the variables set in the activate script wouldn't apply.

The code is here: https://github.com/rtfd/readthedocs.org/blob/master/readthedocs/doc_builder/python_environments.py#L163

[jakirkham]

We aren't running conda in an interactive session, so I'm not sure how we could "call" activate.

Yeah, I wouldn't be surprised if this is a problem. The activate script is a long-ish bash script that needs to be sourced. Doesn't seem like that would mesh well with what you are doing. Fortunately, it doesn't seem to matter.

If there are environment variables that should be supported, we could set them, but we're running these commands independently, so the variables set in the activate script wouldn't apply.

Right, so if we can just tack them onto whatever environment that is used when shelling out, I think this would be fine.

The code is here: https://github.com/rtfd/readthedocs.org/blob/master/readthedocs/doc_builder/python_environments.py#L163

Thanks for the link. I have been looking at the code. I had a question that I put on the merged PR.

[pitrou]

For the record, here is a diff of environment variables when I run activate here:

+PYTHONNOUSERSITE=1
-PATH=/home/antoine/.local/bin:/usr/local/cuda/bin:/home/antoine/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/usr/local/games
+PATH=/home/antoine/35/bin:/home/antoine/.local/bin:/usr/local/cuda/bin:/home/antoine/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/usr/local/games
+CONDA_ENV_PATH=/home/antoine/35
+CONDA_DEFAULT_ENV=/home/antoine/35

As you see there's not much to it. The most important is probably the appending of the conda environment's bin directory to the PATH.

[jakirkham]

Interesting, I don't get this PYTHONNOUSERSITE. Seems like a useful add in general. Not sure if it matters here. Which version of conda are you on?

[pitrou]

Oh, forget it. PYTHONNOUSERSITE is from my own activate wrapper :-)

[jakirkham]

Ah, ok. In any event, I think we can safely ignore it here as there should be only this one Python install that we are worried about.

[ericholscher]

Should be easy enough to support the conda PATH, as we already have the bin_path argument to our run call.

[jakirkham]

I could be wrong, but I think we want to be able to pass environment variables as a kwarg to something like this ( https://github.com/rtfd/readthedocs.org/blob/aba714e82d218d60773955aec62a3df74173348d/readthedocs/doc_builder/backends/sphinx.py#L156 ). Does run permit that?

[jakirkham]

So, we have to change the environment of a BuildCommand then, yes? Maybe we can just add these to the environment before the build is called?

[jakirkham]

So, I tried my hand at a simpler PR ( #1919 ). From what I have gleaned from my brief look at the code here (not an expert), this looks like a workable solution.

It probably could use a test or two, but I am a bit unfamiliar with the test suite here and will need guidance for that. Though, I imagine doing something like asserting the variables are there and checking the output of conda info would be a good start.

[jakirkham]

Turns out this PR ( #1924 ) ends up being a much better way to go about it.

[jakirkham]

So, I think the active environment issue is fixed thanks to @ericholscher and @agjohnson, if people want to try.

[mrocklin]

Verified that this works well for me. Thanks all!

[ericholscher]

Great :)

If there are other issues, please create new tickets for them, as this one is quite old and the main issues seem to be resolved.

Woo!

[astrojuanlu]

I just built the docs for poliastro in rtd and it works like a charm 😄 Thanks to @mrocklin and Clinical Graphics for this!

[astrojuanlu]

Just a quick notice: right now RTD uses docutils 0.11, which is not yet available for Python 3.5 for linux.

http://readthedocs.org/projects/poliastro/builds/3655474/

Hint: the following packages conflict with each other:
  - docutils ==0.11
  - python 3.5*

Packages: https://anaconda.org/anaconda/docutils/files?version=0.11

[astrojuanlu]

(Sorry, my above comment was already reported as #1901)

[jakirkham]

@Juanlu001, see ( #1933 ).

[ericholscher]

#1947 is the fix for that issue. I will get it merged & deployed today.

[tritemio]

I tried the using conda environment with fretbursts to see if I can get rid of the mocking layer.

However, building the docs fails on RTD:

http://readthedocs.org/projects/fretbursts/builds/3813468/

unless I reintroduce the mocking for scipy:

http://readthedocs.org/projects/fretbursts/builds/3813474/

[agjohnson]

@tritemio it would be best if you opened your issue as a bug, we won't be able to track it here. This was a feature addition that has been in production for a few months now.

Closing this issue. Please address bugs with additional tickets.

[tritemio]

@agjohnson, it makes sense. I opened a new issue #2074