Failed to build docs due to missing package, but package listed in setup_requires

[martinpengellyphillips]

I have a package listed in setup_requires in my project's setup.py that is required to build the docs.

Running locally works fine (python setup.py build_sphinx).

Running on readthedocs give the following error:

Extension error:
Could not import extension lowdown (exception: No module named lowdown)

I suspect this is because setup_requires dependencies do not end up in the virtualenv, but is there any way to fix this without doing special logic in setup.py for readthedocs (ie. duplicating setup_requires dependencies into install_requires) or adding an rtd_requirements file to the project?

[gregmuellegger]

Hi, what is the project name that is affected by this? I think as long as a python setup.py install of the project would install the lowdown dependency, then it also should be there in the virtualenv on the build server.

Have you turned on the Install your project inside a virtualenv using setup.py install switch in the advanced settings in the admin panel of your RTD project?

[martinpengellyphillips]

The project is Lucidity

The thing is that python setup.py install won't install dependencies described in setup_requires, which makes sense as those dependencies are only required for building/running setup not execution of the result. Instead it downloads them to a temporary location (usually where the install command is run from).

So my guess is that the environment that the actual sphinx build takes place in on readthedocs is different to the one that install was run in leading to missing dependencies.

See the commented workaround in the setup.py: https://github.com/4degrees/lucidity/blob/master/setup.py#L48 If you fork and remove those lines you should be able to reproduce the issue.

[gregmuellegger]

Ok I see. That is an issue on readthedocs end. But a minor IMO.

Readthedocs has also the ability to install requirements from a requirements.txt file. Maybe that is more the place where you want to store the dependency for now. In that way at least the users of Lucidity won't install lowdown in their environments when running python setup.py install. I know it's a little workaround but maybe sufficient for now.

We are currently implementing a new doc build system so it's unlikely that we get around to fix this issue before we have the new system in place.

[gregmuellegger]

I had another look at this and read the documentation for setup_requires. It states:

setup_requires

A string or list of strings specifying what other distributions need to be present in order for the setup script to run. setuptools will attempt to obtain these (even going so far as to download them using EasyInstall) before processing the rest of the setup script or commands. This argument is needed if you are using distutils extensions as part of your build process; for example, extensions that process setup() arguments and turn them into EGG-INFO metadata files.

(Note: projects listed in setup_requires will NOT be automatically installed on the system where the setup script is being run. They are simply downloaded to the ./.eggs directory if they’re not locally available already. If you want them to be installed, as well as being available when the setup script is run, you should add them to install_requires and setup_requires.)

So I think putting the requirements for the sphinx build there is not appropriate as building the docs is not some part of the setup process of a project. I know that you are currently using a python setup.py command for doing that but it's unlikely that readthedocs will ever adapt that logic.

RTD is building the docs by calling the sphinx-build script. That is the same logic you are using when starting out with Sphinx (Sphinx's make html uses this command internally). So I think you will need to put your requirements for the sphinx build into your requirements.txt file.

[martinpengellyphillips]

My perspective differs - I don't see why you would want to 'install' Sphinx et al when they are never used at runtime (this also pollutes the target environment). setup_requires seems the appropriate place for these build time dependencies and I don't see a disconnect with the docs you refer to in typical usage. Of course the readthedocs case is different where this is exactly what you want to do.

Having said that, I don't think this stuff is particularly well defined in the Python world at present so am fine with your conclusion.

I'll keep the workaround in setup.py rather than add a requirements.txt though as I feel adding that file could be misleading. My understanding of the use case for requirements.txt is for repeatable deployment of a specific setup such as deploying a tested version of an application across a company, but I could be off the mark here.

Thanks for looking into it!