readthedocs broken

[jstac]

Hi all,

At the moment the readthedocs documentation seems to be broken:

http://quanteconpy.readthedocs.org/en/latest/index.html
http://quanteconpy.readthedocs.org/en/latest/tools/lqnash.html

@cc7768 @mmcky Any idea what's happened?

Is anyone able to fix this?

[cc7768]

Tried building docs on my own computer and it returns the same thing which means it isn't necessarily a readthedocs issue, the docs just aren't working.

Correction: make html didn't work, but make myhtml did work.

[jstac]

@mmcky Do you have any idea what's happening here?

[mmcky]

@jstac I just had a quick look at the docs section of the repo. After installing numpydoc and the theme from conda - It seems to build ok on my computer. @cc7768 Could this just be a configuration issue with the online readthedocs site?

[mmcky]

In the build output on readthedocs it also looks like there are some ImportErrors.

ImportError: cannot import name ivp
/var/build/user_builds/quanteconpy/checkouts/latest/docs/source/tools/version.rst:4: WARNING: autodoc: failed to import module u'quantecon.version'; the following exception was raised:
Traceback (most recent call last):
  File "/home/docs/checkouts/readthedocs.org/user_builds/quanteconpy/envs/latest/local/lib/python2.7/site-packages/sphinx/ext/autodoc.py", line 335, in import_object
    __import__(self.modname)
  File "/home/docs/checkouts/readthedocs.org/user_builds/quanteconpy/envs/latest/local/lib/python2.7/site-packages/quantecon/__init__.py", line 5, in <module>
    from . import models as models
  File "/home/docs/checkouts/readthedocs.org/user_builds/quanteconpy/envs/latest/local/lib/python2.7/site-packages/quantecon/models/__init__.py", line 12, in <module>
    from . import solow as solow
  File "/home/docs/checkouts/readthedocs.org/user_builds/quanteconpy/envs/latest/local/lib/python2.7/site-packages/quantecon/models/solow/__init__.py", line 9, in <module>
    from . model import Model
  File "/home/docs/checkouts/readthedocs.org/user_builds/quanteconpy/envs/latest/local/lib/python2.7/site-packages/quantecon/models/solow/model.py", line 89, in <module>
    from ... import ivp
ImportError: cannot import name ivp

@cc7768 Do we need to update qe_apidoc.py with the introduction of the solow sub-package in models?

[sglyon]

Is there any way this is a python 2.7 vs 3.4 issue?

[cc7768]

I don't think it is a Python 2.7 vs 3.4 issue. Readthedocs is using Python 2.7 (I tried on 3 as well earlier, but I think that qe_apidoc.py needs to be updated at some point to work with Python 3 syntax -- Everything that is a map object needs to be converted to lists etc...). I haven't had a chance to look at this very closely, but I am a little confused as to why it is happening. If you look at some of the other print out from readthedocs one of the other reasons seems to be our Mock imports, so that is another thing we should look into:

/var/build/user_builds/quanteconpy/checkouts/latest/docs/source/models/asset_pricing.rst:4: WARNING: autodoc: failed to import module u'quantecon.models.asset_pricing'; the following exception was raised:
Traceback (most recent call last):
  File "/home/docs/checkouts/readthedocs.org/user_builds/quanteconpy/envs/latest/local/lib/python2.7/site-packages/sphinx/ext/autodoc.py", line 335, in import_object
    __import__(self.modname)
  File "/home/docs/checkouts/readthedocs.org/user_builds/quanteconpy/envs/latest/local/lib/python2.7/site-packages/quantecon/__init__.py", line 5, in <module>
    from . import models as models
  File "/home/docs/checkouts/readthedocs.org/user_builds/quanteconpy/envs/latest/local/lib/python2.7/site-packages/quantecon/models/__init__.py", line 12, in <module>
    from . import solow as solow
  File "/home/docs/checkouts/readthedocs.org/user_builds/quanteconpy/envs/latest/local/lib/python2.7/site-packages/quantecon/models/solow/__init__.py", line 9, in <module>
    from . model import Model
  File "/home/docs/checkouts/readthedocs.org/user_builds/quanteconpy/envs/latest/local/lib/python2.7/site-packages/quantecon/models/solow/model.py", line 94, in <module>
    A, k, K, L, Y = sym.symbols('A, k, K, L, Y')
TypeError: 'Mock' object is not iterable

[cc7768]

Related: http://stackoverflow.com/questions/8166633/mocking-file-objects-or-iterables-in-python

[mmcky]

@spencerlyon2 @cc7768 Hey Spencer and Chase - what were the reasons for going with a custom qe_apidoc.py rather than using the sphinx-apidoc script? I imagine it is ease of customization, reduce complexity etc. I am just curious as I see that sphinx-apidoc is based on a recursive search of the package tree which would reduce the need to update it based on changes to future structure etc. Cheers ...

[sglyon]

@mmcky we did it this way to have control over how things were organized. This was an issue back when we had a flat structure for all the code, but now that we have subpackages it isn't as big of a deal.

We might want to try using sphinx-apidoc and see if it helps.

[mmcky]

Thanks @spencerlyon2. I might experiment with the sphinx-apidoc coupled with a post processor (if needed) to modify the rst output this weekend. In the meantime is there a way to upload the static pages to readthedocs? I imagine not - but thought I would ask so we can get the documentation site back up.

[mmcky]

I have had a look into the different release versions of QuantEcon and 0.1.7 is the last version that builds successfully on readthedocs. So I will have a look at the diffs between 0.1.7 and 0.1.9

[mmcky]

I have also flipped readthedocs to 0.1.7 for the time being until we patch the issues.

[mmcky]

I have pushed a branch readthedocs-test that removed the solow subpackage from the models package. It seems to render through readthedocs now: http://quanteconpy.readthedocs.org/en/readthedocs-test/. So there seems to be some issues with ImportErrors and mock iterables with this subpackage.

Does this look as expected?

[mmcky]

I have pushed a branch solow-fix to have a look at those Import Errors. I think they were largely due to the way the imports were structured. I have converted them to use relative imports of objects rather than the modules.

from .models import Models

rather than

from . import models as models

readthedocs is now reporting only the mock issue mentioned by @cc7768. Any ideas on how to fix that? https://readthedocs.org/builds/quanteconpy/2692684/

Also tools are now rendering but not the models section in this branch. http://quanteconpy.readthedocs.org/en/solow-fix/

@davidrpugh Would you mind having a look through the proposed changes in the solow-fix branch and let me know if I have broken anything.

[davidrpugh]

@mmcky

I think RTD must be working with an older version of quantecon. Here are the imports from __init__ for quantecon.models.solow currently in master:

"""
models directory imports
objects imported here will live in the `quantecon.models.solow` namespace
"""
__all__ = ['Model', 'CobbDouglasModel', 'CESModel']

from . model import Model
from . import model
from . cobb_douglas import CobbDouglasModel
from . import cobb_douglas
from . ces import CESModel
from . import ces

These import statements are already consistent with the changes that you made. I am now wondering if model, ces, and cobb_douglas sub-modules are included in the namespace? They may also need to be included in __all__.

[mmcky]

@davidrpugh The current readthedocs has been manually set to release 0.1.7 as that is the last release the docs fully compiled through that service. You can check the build messages by looking at the latest version (in the build list) on readthedocs which is built from the master.

I will look at the changes more closely tomorrow - but the branch solow-fix almost compiles on RTD. The only effective change I made was removed the sub-module imports. It still stumbles on a mock iterable error for sympy - working on that issue.

[davidrpugh]

@mmcky

I have never used mocks before when building readthedocs for projects that use numpy, sympy, scipy, etc but rather I have tended to follow the Astropy instructions. I repeat the relevant section...

If you want the documentation for your project to be hosted by ReadTheDocs, then you need to setup
an account there. The following entries in “Advanced Settings” for your package on ReadTheDocs
should work:

• activate Install your project inside a virtualenv using setup.py install
• Requirements file: docs/rtd-pip-requirements
• activate Give the virtual environment access to the global site-packages dir.
• All other settings can stay on their default value.

I don't know if this will help or not...

[mmcky]

@davidrpugh I checked our settings and they are in line with those above. Thanks.

I have tried a few things to resolve this mock issue but it hasn't resolved the readthedocs compilation issue (https://readthedocs.org/builds/quanteconpy/2700947/). The main problem is that when sym.symbols() is called it is returning an iterator. Our mock object doesn't seem to support iterators so I have tried adding one to the Mock() class and I have also tried the newer MagicMock but it seems to have a different interface. I have run out of time today but will revisit this issue later this week.

Ideas: The readthedocs uses from unittest.mock import MagicMock so perhaps we should try compiling using python 3.3 as that was added into the unittest standard library.

In the meantime (@jstac, @davidrpugh ), if we would like readthedocs to compile with the latest I would propose:

1. Temporarily remove the solow subpackage from master and create a new branch solow-rtd-fix to contain the updates needed to resolve this issue. This isn't a very complex operation and I can setup the branches etc. The readthedocs build of the solow-rtd-fix branch can then be checked for compilation errors.
2. Remove the current diagnostic work by deleting the current solow-fix branch
3. Once solow-rtd-fix can successfully compile on readthedocs properly and qe_apidoc.py is also updated to include models.solow we can then merge this back into the master branch.

Note: This won't affect the current pip install from PyPI so anyone using solow will be fine on the current version. We would just need to merge solow back in before the next release.

[jstac]

@mmcky I think this is the best short term solution. We need that documentation functioning and up to date. Please go ahead.

@davidrpugh Sorry to push this back to you but please see if you can resolve the RTD issue with the Solow code.

[davidrpugh]

@jstac @mmcky

I will do my best. @mmcky did you add sympy to the rtd-pip-requirements.txt? It is not included in the current version on master.

[davidrpugh]

@mmcky

If we include a rtd-pip-requirements.txt file and have checked the checked the " Give the virtual environment access to the global site-packages dir" under the advanced settings on RTD project page, it is not clear to me that we even need to bother with mocks. What is your understanding? I have built documentation on RTD for projects using numpy, scipy, sympy, pandas, etc. before and never had to mess with mocks.

[mmcky]

@davidrpugh I played around with the mock modules and added sympy to the environment. It looks like numba definitely requires a mock import (because of some llvm and c components to the package). I think I have struck a balance between the rtd-pip-requirements and mock - as this seems to compile.

http://quanteconpy.readthedocs.org/en/solow-fix/

@jstac Rather than import all these diagnostic commits on the solow-fix branch. I will delta the current master and the final state of solow-fix and implement those in a new branch to merge into master. This doesn't solve the qe_apidoc.py issue - but that doesn't seem to be preventing compilation - it just means the solow documentation isn't listed in readthedocs until that is updated.

If this fails - then I will implement my suggestion above (temporarily remove solow) tomorrow AM to get latest up and running again on readthedocs.

[sglyon]

I've seen some projects using miniconda to install all required components for RTD. That should just download and untar the needed components and make it easy to install numpy/scipy.

Maybe we could try that?

[sglyon]

I may have spoken too soon about miniconda. I know I have seen it on travis.ci, but maybe not on RTD. I'm still looking for an example on RTD

[sglyon]

readthedocs/readthedocs.org#857 (comment)

[mmcky]

The readthedocs settings have been adjusted to allow compilation. See #150
Leaving this issue open to track update for qe_apidoc.py to include solow documentation in the compilation process.

[mmcky]

This issue is now close to being closed. The last step was to update qe_apidoc.py to include the full folder structure. This file has now been updated.

This branch can be viewed here:
http://quanteconpy.readthedocs.org/en/update-doc-api/index.html

I will submit a PR for review to check if everyone is happy with the layout (i.e. menus and navigation etc.) of this update.

[oyamad]

@mmcky One minor issue: The title of the index.html page reads "QuantEcon 0.1.3" (<head> in index.html has an entry <title>QuantEcon documentation &mdash; QuantEcon 0.1.3 documentation</title>), and the same applies to others such as latest. Can the current version number be automatically printed here?

[mmcky]

@oyamad thanks. This has now been updated for PR #164

[mmcky]

readthedocs has been restored. There is one outstanding issue which is to update qe_apidoc.py to incorporate the new sub packages markov and random. Closing this issue as that comment is contained in PR #164