Sphinx crash: documentation config fix

[isbm]

What does this PR do?

Fixes sphinx doc generator on mocked imports.

What issues does this PR fix or reference?

Previous Behavior

No need to update translations. Skipping...
sphinx-build -b html -d _build/doctrees    . _build/html
Running Sphinx v1.4.6
making output directory...
loading translations [en]... done
loading pickled environment... not yet created
[autosummary] generating autosummary for: contents.rst, faq.rst, glossary.rst,
ref/auth/all/index.rst, ref/auth/all/salt.auth.auto.rst, ref/auth/all/salt.auth.django.rst,
ref/auth/all/salt.auth.keystone.rst, ref/auth/all/salt.auth.ldap.rst, 
ref/auth/all/salt.auth.mysql.rst, ref/auth/all/salt.auth.pam.rst, ...,
topics/tutorials/walkthrough_macosx.rst, topics/tutorials/writing_tests.rst,
topics/using_salt.rst, topics/virt/disk.rst, topics/virt/index.rst, topics/virt/nic.rst,
topics/windows/index.rst, topics/windows/windows-package-manager.rst,
topics/windows/windows-specific-behavior.rst, topics/yaml/index.rst

Exception occurred:
  File "/builds/salt-2016.3.2/salt/config/__init__.py", line 89, in _gather_buffer_space
    return max([total_mem * 0.05, 10 << 20])
TypeError: unsupported operand type(s) for *: 'Mock' and 'float'
The full traceback has been saved in /tmp/sphinx-err-MhWC7u.log, if you want to report the issue to the developers.
Please also report this if it was a user error, so that a better error message can be provided next time.
A bug report can be filed in the tracker at <https://github.com/sphinx-doc/sphinx/issues>. Thanks!
Makefile:73: recipe for target 'html' failed
make: *** [html] Error 1

New Behavior

Problem gone.

Tests written?

No

[isbm]

@cachedout I just stepped on the rake that our package does not generate salt-doc (it is always empty) and because of Sphinx crashes. After further investigation, I found a solution. During merge I found that you've removed psutils with fdbf01d but I am bringing them back and improving Mock() this regard, so we do have the following:

• Better Mock class that is configurable
• Prevented crash while generating docs

What you think? You removed these modules to prevent crash, but this is "shaky" in terms of I would rather keep them mocked properly.

[rallytime]

@whiteinge and @jacobhammons you'll want to know about this also

[whiteinge]

It's a good fix. I vote we merge this now and explore the salt-deps alternative to mocking discussed in #35729 (comment) as a separate experiment off the develop branch.

[rallytime]

Thanks @isbm! We were discussing adding psutil back in internally the other day, so this is good. Thanks for looking @whiteinge.

[cachedout]

I explored this approach already and rejected it for the reasons I outlined here:

#35729 (comment)

Personally, I don't think this is going to be sustainable but we can go with it for now and see how things turn out. :]

[isbm]

@cachedout But now you can do:

mapping = {'this_returns_tuple': tuple(), 
           'this_returns_function': lambda x=None:"Hi there",
           'this_returns_an_int': 0}
mock = Mock(mapping)

This would invalidate your 1st argument. I have an ideas how to return multiple types of values. But that sucks by a design that the function can return different types, at first place...

[cachedout]

In your example this seems to represent an entire python package. My point is that this means that you have to go and mock out every case where a function in psutils would otherwise stacktrace, since each function might return a different type of value. So instead of one mock to psutils, you're now having to mock for each function that exhibits the broken behavior. It just strikes me as something that might grown into an unmanageable beast.

[isbm]

Right (at each function). Problem is that you cannot mock everything at one go, unfortunately. Some values are expected back and if they don't appear as such, we have a problem. I would see two options here:

1. Keep this "unmanageable beast" (it is actually manageable).
2. Suppress this crash inside the sphinx, that is don't blow away everything but only warn and skip.

First approach will require some maintenance of the list, but will make sure docs are within the integrity.
Second approach will tell you that you have an incomplete documentation and never will be.

[cachedout]

I agree that it's manageable at the present time. My concern is about the future. But, as I mentioned earlier, I do think the way to go is to keep this change in and revisit it down the road if it does end up becoming a problem. :]

[isbm]

Well, as of manageability, it doesn't have to be all fat in the doc/conf.py. It can be perfectly in the doc/confmodules.py and keep a simple list of:

# Module description
module = {'method': value}

# Module description
module1 = {'method': value}

...

It looks quite manageable to me even for a big amount of stuff. Problem is that for now I don't see anything else non-intrusive, unless I miss something. 😏 Changing Salt? Not at all. Changing Sphinx? Not either, because the thing needs to know the result, skipping not an option.

At the same time I do not expect we mocking everything. It unlikely will be that big.

[whiteinge]

I agree with both of you. ^_^

Mocking is manageable if it's our only option. But it is a time-sink and sometimes tricky to debug so if we can pull it out all the better. Issue filed here to experiment with an alternative: #35883.

[isbm]

😆

[cachedout]

FWIW, the docs are broken again.

No need to update translations. Skipping...
sphinx-build -b html -d _build/doctrees    . _build/html
Running Sphinx v1.4.5
loading translations [en]... done
loading pickled environment... done
[autosummary] generating autosummary for: contents.rst, faq.rst, glossary.rst, ref/auth/all/index.rst, ref/auth/all/salt.auth.auto.rst, ref/auth/all/salt.auth.django.rst, ref/auth/all/salt.auth.keystone.rst, ref/auth/all/salt.auth.ldap.rst, ref/auth/all/salt.auth.mysql.rst, ref/auth/all/salt.auth.pam.rst, ..., topics/tutorials/walkthrough_macosx.rst, topics/tutorials/writing_tests.rst, topics/using_salt.rst, topics/virt/disk.rst, topics/virt/index.rst, topics/virt/nic.rst, topics/windows/index.rst, topics/windows/windows-package-manager.rst, topics/windows/windows-specific-behavior.rst, topics/yaml/index.rst

Exception occurred:
  File "/home/mp/devel/salt/salt/utils/psutil_compat.py", line 22, in <module>
    if psutil.version_info >= (2, 0):
TypeError: unorderable types: Mock() >= tuple()
The full traceback has been saved in /tmp/sphinx-err-l_uz2ei_.log, if you want to report the issue to the developers.
Please also report this if it was a user error, so that a better error message can be provided next time.
A bug report can be filed in the tracker at <https://github.com/sphinx-doc/sphinx/issues>. Thanks!
make: *** [Makefile:73: html] Error 1

[isbm]

@cachedout hmm, can't reproduce this. Used Sphinx 1.4.3, noticed you used 1.4.5, upgraded to the latest 1.4.6 and still works. Successfully built the docs.

As I think that the /home/mp/devel/... is referring only to a place with sources and a Git repo, what actually branch you use?

[isbm]

Stupid me. Because I have psutil...

[isbm]

OK, disabled psutil, still successfully built docs. Sphinx 1.4.6 version though. Doesn't crash for me.

[cachedout]

@isbm Hmm, OK. Let me poke around and see if I can figure out what's causing my local build to fail. Thanks for having a look here.