sphinx.util.get_module_source fails for builtins

[Chilipp]

Subject: Importing from builtins (or itertools) forces the module file to be marked as changed

Problem

Importing itertools or builtins in a module will always mark this module to be changed when using the BuildEnvironment.get_outdated_files method. This then always triggers a rebuilt of the module, slowing down the build.

Procedure to reproduce the problem

Quick demonstration or the bug source

The source of the problem lies in sphinx.util.get_module_source, you can run

from sphinx.util import get_module_source
print(get_module_source('builtins'), get_module_source('itertools'))
# ('string', None) ('string', None)

which essentially is because builtins (and itertools) do not have an associated __file__ attribute, but a __loader__, and you get

import builtins
print(builtins.__loader__.get_source('builtins'))
# None

The problem here is, that get_module_source returns 'string'.

Example project

To see the result of this, consider the following small project. We create a documentation of the conf.py module using the automodule directive:

conf.py

from itertools import chain
import sys
sys.path.insert(0, '.')

extensions = ['sphinx.ext.autodoc']

contents.rst

API Reference
=============

.. automodule:: conf
    :members:

.. toctree::

and build it with

sphinx-build . html -b html -d doctrees
sphinx-build . latex -b latex -d doctrees

Error logs / results

The first build goes fine, and 1 source file is added.

Running Sphinx v1.8.2
making output directory...
building [mo]: targets for 0 po files that are out of date
building [html]: targets for 1 source files that are out of date
updating environment: 1 added, 0 changed, 0 removed
reading sources... [100%] contents                                                                                                                                                                                                                                              
looking for now-outdated files... none found
pickling environment... done
checking consistency... done
preparing documents... done
writing output... [100%] contents                                                                                                                                                                                                                                               
generating indices... genindex py-modindex
writing additional pages... search
copying static files... done
copying extra files... done
dumping search index in English (code: en) ... done
dumping object inventory... done
build succeeded.

The HTML pages are in html.

The second build marks one file as changed.

Running Sphinx v1.8.2
making output directory...
loading pickled environment... done
building [mo]: targets for 0 po files that are out of date
building [latex]: all documents
updating environment: [] 0 added, 1 changed, 0 removed
reading sources... [100%] contents                                                                                                                                                                                                                                              
looking for now-outdated files... none found
pickling environment... done
checking consistency... done
processing Python.tex...contents 
resolving references...
writing... done
copying TeX support files...
done
build succeeded.

The LaTeX files are in latex.
Run 'make' in that directory to run these through (pdf)latex
(use `make latexpdf' here to do that automatically).

It says updating environment: [] 0 added, 1 changed, 0 removed (which is clearly wrong since none of the files has been changed). For larger projects however, this means that this the documented module is always rebuilt which slows down the built process.

I traced the problem down:

1. The AutodocDirective generates a ModuleDocumenter (see AutodocDirective.run)
2. This then checks the dependencies of the module (in this case, itertools) in it's generate method (see Documenter.generate)
3. The created ModuleAnalyzer uses get_module_source to investigate itertools (see ModuleAnalyzer.for_module)
4. Because of the returned 'string' by get_module_source (see sphinx.util.get_module_source), the filename_set get's a '<string>' added
5. And the AutodocDirective adds this to the dependencies in (see AutodocDirective.run)
6. Finally, when rebuilding the docs, the get_outdated_files method of the BuildEnvironment then looks for a file name called path.join(self.srcdir, '<string>') (see BuildEnvironment.get_outdated_files) which is clearly not existing and hence marks the original module (conf.py) as changed.

Expected results

The builtins module has to be better handled when it comes to the dependencies in a module.

Reproducible project / your project

(see the conf.py and contents.rst shown above)
example.zip

Environment info

• OS: macOS High Sierra, Version 10.13.6
• Python version: 3.7.1 (default, Oct 23 2018, 14:07:42) [Clang 4.0.1 (tags/RELEASE_401/final)]
• Sphinx version: 1.8.2

[tk0miya]

Fixed by #5619.
Thank you for reporting!