[CRITICAL] Sphinx doesn't work with rc releases of docutils due to version parsing

[userzimmermann]

As you can see here: https://circleci.com/gh/coala/coala/12852?utm_campaign=vcs-integration-link&utm_medium=referral&utm_source=github-build-link

sphinx-build -b html -d _build/doctrees  -W . _build/html
Traceback (most recent call last):
  File "/home/ubuntu/.pyenv/versions/3.5.1/bin/sphinx-build", line 11, in <module>
    sys.exit(main())
  File "/home/ubuntu/.pyenv/versions/3.5.1/lib/python3.5/site-packages/sphinx/__init__.py", line 66, in main
    sys.exit(build_main(argv))
  File "/home/ubuntu/.pyenv/versions/3.5.1/lib/python3.5/site-packages/sphinx/__init__.py", line 76, in build_main
    from sphinx import cmdline
  File "/home/ubuntu/.pyenv/versions/3.5.1/lib/python3.5/site-packages/sphinx/cmdline.py", line 23, in <module>
    from sphinx.application import Sphinx
  File "/home/ubuntu/.pyenv/versions/3.5.1/lib/python3.5/site-packages/sphinx/application.py", line 36, in <module>
    from sphinx.domains.std import GenericObject, Target, StandardDomain
  File "/home/ubuntu/.pyenv/versions/3.5.1/lib/python3.5/site-packages/sphinx/domains/std.py", line 27, in <module>
    from sphinx.util.compat import Directive
  File "/home/ubuntu/.pyenv/versions/3.5.1/lib/python3.5/site-packages/sphinx/util/compat.py", line 19, in <module>
    docutils_version = tuple(int(x) for x in _du_version.split('.')[:2])
  File "/home/ubuntu/.pyenv/versions/3.5.1/lib/python3.5/site-packages/sphinx/util/compat.py", line 19, in <genexpr>
    docutils_version = tuple(int(x) for x in _du_version.split('.')[:2])
ValueError: invalid literal for int() with base 10: '14rc1'

cc @sils @jayvdb @thisch

[boumarc1]

The issue also happens on Python2.7

[jfbu]

Sorry for inconvenience. Already reported at #3821, #3838, and is fixed at ff1adc7. Leaving the issue open for a while in order for people to find it more easily.

[jfbu]

Updating to Sphinx 1.6.2 fixes the issue to the extent that sphinx.util.compat.docutils_version got deprecated already at 1.6, and that the whole module will be removed at 1.7. So Sphinx core is not affected if you update to 1.6.2, but extensions may be, and then one needs to wait for 1.6.3.

Summary: rc Docutils version string breaks all Sphinx earlier than 1.6.2, but is ok with 1.6.2 to the extent no extension uses depreacted sphinx.util.compat module, and all will be fine at 1.6.3.

[userzimmermann]

@jfbu Thanks! I also found ff1adc7 meanwhile :) How long to wait for 1.6.3? Or can you release some 1.6.2.post in between?

[jfbu]

The 1.6.2 was just released last Sunday. What is your Sphinx version ?

[userzimmermann]

@jfbu 1.6.2 :)

[jfbu]

ah, so something beyond core sphinx is at work in your project. Would some monkey-patch in conf.py be acceptable ? (notwithstanding date of 1.6.3 release)

[tk0miya]

  File "/home/ubuntu/.pyenv/versions/3.5.1/lib/python3.5/site-packages/sphinx/domains/std.py", line 27, in <module>
    from sphinx.util.compat import Directive

This is not a part of v1.6.2. Please check your version again.

This is line 27 of sphinx.domains.std on 1.6.2:
https://github.com/sphinx-doc/sphinx/blob/1.6.2/sphinx/domains/std.py#L27

[jfbu]

with this at top of conf.py you may fix your project

from docutils import __version__ as _du_version
if _du_version == '0.14rc1':
    _du_version = '0.14'
import docutils
docutils.__version__ = _du_version

as a temporary fix. But best would be already to update to Sphinx 1.6.2. ;-)

(edited:)

1.6.2's compatibility with Docutils pre-release version string is only partial, as deprecated module sphinx.util.compat is broken by it and only fixed at 1.6.3.

I tested the above conf.py hack with 1.6.2 and extension sphinx-argparse for example as it imports sphinx.util.compat.

[userzimmermann]

@tk0miya @jfbu I didn't check the sphinx version in that CircleCI test explicitly. I just assumed that it must be latest because we are upgrading all dependencies on each run and also got docutils-0.14rc1. And I checked locally if latest sphinx-1.6.2 still has that issue in sphinx.util.compat... and it is there...

[jfbu]

@userzimmermann yes the issue is there at 1.6.2 but no part of Sphinx imports anymore the module. Surely, some extension possible does. Whih ones are used in your project?

[userzimmermann]

@jfbu Only sphinx-argparse as far as I can see...

[jfbu]

coincidentally this is precisely the extension I now tested with ;-) that's funny!

[jfbu]

see comment above which I was just editing ;-)

[userzimmermann]

@jfbu And I just also saw that we have to increase the sphinx dependency version in our docs-requirements... OOPS! That's also... kind of... funny :D

[tk0miya]

Yes, sphinx.util.compat of 1.6.2 has an issue like you said. But it is not used from sphinx-core because it is already deprecated. It might be only used by some extension.

It seems the Sphinx on CircleCI is not 1.6 series. As a quick look, it might be 1.5.x series. So your case is simply a bug of old Sphinx. So I recommend you to update it to 1.6 series. (or pin the docutils to 0.13.1.) I believe it resolves your case.

[userzimmermann]

@tk0miya @jfbu Thanks again! And according to my #3846 (comment) before... Sorry for the wasted time ;)

[userzimmermann]

@jfbu @tk0miya Maybe the time was actually not wasted ;) I updated sphinx requirement to 1.6.2, removed our docutils downgrade workaround, but look at the Travis and Circle jobs of coala/coala#4334

Running Sphinx v1.6.2
making output directory...
Exception occurred:
  File "/home/travis/virtualenv/python3.4.2/lib/python3.4/site-packages/sphinx/util/compat.py", line 21, in <genexpr>
    docutils_version = tuple(int(x) for x in _du_version.split('.')[:2])
ValueError: invalid literal for int() with base 10: '14rc1'

The only plugin used is still sphinx-argparse, updated to 0.2.0 in our requirements

[jfbu]

@userzimmermann Thanks for reporting, indeed sphinx-argparse uses deprecated module sphinx.util.compat and this breaks builds. Fix at Sphinx will be released at 1.6.3. All extensions still using sphinx.util.compat at 1.7 will be broken, because the module is deprecated as indicated in CHANGES: 1.6b1, 1.7. At my comment above I indicated a temporary fix in conf.py for your project. It worked for me.

(I have re-worded my earlier comment so as not to suggest upgrading to 1.6.2 fixes completely the issue. Only 1.6.3 will)

[userzimmermann]

@jfbu I did the workaround in conf.py that way:

import docutils
from packaging.version import Version

# temporal workaround for deprecated sphinx.util.compat causing problems with
# rc releases of docutils
docutils.__version__ = Version(docutils.__version__).base_version

Now everything works fine :)

Many thanks again!

[userzimmermann]

I quickly created a PR for sphinx-argparse:

alex-rudakov/sphinx-argparse#66

[tk0miya]

What is current status of this issue? Is this closable?

[jfbu]

This is a duplicate which was kept open to avoid new ones popping up until Docutils 0.14 is released and the problem disappears on older Sphinx'es ;-)

[tk0miya]

Okay. I'll keep this opened for a while.

[tk0miya]

I'm closing this now. It's time to ship next stable release :-)