Documentation builds fail with Sphinx 3.2.1

[m-aciek]

BPO	45618
Nosy	@ned-deily, @ambv, @JulienPalard, @pablogsal, @hugovk, @miss-islington, @m-aciek, @erlend-aasland
PRs	bpo-45618: Fix documentation build by pinning Docutils version #29230 [3.10] bpo-45618: Fix documentation build by pinning Docutils version to 0.17.1 (GH-29230) #29240 [3.9] bpo-45618: Fix documentation build by pinning Docutils version to 0.17.1 (GH-29230) #29241 [3.8] bpo-45618: Fix documentation build by pinning Docutils version to 0.17.1 (GH-29230) (GH-29241) #29245 bpo-45618: Update Sphinx version used to build the documentation to 4.2.0 #29256 [3.7] bpo-45618: Fix documentation build by pinning Docutils version to 0.17.1 #31476

^{Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.}

Show more details

GitHub fields:

assignee = None
closed_at = <Date 2022-03-28.13:55:34.977>
created_at = <Date 2021-10-26.22:34:59.629>
labels = ['3.8', '3.9', '3.10', 'release-blocker', '3.7', 'type-crash', 'docs']
title = 'Documentation builds fail with Sphinx 3.2.1'
updated_at = <Date 2022-03-28.13:55:34.976>
user = 'https://github.com/m-aciek'

bugs.python.org fields:

activity = <Date 2022-03-28.13:55:34.976>
actor = 'ned.deily'
assignee = 'docs@python'
closed = True
closed_date = <Date 2022-03-28.13:55:34.977>
closer = 'ned.deily'
components = ['Documentation']
creation = <Date 2021-10-26.22:34:59.629>
creator = 'Maciej Olko'
dependencies = []
files = []
hgrepos = []
issue_num = 45618
keywords = ['patch']
message_count = 16.0
messages = ['405060', '405061', '405099', '405101', '405102', '405118', '405125', '405134', '405135', '405229', '405420', '413665', '415979', '416017', '416148', '416150']
nosy_count = 9.0
nosy_names = ['ned.deily', 'docs@python', 'lukasz.langa', 'mdk', 'pablogsal', 'hugovk', 'miss-islington', 'Maciej Olko', 'erlendaasland']
pr_nums = ['29230', '29240', '29241', '29245', '29256', '31476']
priority = 'release blocker'
resolution = 'fixed'
stage = 'resolved'
status = 'closed'
superseder = None
type = 'crash'
url = 'https://bugs.python.org/issue45618'
versions = ['Python 3.7', 'Python 3.8', 'Python 3.9', 'Python 3.10']

[m-aciek]

Since the release of Docutils 0.18 today building Python documentation fails.

In CPython repository we have Sphinx pinned to 3.2.1, which has too loose requirement for Docutils version in install_requires – docutils>=0.12.

Example of failing build: https://github.com/python/python-docs-pl/runs/4009459257.

My suggestion is to pin down docutils in docs requirements.

[m-aciek]

The pin can be removed after bumping Sphinx to at least 3.5.4, which introduced upper limit for docutils version.

sphinx-doc/sphinx@9263eea

[ned-deily]

New changeset bcee6aa by m-aciek in branch 'main':
bpo-45618: Fix documentation build by pinning Docutils version to 0.17.1 (GH-29230)
bcee6aa

[ned-deily]

New changeset 2b7b7c7 by Miss Islington (bot) in branch '3.10':
bpo-45618: Fix documentation build by pinning Docutils version to 0.17.1 (GH-29230) (GH-29240)
2b7b7c7

[ned-deily]

New changeset 30c1f18 by Ned Deily in branch '3.9':
[3.9] bpo-45618: Fix documentation build by pinning Docutils version to 0.17.1 (GH-29230) (GH-29241)
30c1f18

[ned-deily]

All of the open branches are affected by this since we also have to build documentation for security-fix-only branches when releases. PRs for 3.7 and 3.6 forthcoming.

[m-aciek]

For what it's worth I have just successfully built documentation on 3.7 and 3.6 branches locally. It looks like Sphinx 2.3.1 used there doesn't use this part of docutils API that have changed recently.

Sphinx 2.3.1 has requirement of "docutils>=0.12", just like 3.2.1.

We can pin docutils version to current (0.18) "just in case" to prevent a breakage in the future or do nothing – maybe it won't break – maybe the docutils API used is narrow enough.

[m-aciek]

It occurs that the documentation builds without warnings on most recent Sphinx -- version 4.2.0. Probably we should bump the version in 3.11, 3.10 and 3.9 to 4.2.0. That would enable us to remove the pin on docutils.

[m-aciek]

I'm sorry for the confusion. I might have make more checking on my side before proposing a fix.

[ambv]

New changeset f240714 by Ned Deily in branch '3.8':
[3.9] bpo-45618: Fix documentation build by pinning Docutils version to 0.17.1 (GH-29230) (GH-29241) (GH-29245)
f240714

[miss-islington]

New changeset 14a4fce by m-aciek in branch 'main':
bpo-45618: Update Sphinx version used to build the documentation to 4.2.0 (GH-29256)
14a4fce

[ned-deily]

New changeset 7a58509 by Ned Deily in branch '3.7':
bpo-45618: Fix documentation build by pinning Docutils version to 0.17.1 (GH-31476)
7a58509

[m-aciek]

After the Jinja version 3.1.0 release today [1], documentation builds fail for all stable branches. Sphinx in versions pinned on those branches fails with ImportError. [2]

Backporting 14a4fce to 3.10 and 3.9 fixes the problem for them (build is successful with 4.2.0). For 3.8 and 3.7 pinning down Jinja is required (docs don't build smoothly with 4.2.0 for those branches).

Could some core developer help with the backports?

I am starting to work on PRs with pins for 3.8 and 3.7.

[1] https://jinja.palletsprojects.com/en/3.1.x/changes/#version-3-1-0
[2] % sphinx-build --version                                  
Traceback (most recent call last):
  File "…/cpython/Doc/./venv/bin/sphinx-build", line 5, in <module>
    from sphinx.cmd.build import main
  File "…/cpython/Doc/venv/lib/python3.9/site-packages/sphinx/cmd/build.py", line 25, in <module>
    from sphinx.application import Sphinx
  File "…/cpython/Doc/venv/lib/python3.9/site-packages/sphinx/application.py", line 42, in <module>
    from sphinx.registry import SphinxComponentRegistry
  File "…/cpython/Doc/venv/lib/python3.9/site-packages/sphinx/registry.py", line 24, in <module>
    from sphinx.builders import Builder
  File "…/cpython/Doc/venv/lib/python3.9/site-packages/sphinx/builders/__init__.py", line 26, in <module>
    from sphinx.util import import_object, logging, rst, progress_message, status_iterator
  File "…/cpython/Doc/venv/lib/python3.9/site-packages/sphinx/util/rst.py", line 22, in <module>
    from jinja2 import environmentfilter
ImportError: cannot import name 'environmentfilter' from 'jinja2' (…/cpython/Doc/venv/lib/python3.9/site-packages/jinja2/__init__.py)

[ned-deily]

This really should be handled in a new issue since the original fixes for all affected releases are already in the field. Moreover, we should be a bit careful about bumping the required Sphinx version for older releases as this has caused problems for downstream distributors of Python in the past. Can't we just pin the Jinja2 version for current releases?

[m-aciek]

This really should be handled in a new issue since the original fixes for all affected releases are already in the field.

Sorry, I got suggested too much with expected fix which is common for both of those issues. Shall I create a new issue?

Moreover, we should be a bit careful about bumping the required Sphinx version for older releases as this has caused problems for downstream distributors of Python in the past. Can't we just pin the Jinja2 version for current releases?

Didn't know about the problems for distributors. Definitely just pinning Jinja2 is enough.

[hugovk]

> This really should be handled in a new issue since the original fixes for all affected releases are already in the field.

Sorry, I got suggested too much with expected fix which is common for both of those issues. Shall I create a new issue?

I created https://bugs.python.org/issue47138, let's continue there.