environment.pickle can't be loaded

[thawatchai]

Describe the bug

When I run sphinx-build or make html, I see the following error in the log, and it doesn't do incremental build.

loading pickled environment... failed
failed: [Errno 2] No such file or directory

I try to dig in sphinx code and find that it's from application.py in Sphinx._load_existing_env(), so I try to load the pickle file directly like this.

In [1]: import pickle

In [2]: with open('_build/doctrees/environment.pickle', 'rb') as f:
   ...:     env = pickle.load(f)
   ...: 
---------------------------------------------------------------------------
FileNotFoundError                         Traceback (most recent call last)
Cell In[2], line 2
      1 with open('_build/doctrees/environment.pickle', 'rb') as f:
----> 2     env = pickle.load(f)

File /opt/homebrew/Cellar/python@3.9/3.9.17/Frameworks/Python.framework/Versions/3.9/lib/python3.9/multiprocessing/managers.py:934, in RebuildProxy(func, token, serializer, kwds)
    928         server.id_to_local_proxy_obj[token.id] = \
    929             server.id_to_obj[token.id]
    930 incref = (
    931     kwds.pop('incref', True) and
    932     not getattr(process.current_process(), '_inheriting', False)
    933     )
--> 934 return func(token, serializer, incref=incref, **kwds)

File /opt/homebrew/Cellar/python@3.9/3.9.17/Frameworks/Python.framework/Versions/3.9/lib/python3.9/multiprocessing/managers.py:982, in AutoProxy(token, serializer, manager, authkey, exposed, incref, manager_owned)
    979     authkey = process.current_process().authkey
    981 ProxyType = MakeProxyType('AutoProxy[%s]' % token.typeid, exposed)
--> 982 proxy = ProxyType(token, serializer, manager=manager, authkey=authkey,
    983                   incref=incref, manager_owned=manager_owned)
    984 proxy._isauto = True
    985 return proxy

File /opt/homebrew/Cellar/python@3.9/3.9.17/Frameworks/Python.framework/Versions/3.9/lib/python3.9/multiprocessing/managers.py:784, in BaseProxy.__init__(self, token, serializer, manager, authkey, exposed, incref, manager_owned)
    781     self._authkey = process.current_process().authkey
    783 if incref:
--> 784     self._incref()
    786 util.register_after_fork(self, BaseProxy._after_fork)

File /opt/homebrew/Cellar/python@3.9/3.9.17/Frameworks/Python.framework/Versions/3.9/lib/python3.9/multiprocessing/managers.py:838, in BaseProxy._incref(self)
    835     util.debug('owned_by_manager skipped INCREF of %r', self._token.id)
    836     return
--> 838 conn = self._Client(self._token.address, authkey=self._authkey)
    839 dispatch(conn, None, 'incref', (self._id,))
    840 util.debug('INCREF %r', self._token.id)

File /opt/homebrew/Cellar/python@3.9/3.9.17/Frameworks/Python.framework/Versions/3.9/lib/python3.9/multiprocessing/connection.py:502, in Client(address, family, authkey)
    500     c = PipeClient(address)
    501 else:
--> 502     c = SocketClient(address)
    504 if authkey is not None and not isinstance(authkey, bytes):
    505     raise TypeError('authkey should be a byte string')

File /opt/homebrew/Cellar/python@3.9/3.9.17/Frameworks/Python.framework/Versions/3.9/lib/python3.9/multiprocessing/connection.py:630, in SocketClient(address)
    628 with socket.socket( getattr(socket, family) ) as s:
    629     s.setblocking(True)
--> 630     s.connect(address)
    631     return Connection(s.detach())

FileNotFoundError: [Errno 2] No such file or directory

After searching around, it seems that this issue is related to sphinx pickles objects that can't be pickled (probably multiprocessing.Manager). This starts to get beyond my Python knowledge. Please help.

How to Reproduce

My project is really minimal. At first, I don't even realize that the incremental build doesn't work because the project is so small. I notice this issue after I examine the log closely.

Here is my conf.py.

extensions = [
    "myst_parser",
    "sphinx_sitemap",
    "sphinxcontrib.bibtex",
]

templates_path = ["tools/templates"]
exclude_patterns = [
    "_build",
    "README.md",
    "Thumbs.db",
    ".DS_Store",
    "venv/*",
    "**/tools/*",
]

# language = "th"

bibtex_bibfiles = ["basic-python-programming/book.bib"]

html_baseurl = "https://books.classstart.org/"
# html_theme = "sphinx_rtd_theme"
html_theme = "furo"
html_static_path = ["tools/static"]
html_css_files = ["custom.css"]
html_show_sourcelink = False
html_copy_source = False
html_show_sphinx = False
html_logo = "images/logo.png"
html_favicon = "images/favicon.ico"
html_extra_path = [
    "ads.txt",
    "robots.txt",
]
html_theme_options = {
    "logo_only": True,
    "navigation_depth": 4,
}

sitemap_url_scheme = "{link}"


def setup(app):
    app.add_css_file("custom.css")

Environment Information

Platform:              darwin; (macOS-13.4-arm64-arm-64bit)
Python version:        3.9.17 (main, Jun  6 2023, 14:33:55) 
[Clang 14.0.3 (clang-1403.0.22.14.1)])
Python implementation: CPython
Sphinx version:        7.0.1
Docutils version:      0.20.1
Jinja2 version:        3.1.2
Pygments version:      2.15.1

Sphinx extensions

[
    "myst_parser",
    "sphinx_sitemap",
    "sphinxcontrib.bibtex",
]

Additional context

No response

[thawatchai]

It turned out that this issue causes by the sphinx-sitemap extension, which uses multiprocessing.Manager.

I've reported this issue in jdillard/sphinx-sitemap#67