“'myst' reference target not found: b.md#anchor” when building in parallel

[andersk]

Describe the problem

I configured myst_heading_anchors = 2 as documented, but anchor links are giving me this warning if I enable parallelism (sphinx-build -W -j N for any N other than 1):

Warning, treated as error:
/tmp/myst-anchor-test/docs/dir/a.md:2:'myst' reference target not found: b.md#anchor

When building without parallelism, there is no warning and the anchor works fine.

Link to your repository or website

https://github.com/andersk/myst-anchor-test

Steps to reproduce

$ pip install Sphinx myst-parser
…
Successfully installed Jinja2-3.0.1 MarkupSafe-2.0.1 Pygments-2.9.0 Sphinx-4.1.2 alabaster-0.7.12 attrs-21.2.0 babel-2.9.1 certifi-2021.5.30 charset-normalizer-2.0.4 docutils-0.17.1 idna-3.2 imagesize-1.2.0 markdown-it-py-1.1.0 mdit-py-plugins-0.2.8 myst-parser-0.15.1 packaging-21.0 pyparsing-2.4.7 pytz-2021.1 pyyaml-5.4.1 requests-2.26.0 snowballstemmer-2.1.0 sphinxcontrib-applehelp-1.0.2 sphinxcontrib-devhelp-1.0.2 sphinxcontrib-htmlhelp-2.0.0 sphinxcontrib-jsmath-1.0.1 sphinxcontrib-qthelp-1.0.3 sphinxcontrib-serializinghtml-1.1.5 urllib3-1.26.6
$ git clone https://github.com/andersk/myst-anchor-test.git
$ cd myst-anchor-test
$ sphinx-build -W -j2 docs build
Running Sphinx v4.1.2
…
Warning, treated as error:
/tmp/myst-anchor-test/docs/dir/a.md:2:'myst' reference target not found: b.md#anchor
$ sphinx-build -W docs build
Running Sphinx v4.1.2
build succeeded.

The HTML pages are in build.

If you want to reproduce again, rm -rf build.

The version of Python you're using

3.9.6

Your operating system

NixOS 21.11

Versions of your packages

$ pip freeze
alabaster==0.7.12
attrs==21.2.0
Babel==2.9.1
certifi==2021.5.30
charset-normalizer==2.0.4
docutils==0.17.1
idna==3.2
imagesize==1.2.0
Jinja2==3.0.1
markdown-it-py==1.1.0
MarkupSafe==2.0.1
mdit-py-plugins==0.2.8
myst-parser==0.15.1
packaging==21.0
Pygments==2.9.0
pyparsing==2.4.7
pytz==2021.1
PyYAML==5.4.1
requests==2.26.0
snowballstemmer==2.1.0
Sphinx==4.1.2
sphinxcontrib-applehelp==1.0.2
sphinxcontrib-devhelp==1.0.2
sphinxcontrib-htmlhelp==2.0.0
sphinxcontrib-jsmath==1.0.1
sphinxcontrib-qthelp==1.0.3
sphinxcontrib-serializinghtml==1.1.5
urllib3==1.26.6

Additional context

No response

[welcome]

Thanks for opening your first issue here! Engagement like this is essential for open source projects! 🤗

If you haven't done so already, check out EBP's Code of Conduct. Also, please try to follow the issue template as it helps other community members to contribute more effectively.

If your issue is a feature request, others may react to it, to raise its prominence (see Feature Voting).

Welcome to the EBP community! 🎉

[chrisjsewell]

thanks, I'll try to have a look soon(ish) 😬

[andersk]

Not fixed in 90c98aa (#467), despite #467 being marked as related.

[chrisjsewell]

Not fixed in 90c98aa (#467), despite #467 being marked as related.

Well that's why it as related and not fixed 😅

I had a brief look in that PR, but could not see anything obviously wrong with the myst-parser implementation.

The issue is possibly to do with the merging of global environments from the parallel processes, after reading documents, here: https://github.com/sphinx-doc/sphinx/blob/a284aa65717f7a0c494c4c0859d2158094284ad1/sphinx/environment/__init__.py#L316-L317

The myst-parser renderer add to the standard domain here:

MyST-Parser/myst_parser/sphinx_renderer.py

Lines 155 to 160 in 0846a1f

	domain.anonlabels[doc_slug] = self.doc_env.docname, labelid
	domain.labels[doc_slug] = (
	self.doc_env.docname,
	labelid,
	clean_astext(section[0]),
	)

Then sphinx should merge this into the final global environment here: https://github.com/sphinx-doc/sphinx/blob/a284aa65717f7a0c494c4c0859d2158094284ad1/sphinx/domains/std.py#L727-L743

The data should then be available for the post-transform (performed after all documents have been read), here:

MyST-Parser/myst_parser/myst_refs.py

Line 201 in 0846a1f

def _resolve_anchor(

I would maybe check if the final global environment: docs/_build/html/.doctrees/environment.pickle actually contains all the requisite data that it should (compared to the serial build)

I probably won't have time to do this soon, but you or anyone is welcome to have a go

[andersk]

I’m not exactly sure what to look for, but I do notice that the pickled environment from the parallel build is missing the myst_anchors attribute that the one from the serial build has.

>>> a = pickle.load(open("build-serial/.doctrees/environment.pickle", "rb"))
>>> a
<sphinx.environment.BuildEnvironment object at 0x7f47e873feb0>
>>> a.myst_anchors
True
>>> b = pickle.load(open("build-parallel/.doctrees/environment.pickle", "rb"))
>>> b
<sphinx.environment.BuildEnvironment object at 0x7f47e8701fd0>
>>> b.myst_anchors
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
AttributeError: 'BuildEnvironment' object has no attribute 'myst_anchors'

[andersk]

Indeed, removing the myst_anchors test seems to fix the problem completely:

--- a/myst_parser/myst_refs.py
+++ b/myst_parser/myst_refs.py
@@ -203,7 +203,7 @@ class MystReferenceResolver(ReferencesResolver):
     ) -> Optional[Element]:
         """Resolve doc with anchor."""
         target = node["reftarget"]  # type: str
-        if not ("#" in target and hasattr(self.env, "myst_anchors")):
+        if "#" not in target:
             return None
         # the link may be a heading anchor; we need to first get the relative path
         rel_path, anchor = target.rsplit("#", 1)

[chrisjsewell]

Its okay, I know why it is, fixing now