Interaction with autodoc plugin not working ?

[henridb]

I'm using the autodoc sphinx plugin, so in order to work on the documentation from the docstrings in my code I would expect the following command to work:

sphinx-autobuild docs build --watch src

It seems to partially work, in the sense that editing a source file and saving indeed triggers a rebuild, but the output does not change. From my experimentation, it seems like the code is somehow cached, and this it's always the version existing at the first launch of the above mentioned command that gets built.

I haven't found a solution so far, is this a bug ? a configuration problem ?

[henridb]

By digging some more, the problem might come from the fact that we are not exiting the python interpreter between each build, meaning that my package gets imported once, and then each subsequent import does nothing. This might be solved by using importlib's reload function. I'm still digging :)

[aakhavanQC]

@henridb - any luck debugging this? I'm seeing the same

[pradyunsg]

See https://github.com/sphinx-doc/sphinx-autobuild#relevant-sphinx-bugs. I guess #142 should be reverted/amended since it sounds like rebuilds aren't detecting rescans of source code.

Can you confirm what version of Sphinx you're using?

[aakhavanQC]

See https://github.com/sphinx-doc/sphinx-autobuild#relevant-sphinx-bugs. I guess #142 should be reverted/amended since it sounds like rebuilds aren't detecting rescans of source code.

Can you confirm what version of Sphinx you're using?

I'm on 7.2.6 which appears to be the latest. And autobuild 2024.02.04. I did see the note in the readme, but the issue isn't that the changes in the source code aren't detected. If I make a change to the file, the rebuild runs, but the outputs don't change, even if I disable incremental builds. I'm guessing the reporter is right that because the interpreter lives between rebuilds, it's not actually re-importing the updated source code file.

[henridb]

@henridb - any luck debugging this? I'm seeing the same

Nop, I sorry. In the end I gave up and put the sphinx build on a key building on my IDE. It was the easiest and it's sufficient for my purpose. But if ever this get fixed (or if someone is ready to guide me through the steps to contribute to this repo :) ), I would be happy to come back to this tool!

[pradyunsg]

I'm guessing the reporter is right that because the interpreter lives between rebuilds, it's not actually re-importing the updated source code file.

Hmm... It doesn't -- sphinx-build is invoked as a separate subprocess.

[aakhavanQC]

I'm guessing the reporter is right that because the interpreter lives between rebuilds, it's not actually re-importing the updated source code file.

Hmm... It doesn't -- sphinx-build is invoked as a separate subprocess.

Is there a way to show exactly what command is being invoked? Then I can run that separately to see what happens.

[aakhavanQC]

Did some more digging; the issue isn't with autobuild. It runs sphinx-build -v which doesn't detect the changes in the file. Seems to be fixed by sphinx-doc/sphinx#11939 which has been merged but not yet released.

[aakhavanQC]

@pradyunsg - there is definitely a sphinx-autobuild specific issue here. I installed sphinx from github, so the latest code, and I'm seeing the following:

1. rm -rf docs/_build
2. sphinx-build doc docs/_build
   doc builds correctly
3. change source code file
4. sphinx-build doc docs/_build
   docs update correctly
5. sphinx-autobuild docs docs/_build --watch src
6. change source code file
   docs update correctly
7. change source code file again
   docs do not update
8. exit sphinx-autobuild
9. sphinx-build doc docs/_build
   docs do not update
10. change source code file again
11. sphinx-build doc docs/_build
    docs update correctly

Key takeaways:

• sphinx-build correctly updates the docs every time source code is changed
• sphinx-autobuild correctly updates the docs the first time
• after the first change (or the second build, possibly), sphinx-autobuild corrupts the docs in such a way that sphinx-build no longer updates the docs on source code changes

I can reproduce this consistently

[dshemnv]

Until there is a solution, the workaround I found for this is:

find source/ ../mymodule/ -name "*.rst" -o -name "*.py" | entr make html

I can launch a local server with my IDE (vscode) from the build folder that does the reloading automatically.

[return42]

I'm guessing the reporter is right that because the interpreter lives between rebuilds, it's not actually re-importing the updated source code file.

Hmm... It doesn't -- sphinx-build is invoked as a separate subprocess.

Sure? .. If I interpret the source code correctly; the pre-build tasks are executed in a separate subprocess ..

sphinx-autobuild/sphinx_autobuild/build.py

Lines 22 to 26 in 674264e

	try:
	for command in self.pre_build_commands:
	show(context="pre-build", command=command)
	subprocess.run(command, check=True)
	except subprocess.CalledProcessError as e:

.. but not the sphinx build itself, the sphinx build is in-process (sphinx.cmd.build.build_main) :

sphinx-autobuild/sphinx_autobuild/build.py

Lines 5 to 6 in 674264e

	# This isn't public API, but we want to avoid a subprocess call
	from sphinx.cmd.build import build_main

sphinx-autobuild/sphinx_autobuild/build.py

Lines 36 to 40 in 674264e

	# NOTE:
	# sphinx.cmd.build.build_main is not considered to be public API,
	# but as this is a first-party project, we can cheat a little bit.
	if return_code := build_main(self.sphinx_args):
	print(f"Sphinx exited with exit code: {return_code}")

[pradyunsg]

.. but not the sphinx build itself, the sphinx build is in-process (sphinx.cmd.build.build_main) :

Ah, looks like a change made directly on main that I didn't realise was made. I'm gonna defer to @AA-Turner on this, since it might just be a case of Sphinx's internals holding state that's causing issues here (which is part of why the project was using subprocesses earlier, even though Sphinx has a -m sphinx which could've been invoked via runpy.run_module earlier).

[return42]

Ah, looks like a change made directly on main that I didn't realise was made.

Not the only issue committed in the last few month .. I send #164 to fix two issues I have found.

---

Update: #158 might also related.