docbuild errors out with "cannot copy static file FileExistsError" #32262
Comments
|
comment:1
This is just a gut feeling but I suspect the issue is related with the option + $(MAKE) SAGE_DOCBUILD_OPTS="$(SAGE_DOCBUILD_OPTS) --no-prune-empty-dirs" $(foreach doc, $(OTHER_DOCS), doc-inventory--$(subst /,-,$(doc)))
+ $(MAKE) SAGE_DOCBUILD_OPTS="$(SAGE_DOCBUILD_OPTS) --no-prune-empty-dirs" doc-inventory--reference_topThe other issue in #32043 may be related with this. |
This comment has been minimized.
This comment has been minimized.
|
comment:3
|
|
comment:4
This is where the error is raised: in def copy_html_static_files(self, context: Dict) -> None:
def onerror(filename: str, error: Exception) -> None:
logger.warning(__('Failed to copy a file in html_static_file: %s: %r'),
filename, error)
excluded = Matcher(self.config.exclude_patterns + ["**/.*"])
for entry in self.config.html_static_path:
copy_asset(path.join(self.confdir, entry),
path.join(self.outdir, '_static'),
excluded, context=context, renderer=self.templates, onerror=onerror)
def copy_html_logo(self) -> None:
if self.config.html_logo and not isurl(self.config.html_logo):
copy_asset(path.join(self.confdir, self.config.html_logo),
path.join(self.outdir, '_static'))
def copy_html_favicon(self) -> None:
if self.config.html_favicon and not isurl(self.config.html_favicon):
copy_asset(path.join(self.confdir, self.config.html_favicon),
path.join(self.outdir, '_static'))
def copy_static_files(self) -> None:
try:
with progress_message(__('copying static files')):
ensuredir(path.join(self.outdir, '_static'))
# prepare context for templates
context = self.globalcontext.copy()
if self.indexer is not None:
context.update(self.indexer.context_for_searchtool())
self.create_pygments_style_file()
self.copy_translation_js()
self.copy_stemmer_js()
self.copy_theme_static_files(context)
self.copy_html_static_files(context)
self.copy_html_logo()
self.copy_html_favicon()
except OSError as err:
logger.warning(__('cannot copy static file %r'), err) |
|
comment:5
What is the command to reproduce the bug? |
This comment has been minimized.
This comment has been minimized.
|
comment:11
Investigating this, I find somethings strange. My sage uses system python 3.9 but Sphinx building the doc uses sage's own python 3.9. Is this normal? How can I run manually sage's own python? The bug appeared after #27754 made special tweaks for mac regarding parallel building. Might be related... |
|
comment:12
Replying to @jhpalmieri:
Right. Now I saw the crawling bug. Somehow matplotlib writes a file named into the directory |
|
comment:13
In https://matplotlib.org/stable/_modules/matplotlib/sphinxext/plot_directive.html Find def _copy_css_file(app, exc):
if exc is None and app.builder.format == 'html':
src = cbook._get_data_path('plot_directive/plot_directive.css')
dst = app.outdir / Path('_static')
shutil.copy(src, dst) |
|
comment:14
Ah, hence Sphinx needs to create |
|
comment:15
I think that this function is called by Sphinx+matplotlib when it invokes (or uses?) the |
|
comment:16
Now I see that the issue of this ticket is caused by abandoning class AllBuilder(object):
"""
A class used to build all of the documentation.
"""
...
def _wrapper(self, name, *args, **kwds):
"""
This is the function which goes through all of the documents
and does the actual building.
"""
...
logger.warning("Building reference manual, second pass.\n")
sage_makedirs(os.path.join(SAGE_DOC, "html", "en", "reference", "_static"))
for document in refs:
getattr(get_builder(document), name)(*args, **kwds) I think the issue of #32043 is similarly caused by #31948 by not invoking I leave the necessary work to fix these to the experts on the |
|
comment:17
My experiments show the opposite: if creating --- a/src/sage_docbuild/__init__.py
+++ b/src/sage_docbuild/__init__.py
@@ -345,7 +345,7 @@ class AllBuilder(object):
getattr(get_builder(document), 'inventory')(*args, **kwds)
logger.warning("Building reference manual, second pass.\n")
- sage_makedirs(os.path.join(SAGE_DOC, "html", "en", "reference", "_static"))
+ # sage_makedirs(os.path.join(SAGE_DOC, "html", "en", "reference", "_static"))
for document in refs:
getattr(get_builder(document), name)(*args, **kwds)then docs build. Indeed, in the big picture doing things in the docs directory should be left to |
|
comment:18
oops, no, I take comment:17 back. The patch there has no effect. On the other hand, I really don't see why the builder should explicitly mess around with the sphinx's directory. |
|
comment:19
I think the fault lies with The following (where we forcibly create the diff --git a/src/sage_docbuild/__init__.py b/src/sage_docbuild/__init__.py
index b63e53539e..41140f4e1e 100644
--- a/src/sage_docbuild/__init__.py
+++ b/src/sage_docbuild/__init__.py
@@ -783,6 +783,7 @@ class ReferenceSubBuilder(DocBuilder):
"""
# Force regeneration of all modules if the inherited
# and/or underscored members options have changed.
+ sage_makedirs(os.path.join(SAGE_DOC, "html", "en", "reference", "_static"))
cache = self.get_cache()
force = False
try: |
|
comment:20
Replying to @jhpalmieri:
yes, looks like an upstream bug. Would it be better to patch matplotlib? |
|
comment:21
Thank you for reporting that upstream. I prefer avoiding patches to other packages if possible. If there ends up being a |
|
comment:22
Should I create a branch with my proposed change? |
|
comment:23
Replying to @dimpase:
As far as I understand, sphinx builder ( |
|
comment:24
Replying to @jhpalmieri:
The |
|
comment:25
Replying to @kwankyu:
Everything needs to be tested, but someone could want to build only We didn't run into this before because it's a new Sphinx extension which happens to have a bug (now with a fix at matplotlib/matplotlib#20748). |
|
comment:26
Or maybe we did run into it before: I remember years ago having problems with the |
|
comment:27
Replying to @jhpalmieri:
Also, people may invoke docbuilding using |
|
comment:28
Replying to @jhpalmieri:
Isn't it the initial |
|
comment:29
If someone does |
|
comment:30
Replying to @jhpalmieri:
Okay. Then I have no better idea where the fix should be placed. |
|
Upstream: Fixed upstream, but not in a stable release. |
|
comment:31
To fix this particular issue I'm going to add the patch matplotlib/matplotlib#20748 already produced by Matplotlib in #31580 |
|
comment:32
Replying to @dimpase:
Good. Then there's nothing to do here. |
|
comment:33
I think that conda builds its own matplotlib. If this is right, then until this fix is included in matplotlib, we either shouldn't merge #31580 or we should accept that the documentation won't build in parallel on conda. |
|
comment:34
I've left a comment for Isuru on matplotlib/matplotlib#20748 |
|
Dependencies: #31580 |
|
comment:36
matplotlib 3.4.3 is out and appears to have the fix (matplotlib/matplotlib#20751) |
|
comment:37
Let's close this, then. |
There is apparently a race condition in docbuild which leads to
on many GH Action runs on #31580, e.g.
https://github.com/sagemath/sagetrac-mirror/runs/3104375540 (this is
docker (ubuntu-bionic, standard))https://github.com/sagemath/sagetrac-mirror/runs/3104375098 (this is
local-macos (homebrew-macos-python3_xcode, standard, default, macos-11.0))also reproducible locally on macOS/Homebrew with
make -j8.No failure with
make -j1.This is observed with the latest beta on Linux and macOS after #31948
Reported on sage-dev: https://groups.google.com/g/sage-devel/c/ts6u19KpapA/m/22BdxXZhBAAJ
Depends on #31580
Upstream: Fixed upstream, but not in a stable release.
CC: @jhpalmieri @isuruf
Component: build
Issue created by migration from https://trac.sagemath.org/ticket/32262
The text was updated successfully, but these errors were encountered: