MAINT, DOC: JupyterLite build fails via "make dist" on release branch

[tylerjereddy]

I think running make dist in scipy/doc is supposed to proceed via a venv that pulls in the deps it needs, so this may be good to fix eventually.

Traceback below the fold

copying images... [100%] ../build/plot_directive/tutorial/stats/sampling_srou-1.png
dumping search index in English (code: en)... done
dumping object inventory... done
[jupyterlite-sphinx] Running JupyterLite build

Traceback (most recent call last):
  File "/home/treddy/github_projects/scipy/doc/build/env/lib/python3.9/site-packages/sphinx/events.py", line 97, in emit
    results.append(listener.handler(self.app, *args))
  File "/home/treddy/github_projects/scipy/doc/build/env/lib/python3.9/site-packages/jupyterlite_sphinx/jupyterlite_sphinx.py", line 602, in jupyterlite_build
    completed_process = subprocess.run(command, **kwargs)
  File "/home/treddy/github_projects/spack/opt/spack/linux-ubuntu22.04-skylake/gcc-11.3.0/python-3.9.15-56zjnnbsvkr5pek4sej25bk7odzuysaf/lib/python3.9/subprocess.py", line 505, in run
    with Popen(*popenargs, **kwargs) as process:
  File "/home/treddy/github_projects/spack/opt/spack/linux-ubuntu22.04-skylake/gcc-11.3.0/python-3.9.15-56zjnnbsvkr5pek4sej25bk7odzuysaf/lib/python3.9/subprocess.py", line 951, in __init__
    self._execute_child(args, executable, preexec_fn, close_fds,
  File "/home/treddy/github_projects/spack/opt/spack/linux-ubuntu22.04-skylake/gcc-11.3.0/python-3.9.15-56zjnnbsvkr5pek4sej25bk7odzuysaf/lib/python3.9/subprocess.py", line 1821, in _execute_child
    raise child_exception_type(errno_num, err_msg, err_filename)
FileNotFoundError: [Errno 2] No such file or directory: 'jupyter'

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/home/treddy/github_projects/scipy/doc/build/env/lib/python3.9/site-packages/sphinx/cmd/build.py", line 298, in build_main
    app.build(args.force_all, args.filenames)
  File "/home/treddy/github_projects/scipy/doc/build/env/lib/python3.9/site-packages/sphinx/application.py", line 357, in build
    self.events.emit('build-finished', None)
  File "/home/treddy/github_projects/scipy/doc/build/env/lib/python3.9/site-packages/sphinx/events.py", line 108, in emit
    raise ExtensionError(__("Handler %r for event %r threw an exception") %
sphinx.errors.ExtensionError: Handler <function jupyterlite_build at 0x7f3ba4780940> for event 'build-finished' threw an exception (exception: [Errno 2] No such file or directory: 'jupyter')

Extension error (jupyterlite_sphinx.jupyterlite_sphinx):
Handler <function jupyterlite_build at 0x7f3ba4780940> for event 'build-finished' threw an exception (exception: [Errno 2] No such file or directory: 'jupyter')
make[1]: *** [Makefile:114: html-build] Error 2
make[1]: Leaving directory '/home/treddy/github_projects/scipy/doc'
make: *** [Makefile:80: dist] Error 2

I do see that pip install -r ../requirements/doc.txt succeeded, but I also see this problem that we should likely backport for vs. main (this alone doesn't fix the issue, and I haven't yet checked if make dist fails on main, but it very well might).

diff --git a/requirements/doc.txt b/requirements/doc.txt
index fa160cc5d..5f1b5d30b 100644
--- a/requirements/doc.txt
+++ b/requirements/doc.txt
@@ -8,5 +8,5 @@ numpydoc
 jupytext
 myst-nb
 pooch
-jupyterlite-sphinx>=0.13.1
+jupyterlite-sphinx>=0.12.0
 jupyterlite-pyodide-kernel

I wonder if some extra moving around/directory switching might be throwing things off, but I haven't investigated in detail yet.

[tylerjereddy]

Suggestions for temporary hack would be useful so I can update the docs for 1.13.0 rel.

[steppi]

Just a hunch because I’m away from my computer, but try pinning jupyterlite<0.3. 0.3.0 just came out last week, https://github.com/jupyterlite/jupyterlite/releases/tag/v0.3.0, and it’s possible a breaking change broke jupyterlite-sphinx.

[steppi]

When I get back to my computer I can try to fix it upstream in jupyterlite-sphinx

[tylerjereddy]

No change with clean build using:

--- a/requirements/doc.txt
+++ b/requirements/doc.txt
@@ -8,5 +8,6 @@ numpydoc
 jupytext
 myst-nb
 pooch
-jupyterlite-sphinx>=0.12.0
+jupyterlite<0.3
+jupyterlite-sphinx>=0.13.1
 jupyterlite-pyodide-kernel

[steppi]

I see, I can look into it when I get home in around 1.5 hours.

[steppi]

I'm unable to reproduce this locally. The builds succeed with make dist on my machine, both on main and on the v1.13.0 branch.

[tylerjereddy]

I do the release in a Python 3.9 venv on x86_64 Ubuntu Linux if that helps. Here is the pip freeze output from the venv used by the Makefile:

accessible-pygments==0.0.4
alabaster==0.7.16
anyio==4.3.0
argon2-cffi==23.1.0
argon2-cffi-bindings==21.2.0
arrow==1.3.0
asttokens==2.4.1
attrs==23.2.0
Babel==2.14.0
beautifulsoup4==4.12.3
bleach==6.1.0
certifi==2024.2.2
cffi==1.16.0
charset-normalizer==3.3.2
click==8.1.7
cloudpickle==3.0.0
comm==0.2.2
contourpy==1.2.1
cycler==0.12.1
debugpy==1.8.1
decorator==5.1.1
defusedxml==0.7.1
docutils==0.20.1
doit==0.36.0
exceptiongroup==1.2.0
executing==2.0.1
fastjsonschema==2.19.1
fonttools==4.50.0
fqdn==1.5.1
greenlet==3.0.3
idna==3.6
imagesize==1.4.1
importlib_metadata==7.1.0
importlib_resources==6.4.0
ipykernel==6.29.4
ipython==8.18.1
isoduration==20.11.0
jedi==0.19.1
Jinja2==3.1.3
json5==0.9.24
jsonpointer==2.4
jsonschema==4.21.1
jsonschema-specifications==2023.12.1
jupyter-cache==1.0.0
jupyter-events==0.10.0
jupyter_client==8.6.1
jupyter_core==5.7.2
jupyter_server==2.13.0
jupyter_server_terminals==0.5.3
jupyterlab_pygments==0.3.0
jupyterlab_server==2.25.4
jupyterlite==0.2.3
jupyterlite-core==0.2.3
jupyterlite-pyodide-kernel==0.2.3
jupyterlite-sphinx==0.13.1
jupyterlite_javascript_kernel==0.3.0
jupytext==1.16.1
kiwisolver==1.4.5
markdown-it-py==3.0.0
MarkupSafe==2.1.5
matplotlib==3.8.3
matplotlib-inline==0.1.6
mdit-py-plugins==0.4.0
mdurl==0.1.2
mistune==3.0.2
myst-nb==1.0.0
myst-parser==2.0.0
nbclient==0.10.0
nbconvert==7.16.3
nbformat==5.10.3
nest-asyncio==1.6.0
numpy==1.26.4
numpydoc==1.7.0
overrides==7.7.0
packaging==24.0
pandocfilters==1.5.1
parso==0.8.3
pexpect==4.9.0
pillow==10.3.0
pkginfo==1.10.0
platformdirs==4.2.0
pooch==1.8.1
prometheus_client==0.20.0
prompt-toolkit==3.0.43
psutil==5.9.8
ptyprocess==0.7.0
pure-eval==0.2.2
pycparser==2.22
pydata-sphinx-theme==0.15.2
Pygments==2.17.2
pyparsing==3.1.2
python-dateutil==2.9.0.post0
python-json-logger==2.0.7
PyYAML==6.0.1
pyzmq==25.1.2
referencing==0.34.0
requests==2.31.0
rfc3339-validator==0.1.4
rfc3986-validator==0.1.1
rpds-py==0.18.0
scipy @ file:///home/treddy/github_projects/scipy
Send2Trash==1.8.2
six==1.16.0
sniffio==1.3.1
snowballstemmer==2.2.0
soupsieve==2.5
Sphinx==7.2.6
sphinx_design==0.5.0
sphinxcontrib-applehelp==1.0.8
sphinxcontrib-devhelp==1.0.6
sphinxcontrib-htmlhelp==2.0.5
sphinxcontrib-jsmath==1.0.1
sphinxcontrib-qthelp==1.0.7
sphinxcontrib-serializinghtml==1.1.10
SQLAlchemy==2.0.29
stack-data==0.6.3
tabulate==0.9.0
terminado==0.18.1
tinycss2==1.2.1
toml==0.10.2
tomli==2.0.1
tornado==6.4
traitlets==5.14.2
types-python-dateutil==2.9.0.20240316
typing_extensions==4.10.0
uri-template==1.3.0
urllib3==2.2.1
wcwidth==0.2.13
webcolors==1.13
webencodings==0.5.1
websocket-client==1.7.0
zipp==3.18.1

Also, it looks like make clean doesn't clean up doc/source/_contents used by the notebook stuff I think.

[tylerjereddy]

I can circumvent by using Python 3.11 instead of Python 3.9 as the base venv Python version after the rest of the release process. Since 3.9 support is about to be removed from main I'm not sure how much time this is worth.

I did do git diff --no-index py_311_reqs.txt py_39_reqs.txt on the frozen respective envs before moving in case we care:

diff --git a/py_311_reqs.txt b/py_39_reqs.txt
index 5c78395a2..df8cd54fb 100644
--- a/py_311_reqs.txt
+++ b/py_39_reqs.txt
@@ -22,6 +22,7 @@ decorator==5.1.1
 defusedxml==0.7.1
 docutils==0.20.1
 doit==0.36.0
+exceptiongroup==1.2.0
 executing==2.0.1
 fastjsonschema==2.19.1
 fonttools==4.50.0
@@ -30,8 +31,9 @@ greenlet==3.0.3
 idna==3.6
 imagesize==1.4.1
 importlib_metadata==7.1.0
+importlib_resources==6.4.0
 ipykernel==6.29.4
-ipython==8.23.0
+ipython==8.18.1
 isoduration==20.11.0
 jedi==0.19.1
 Jinja2==3.1.3
@@ -47,9 +49,11 @@ jupyter_server==2.13.0
 jupyter_server_terminals==0.5.3
 jupyterlab_pygments==0.3.0
 jupyterlab_server==2.25.4
+jupyterlite==0.2.3
 jupyterlite-core==0.2.3
 jupyterlite-pyodide-kernel==0.2.3
 jupyterlite-sphinx==0.13.1
+jupyterlite_javascript_kernel==0.3.0
 jupytext==1.16.1
 kiwisolver==1.4.5
 markdown-it-py==3.0.0
@@ -114,6 +118,7 @@ tabulate==0.9.0
 terminado==0.18.1
 tinycss2==1.2.1
 toml==0.10.2
+tomli==2.0.1
 tornado==6.4
 traitlets==5.14.2
 types-python-dateutil==2.9.0.20240316

[steppi]

I can circumvent by using Python 3.11 instead of Python 3.9 as the base venv Python version after the rest of the release process. Since 3.9 support is about to be removed from main I'm not sure how much time this is worth.

Glad you got it working. It's interesting. I ran make dist with an active Python 3.9.15 virtualenv also running on Ubuntu Linux on x86_64 and things worked. I assume the venv from make dist will use the Python from the active virtualenv? I think you're right that it's probably not worth the effort to fix this on the SciPy side, but I think I still need to figure out what's going on with jupyterlite-sphinx since its pyproject.toml says it supports Python 3.8 and up.