Skip to content

Migrate CI build-doc to Meson #39973

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 27 commits into from
Jul 6, 2025
Merged

Migrate CI build-doc to Meson #39973

merged 27 commits into from
Jul 6, 2025

Conversation

tobiasdiez
Copy link
Contributor

@tobiasdiez tobiasdiez commented Apr 19, 2025

Make it possible to build the documentation with meson via

meson compile -C builddir doc-html

For this a couple of changes was necessary in the docbuilder:

  • Make sage_docbuild independent of sage so that meson can use sage_docbuild during config time to construct all the docbuild targets (otherwise one needs to first install sage, and then could configure the docbuild)
  • Properly handle input and output dirs for the docbuild, without relying on magic sage env variables

📝 Checklist

  • The title is concise and informative.
  • The description explains in detail what this PR is about.
  • I have linked a relevant issue or discussion.
  • I have created tests covering the changes.
  • I have updated the documentation and checked the documentation preview.

⌛ Dependencies

@user202729
Copy link
Contributor

Looks like some file permission goes wrong? (symbolic link get converted to normal file?)

docs = [doc[1] for doc in sorted(documents)]
# Put the bibliography first, because it needs to be built first:
docs.remove(Path('reference/references'))
docs.insert(0, Path('reference/references'))
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Why is such a large change necessary in this file?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

To be honest, I got carried a way a bit: the strictly necessary change is to remove mentions of import sage, then I used the chance to use pathlib, and to catch all those paths vs strings issues added typing info. Oh, and there were a few old deprecated stuff in there which was easier to remove than to migrate to pathlib 🦊

Copy link

github-actions bot commented May 17, 2025

Documentation preview for this PR (built with commit 3408887; changes) is ready! 🎉
This preview will update shortly after each push to this PR.

@tobiasdiez tobiasdiez marked this pull request as ready for review May 17, 2025 10:23
@tobiasdiez
Copy link
Contributor Author

Doc build is working now, and the changes relative to the last develop build seem to be minimal and mostly due to using a newer sphinx version now.

Marking it as blocker since currently the docbuild CI for other PRs is completely failing.

@dimpase
Copy link
Member

dimpase commented May 17, 2025

https://github.com/sagemath/sage/actions/runs/15081532937/job/42398821050#step:9:1

fails while building docs the old way - is it normal?

@tobiasdiez
Copy link
Contributor Author

It fails with

ImportError: libSingular-4.4.0.so: cannot open shared object file: No such file or directory

https://github.com/sagemath/sage/actions/runs/15081532937/job/42398821050#step:9:3982 so this is the same problem as for the other PRs.

@dimpase
Copy link
Member

dimpase commented May 17, 2025

no, my question was - why does that CI run build docs the old way?

@tobiasdiez
Copy link
Contributor Author

according to our docs one should run the following command to bump primecountpy's version - but it errors out:

$ uv lock --upgrade-package primecountpy
      Built sagemath @ file:///home/dima/software/sage
  × No solution found when resolving dependencies:
  ╰─▶ Because sagemath-giac was not found in the package registry and sagemath[giac] depends on sagemath-giac, we can conclude that sagemath[giac]'s requirements are unsatisfiable.
      And because your project requires sagemath[giac], we can conclude that your project's requirements are unsatisfiable.

It seems its not published to pypi (tracked at sagemath/sagemath-giac#3)

I've now updated primecountpy manually.

@dimpase
Copy link
Member

dimpase commented Jun 24, 2025

OK, thanks for the update, now I am finally able to run meson compile ... doc-html locally, but not until I pip-installed
sphinx_inline_tabs and sphinx_copybutton. It looks like that the configuration (not sure where exactly) misses these two dependencies, and I got runtime errors as these packages weren't there.

@dimpase
Copy link
Member

dimpase commented Jun 24, 2025

getting an error related to graphs.db - database_graphs. This thing is optional, its absence should not cause an error.

also, can you tell me how to rebuild a part of sagelib, namely, sage/rings/polynomial/pbori/ ?
I've upgraded system m4ri, and now get a weird linking error:

$ ldd build/cp313/src/sage/rings/polynomial/pbori/pbori.cpython-313-x86_64-linux-gnu.so
	linux-vdso.so.1 (0x00007fe32f0c1000)
	libbrial.so.3 => /usr/lib64/libbrial.so.3 (0x00007fe32eedf000)
	libbrial_groebner.so.3 => /usr/lib64/libbrial_groebner.so.3 (0x00007fe32ea00000)
	libm4ri.so.1 => /usr/lib64/libm4ri.so.1 (0x00007fe32eead000)
	libstdc++.so.6 => /usr/lib/gcc/x86_64-pc-linux-gnu/15/libstdc++.so.6 (0x00007fe32e600000)
	libgcc_s.so.1 => /usr/lib/gcc/x86_64-pc-linux-gnu/15/libgcc_s.so.1 (0x00007fe32ee7e000)
	libc.so.6 => /usr/lib64/libc.so.6 (0x00007fe32e411000)
	libm.so.6 => /usr/lib64/libm.so.6 (0x00007fe32ed95000)
	libm4ri-0.0.20200125.so => not found
	libpng16.so.16 => /usr/lib64/libpng16.so.16 (0x00007fe32ed5c000)
	libgomp.so.1 => /usr/lib/gcc/x86_64-pc-linux-gnu/15/libgomp.so.1 (0x00007fe32e9a8000)
	/lib64/ld-linux-x86-64.so.2 (0x00007fe32f0c3000)
	libz.so.1 => /usr/lib64/libz.so.1 (0x00007fe32ed3f000)

so it seems that the Cython/gcc compilation result for pbory.pyx is cached somewhere, so it has to be purged, but how?

@dimpase
Copy link
Member

dimpase commented Jun 24, 2025

libm4ri-0.0.20200125.so => not found is surely an artifact of hardcoding library versions

environment-3.11-linux-aarch64.yml:  - m4rie=20200125=hedfd65a_0
environment-3.11-macos-x86_64.yml:  - m4rie=20200125=hd82a5f3_0
environment-3.11-macos.yml:  - m4rie=20200125=hc97c1ff_0
environment-3.12-linux-aarch64.yml:  - m4rie=20200125=hedfd65a_0
environment-3.12-linux.yml:  - m4rie=20200125=h051dbe0_0
environment-3.12-macos-x86_64.yml:  - m4rie=20200125=hd82a5f3_0
environment-3.12-macos.yml:  - m4rie=20200125=hc97c1ff_0

while it's OK for conda, for uv-maintained local venv this is quite suboptimal

@dimpase
Copy link
Member

dimpase commented Jun 24, 2025

OK, thanks for the update, now I am finally able to run meson compile ... doc-html locally, but not until I pip-installed sphinx_inline_tabs and sphinx_copybutton. It looks like that the configuration (not sure where exactly) misses these two dependencies, and I got runtime errors as these packages weren't there.

hmm, maybe I should have run uv sync to get them all installed

@dimpase
Copy link
Member

dimpase commented Jun 24, 2025

doc-html works, but something is wrong with meson compile -C build/cp313/ doc-pdf for me.
I am getting many errors like

/home/dima/software/sage/.venv/bin/python3 /home/dima/software/sage/src/build-docs.py --no-pdf-links es/a_tour_of_sage pdf -o src/doc --source /home/dima/software/sage/src/doc
[a_tour_of] inventory <https://ipywidgets.readthedocs.io/en/stable/> contains multiple definitions for std:label:examples/Widget Layout.ipynb#display
[a_tour_of] building [latex]: all documents
[a_tour_of] processing a_tour_of_sage.tex...
[a_tour_of] index
[a_tour_of] resolving references...
[a_tour_of] The LaTeX files are in src/doc/latex/es/a_tour_of_sage.
[a_tour_of] Run 'make' in that directory to run these through (pdf)latex
[a_tour_of] (use `make latexpdf' here to do that automatically).
LaTeX files can be found in /home/dima/software/sage/build/cp313/src/doc/latex/es/a_tour_of_sage.
/bin/sh: line 1: all-pdf: command not found
Error building the documentation.
Traceback (most recent call last):
  File "/home/dima/software/sage/src/build-docs.py", line 4, in <module>
    main()
    ~~~~^^
  File "/home/dima/software/sage/src/sage_docbuild/__main__.py", line 548, in main
    build()
    ~~~~~^^
  File "/home/dima/software/sage/src/sage_docbuild/builders.py", line 273, in pdf
    raise RuntimeError(error_message % (command, tex_dir))
RuntimeError: failed to run $MAKE all-pdf in /home/dima/software/sage/build/cp313/src/doc/latex/es/a_tour_of_sage
[97/190] Generating src/doc/doc-pdf-other-en-thematic_tutorials with a custom command
FAILED: src/doc/pdfen-thematic_tutorials 

and there are make.bat files generated:

$ find . -name make.bat
./build/cp313/src/doc/latex/en/developer/make.bat
./build/cp313/src/doc/latex/en/website/make.bat
./build/cp313/src/doc/latex/en/reference/references/make.bat
./build/cp313/src/doc/latex/en/tutorial/make.bat
./build/cp313/src/doc/latex/en/constructions/make.bat
./build/cp313/src/doc/latex/en/faq/make.bat
./build/cp313/src/doc/latex/en/thematic_tutorials/make.bat
./build/cp313/src/doc/latex/en/a_tour_of_sage/make.bat
./build/cp313/src/doc/latex/en/prep/make.bat
./build/cp313/src/doc/latex/en/installation/make.bat
./build/cp313/src/doc/latex/ca/intro/make.bat
./build/cp313/src/doc/latex/de/a_tour_of_sage/make.bat
./build/cp313/src/doc/latex/de/tutorial/make.bat
./build/cp313/src/doc/latex/de/thematische_anleitungen/make.bat
./build/cp313/src/doc/latex/el/a_tour_of_sage/make.bat
./build/cp313/src/doc/latex/es/a_tour_of_sage/make.bat

for instance

@ECHO OFF

REM Command file for Sphinx documentation

pushd %~dp0

set PDFLATEX=latexmk -pdf -dvi- -ps-

set "LATEXOPTS= "

set XINDYOPTS=-L spanish-modern -C utf8  -M sphinx.xdy
set XINDYOPTS=%XINDYOPTS% -I xelatex
if "%1" == "" goto all-pdf

if "%1" == "all-pdf" (
        :all-pdf
        for %%i in (*.tex) do (
                %PDFLATEX% %LATEXMKOPTS% %%i
        )
        goto end
)

if "%1" == "all-pdf-ja" (
        goto all-pdf
)

if "%1" == "clean" (
        del /q /s *.dvi *.log *.ind *.aux *.toc *.syn *.idx *.out *.ilg *.pla *.ps *.tar *.tar.gz *.tar.bz2 *.tar.xz *.fls *.fdb_latexmk
        goto end
)

:end
popd
  • that's probably where /bin/sh: line 1: all-pdf: command not found comes from.

@dimpase
Copy link
Member

dimpase commented Jun 24, 2025

As far as I understand, doc-pdf isn't there yet. OK, good. Next PR then.

@dimpase dimpase self-requested a review June 24, 2025 22:00
@user202729
Copy link
Contributor

Build and test (fedora:{41,42}) is failing, what's going on?

@dimpase
Copy link
Member

dimpase commented Jun 25, 2025

a typo in the script: setuptool in place of setuptools (which has been fixed here, but its propagation to CI could be delayed)

@tobiasdiez
Copy link
Contributor Author

Thanks for the review @dimpase. I've created new issues for the problems you mentioned.

@user202729 This is fixed in #40071

vbraun pushed a commit to vbraun/sage that referenced this pull request Jun 27, 2025
sagemathgh-39973: Migrate CI build-doc to Meson
    
<!-- ^ Please provide a concise and informative title. -->
<!-- ^ Don't put issue numbers in the title, do this in the PR
description below. -->
<!-- ^ For example, instead of "Fixes sagemath#12345" use "Introduce new method
to calculate 1 + 2". -->
<!-- v Describe your changes below in detail. -->
<!-- v Why is this change required? What problem does it solve? -->
<!-- v If this PR resolves an open issue, please link to it here. For
example, "Fixes sagemath#12345". -->

Make it possible to build the documentation with meson via
```
meson compile -C builddir doc-html
```

For this a couple of changes was necessary in the docbuilder:
- Make `sage_docbuild` independent of `sage` so that meson can use
`sage_docbuild` during config time to construct all the docbuild targets
(otherwise one needs to first install sage, and then could configure the
docbuild)
- Properly handle input and output dirs for the docbuild, without
relying on magic sage env variables

### 📝 Checklist

<!-- Put an `x` in all the boxes that apply. -->

- [ ] The title is concise and informative.
- [ ] The description explains in detail what this PR is about.
- [ ] I have linked a relevant issue or discussion.
- [ ] I have created tests covering the changes.
- [ ] I have updated the documentation and checked the documentation
preview.

### ⌛ Dependencies

<!-- List all open PRs that this PR logically depends on. For example,
-->
<!-- - sagemath#12345: short description why this is a dependency -->
<!-- - sagemath#34567: ... -->
    
URL: sagemath#39973
Reported by: Tobias Diez
Reviewer(s): Dima Pasechnik, Tobias Diez, user202729
@tobiasdiez tobiasdiez mentioned this pull request Jul 6, 2025
5 tasks
@vbraun vbraun merged commit 7b79f0e into sagemath:develop Jul 6, 2025
22 of 27 checks passed
@antonio-rojas
Copy link
Contributor

I can't figure out how to build the documentation now. I used to do

python -m sage_docbuild --no-pdf-links --mathjax all html

which no longer works since the all option was removed here. If I do

meson setup sage build
meson compile -C build doc-html

then I get errors such as

FAILED: src/doc/inventoryreferences 
/usr/bin/python /build/sagemath-doc/src/sage/src/build-docs.py --no-pdf-links reference/references inventory -o src/doc --source /build/sagemath-doc/src/sage/src/doc
Error building the documentation.
Traceback (most recent call last):
  File "/build/sagemath-doc/src/sage/src/build-docs.py", line 4, in <module>
    main()
    ~~~~^^
  File "/build/sagemath-doc/src/sage/src/sage_docbuild/__main__.py", line 545, in main
    import sage.all  # TODO: Remove once all modules can be imported independently  # noqa: F401
    ^^^^^^^^^^^^^^^
  File "/build/sagemath-doc/src/sage/src/sage/all.py", line 64, in <module>
    from sage.all__sagemath_repl import *
  File "/build/sagemath-doc/src/sage/src/sage/all__sagemath_repl.py", line 108, in <module>
    from sage.all__sagemath_objects import *
  File "/build/sagemath-doc/src/sage/src/sage/all__sagemath_objects.py", line 17, in <module>
    from sage.misc.all__sagemath_objects import *
  File "/build/sagemath-doc/src/sage/src/sage/misc/all__sagemath_objects.py", line 4, in <module>
    import sage.structure.all   # to break a cyclic import
    ^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/build/sagemath-doc/src/sage/src/sage/structure/__init__.py", line 3, in <module>
    import sage.structure.element
ModuleNotFoundError: No module named 'sage.structure.element'

since sage hasn't been compiled. Even if I compile it with

meson setup sage build
meson compile -C build
meson compile -C build doc-html

I get the same error, since the compiled modules aren't available in the python path. The closest I get is if I delete the sage source dir and replace it with a symlink to the installed sage python module, but then I get cryptic errors such as

FAILED: src/doc/inventorymisc 
/usr/bin/python /build/sagemath-doc/src/sage/src/build-docs.py --no-pdf-links reference/misc inventory -o src/doc --source /build/sagemath-doc/src/sage/src/doc
[misc     ] building [inventory]: targets for 100 source files that are out of date
[misc     ] updating environment: [new config] 100 added, 0 changed, 0 removed
[misc     ] /usr/lib/python3.13/site-packages/sphinx/events.py:394: DeprecationWarning:
[misc     ] Importing benchmark from here is deprecated; please use "from sage.misc.benchmark import benchmark" instead.
[misc     ] See https://github.com/sagemath/sage/issues/34259 for details.
[misc     ]   repr_args = repr(args)
[misc     ] WARNING: error while formatting arguments for sage.misc.c3_controlled.HierarchyElement.bases:  [autodoc]
[misc     ] WARNING: error while formatting arguments for sage.misc.c3_controlled.HierarchyElement.cls:  [autodoc]
[misc     ] WARNING: error while formatting arguments for sage.misc.c3_controlled.HierarchyElement.mro:  [autodoc]
[misc     ] WARNING: error while formatting arguments for sage.misc.c3_controlled.HierarchyElement.mro_controlled:  [autodoc]
[misc     ] WARNING: error while formatting arguments for sage.misc.c3_controlled.HierarchyElement.mro_standard:  [autodoc]
[misc     ] The inventory file is in src/doc/inventory/en/reference/misc.
Error building the documentation.
Traceback (most recent call last):
  File "/build/sagemath-doc/src/sage/src/build-docs.py", line 4, in <module>
    main()
    ~~~~^^
  File "/build/sagemath-doc/src/sage/src/sage_docbuild/__main__.py", line 548, in main
    build()
    ~~~~~^^
  File "/build/sagemath-doc/src/sage/src/sage_docbuild/builders.py", line 645, in _wrapper
    getattr(DocBuilder, build_type)(self, *args, **kwds)
    ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^
  File "/build/sagemath-doc/src/sage/src/sage_docbuild/builders.py", line 136, in f
    runsphinx()
    ~~~~~~~~~^^
  File "/build/sagemath-doc/src/sage/src/sage_docbuild/sphinxbuild.py", line 324, in runsphinx
    sys.stderr.raise_errors()
    ~~~~~~~~~~~~~~~~~~~~~~~^^
  File "/build/sagemath-doc/src/sage/src/sage_docbuild/sphinxbuild.py", line 255, in raise_errors
    raise OSError(self._error)
OSError: WARNING: error while formatting arguments for sage.misc.c3_controlled.HierarchyElement.bases:  [autodoc]

So how do I do it?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging this pull request may close these issues.

5 participants