BUG: numpy doesn't build correct texinfo (and info target in doc fails)

[ilitzroth]

make info in the doc folder of numpy
....
numpy.texi:26442: warning: @menu in invalid context
numpy.texi:26497: @node seen before @EnD deffn
numpy.texi:27903: unmatched @end deffn' numpy.texi:28524: warning: @menu in invalid context numpy.texi:28529: @node seen before @end deffn numpy.texi:28661: unmatched @EnD deffn'
numpy.texi:32180: warning: @menu in invalid context
numpy.texi:32191: @node seen before @EnD enumerate
numpy.texi:32412: unmatched @end enumerate' numpy.texi:33111: warning: @menu in invalid context numpy.texi:33176: @node seen before @end deffn numpy.texi:36525: unmatched @EnD deffn'
numpy.texi:37031: warning: @menu in invalid context
numpy.texi:37036: @node seen before @EnD deffn
numpy.texi:37152: unmatched @end deffn' numpy.texi:37919: warning: @menu in invalid context numpy.texi:37985: @node seen before @end deffn numpy.texi:40488: unmatched @EnD deffn'
numpy.texi:41352: warning: @menu in invalid context
numpy.texi:41412: @node seen before @EnD deffn
numpy.texi:44141: unmatched @end deffn' numpy.texi:44641: warning: @menu in invalid context numpy.texi:44697: @node seen before @end deffn numpy.texi:46119: unmatched @EnD deffn'
numpy.texi:46403: warning: @menu in invalid context
numpy.texi:46408: @node seen before @EnD deffn
numpy.texi:46444: unmatched @end deffn' numpy.texi:46631: warning: @menu in invalid context numpy.texi:46636: @node seen before @end deffn numpy.texi:46705: unmatched @EnD deffn'
numpy.texi:46896: warning: @menu in invalid context
numpy.texi:46917: @node seen before @EnD itemize
numpy.texi:47021: @item outside of table or list
numpy.texi:47040: @item outside of table or list
numpy.texi:48228: unmatched @end itemize' numpy.texi:84850: warning: @menu in invalid context numpy.texi:84916: @node seen before @end deffn numpy.texi:87419: unmatched @EnD deffn'
numpy.texi:95122: warning: @menu in invalid context
numpy.texi:95127: @node seen before @EnD deffn
numpy.texi:95150: unmatched @end deffn' numpy.texi:99377: warning: @menu in invalid context numpy.texi:99389: @node seen before @end deffn numpy.texi:99574: unmatched @EnD deffn'
numpy.texi:99647: warning: @menu in invalid context
numpy.texi:99653: @node seen before @EnD deffn
numpy.texi:99705: unmatched @end deffn' numpy.texi:99921: warning: @menu in invalid context numpy.texi:99926: @node seen before @end deffn numpy.texi:99968: unmatched @EnD deffn'
numpy.texi:102753: warning: @menu in invalid context
numpy.texi:102760: @node seen before @EnD deffn
numpy.texi:102926: unmatched @end deffn' numpy.texi:122736: warning: @menu in invalid context numpy.texi:122761: @node seen before @end deffn numpy.texi:123751: unmatched @EnD deffn'
numpy.texi:126293: warning: @menu in invalid context
numpy.texi:126319: @node seen before @EnD deffn
numpy.texi:127375: unmatched @end deffn' numpy.texi:130080: warning: @menu in invalid context numpy.texi:130105: @node seen before @end deffn numpy.texi:131095: unmatched @EnD deffn'
numpy.texi:133801: warning: @menu in invalid context
numpy.texi:133826: @node seen before @EnD deffn
numpy.texi:134816: unmatched @end deffn' numpy.texi:137549: warning: @menu in invalid context numpy.texi:137574: @node seen before @end deffn numpy.texi:138564: unmatched @EnD deffn'
numpy.texi:141301: warning: @menu in invalid context
numpy.texi:141326: @node seen before @EnD deffn
numpy.texi:142316: unmatched @end deffn' numpy.texi:145741: warning: @menu in invalid context numpy.texi:145748: @node seen before @end deffn numpy.texi:145846: unmatched @EnD deffn'
numpy.texi:152896: warning: @menu in invalid context
numpy.texi:152948: @node seen before @EnD deffn
numpy.texi:157916: unmatched @end deffn' numpy.texi:166871: warning: @menu in invalid context numpy.texi:166878: @node seen before @end deffn numpy.texi:166997: unmatched @EnD deffn'
numpy.texi:199639: warning: @subsubsection missing argument
Makefile:32: recipe for target 'numpy.info' failed
make[1]: *** [numpy.info] Error 1
make[1]: Leaving directory '/home/immanuel/workspace/Python/numpy-1.15.3/doc/build/texinfo'
Makefile:189: recipe for target 'info' failed
make: *** [info] Error 2

from numpy-1.15.3 tarbal

[charris]

Did that used to work? I didn't even know that info files were ever being produced for numpy.

[ilitzroth]

Well as you can see it's in the Makefile and I didn't add it...

[Apteryks]

It seems many (but not all) of the problems are caused by the numpydoc Sphinx extension. Commenting it out in doc/source/conf.py, the failures/warning are down to:

make[1]: Entering directory '/tmp/guix-build-python-numpy-documentation-1.21.6.drv-0/numpy-1.21.6/doc/build/texinfo'
makeinfo --no-split -o 'numpy.info' 'numpy.texi'
numpy.texi:47842: warning: @subsubsection missing argument
numpy.texi:60609: warning: @menu in invalid context
numpy.texi:60620: @node seen before @end enumerate
numpy.texi:60781: unmatched `@end enumerate'
numpy.texi:62943: warning: @menu in invalid context
numpy.texi:62964: @node seen before @end itemize
numpy.texi:63118: @item outside of table or list
numpy.texi:63137: @item outside of table or list
numpy.texi:64029: unmatched `@end itemize'
make[1]: *** [Makefile:39: numpy.info] Error 1
make[1]: Leaving directory '/tmp/guix-build-python-numpy-documentation-1.21.6.drv-0/numpy-1.21.6/doc/build/texinfo'
make: *** [Makefile:254: info] Error 2

[Apteryks]

OK, the remaining problems were caused by the 'sphinx.ext.autosummary' extension (it autogenerates section and places them in the middle of enumerations, which causes @node to appear before the enumeration is closed, which is invalid in Texinfo.

[rossbar]

It seems many (but not all) of the problems are caused by the numpydoc Sphinx extension. Commenting it out in doc/source/conf.py, the failures/warning are down to:

Commenting out the numpydoc extension is not really possible, at least to have the docstrings parsed correctly. I assume you did this after the doctrees were already successfully populated. It would be unlikely that numpydoc would fail on the writing output part of the doc build process as I believe it is only involved in the docstring parsing process.

[Apteryks]

It seems numpydoc had something to do with the Texinfo generation as I could see autogenerated comments like "processed by numpydoc" or similar. The recipe in the form of a GNU Guix package looks like:

(define-public python-numpy-documentation
  (package
    (inherit python-numpy)
    (name "python-numpy-documentation")
    (arguments
     (list
      #:tests? #f                     ;we're only generating the documentation
      #:phases
      #~(modify-phases %standard-phases
          (add-before 'build 'add-gnu-freefont-to-texmf
            (lambda _
              ;; XXX: The Sphinx-generated tex output specifies the GNU
              ;; FreeFont font to be searched via its extension, which uses
              ;; kpathsea instead of fontconfig and fail (see:
              ;; https://github.com/sphinx-doc/sphinx/issues/10347).  Create a
              ;; symlink to GNU FreeFont and add it to the TEXMF tree via
              ;; GUIX_TEXMF.
              (mkdir-p "texmf-dist/fonts/opentype/public")
              (symlink (string-append
                        #$(this-package-native-input "font-gnu-freefont")
                        "/share/fonts/opentype")
                       (string-append
                        (getcwd) "/"
                        "texmf-dist/fonts/opentype/public/gnu-freefont"))
              (setenv "GUIX_TEXMF" (string-append (getenv "GUIX_TEXMF") ":"
                                                  (getcwd) "/texmf-dist"))))
          (delete 'build)
          (replace 'install
            (lambda _
              (let* ((data (string-append #$output "/share"))
                     (doc (string-append data "/doc/numpy"))
                     (html (string-append doc "/html"))
                     (info (string-append data "/info"))
                     (sphinxopts (string-append
                                  "SPHINXOPTS=-j"
                                  (number->string (parallel-job-count)))))
                (with-directory-excursion "doc"
                  ;; Do not treat warnings as errors.
                  (substitute* "Makefile"
                    ((" -WT ") " -T "))
                  ;; The intersphinx extension causes Sphinx to attempt
                  ;; fetching resources from the network.
                  (substitute* "source/conf.py"
                    ((".*'sphinx.ext.intersphinx'.*") ""))
                  (setenv "HOME" "/tmp")
                  ;; Build the PDF documentation.
                  (invoke "make" "latex-build" sphinxopts)
                  (invoke "make" "-C" "build/latex" "all-pdf" sphinxopts)
                  ;; Build the HTML documentation
                  (invoke "make" "html" sphinxopts)
                  ;; Build the Info documentation.  The issues worked around
                  ;; below can be tracked at
                  ;; https://github.com/numpy/numpy/issues/12278.
                  (substitute* "source/conf.py"
                    ;; The root document should be "index", not "contents".
                    (("\"contents\"") "'index'")
                    ;; Disable other Sphinx extensions such as numpydoc and
                    ;; autosummary as they produce broken Texinfo.
                    ((".*'numpydoc'.*") "")
                    ((".*'sphinx.ext.autosummary'.*") ""))
                  (invoke "make" "info" sphinxopts)
                  ;; Install the HTML documentation.
                  (mkdir-p html)
                  (copy-recursively "build/html" html)
                  ;; Install the PDF reference and user manuals.
                  (install-file "build/latex/numpy-ref.pdf" doc)
                  (install-file "build/latex/numpy-user.pdf" doc)
                  ;; Install the info manual.
                  (install-file "build/texinfo/numpy.info" info)
                  (copy-recursively "build/texinfo/numpy-figures"
                                    (string-append info "/numpy-figures")))))))))
    (native-inputs
     (list font-gnu-freefont
           fontconfig                   ;for XDG_DATA_DIRS
           perl
           python-breathe
           python-ipython
           python-matplotlib
           python-numpy
           python-numpydoc
           python-pandas
           python-pydata-sphinx-theme
           python-scipy                 ;used by matplotlib
           ;; With Sphinx 4.5, we get: sphinx.errors.SphinxParallelError:
           ;; AttributeError: can't set attribute; probably due to Sphinx
           ;; extensions (related:
           ;; https://github.com/sphinx-doc/sphinx/issues/8730).
           python-sphinx-4.2
           python-sphinx-panels
           texinfo
           texlive-bin
           texlive-cbfonts
           texlive-cm-super
           texlive-greek-fontenc
           texlive-latex-expdlist
           texlive-polyglossia
           texlive-xindy))
    (inputs '())
    (propagated-inputs '())
    (synopsis "Documentation for the @code{python-numpy} package")
    (description "This package provides the complete NumPy documentation in
the Texinfo, HTML, and PDF formats.")))

At least the s/"contents"/"index"/ change in source/conf.py should be adjusted.

[Apteryks]

The resulting numpy.info Texinfo manual is usuable and light at 4.1 MiB (including figures), perfect for offline navigation. The HTML doc weighs 711MiB and the two PDFs 12.2 MiB combined.