FreeBSD fails build PDF output with sphinx 1.8

[kiwichris]

FreeBSD has texlive-2015 and it does not have xindy. As a result running the Makefile results in an error, no xindy.

The RTEMS project's documentation is built with sphinx-build we cannot update to 1.8. Our servers run FreeBSD.

I cannot find a way to work around this issue. We seem to need \spxpagem which is defined in sphinx.xdy and that is used by `xindy.

[jfbu]

Are you using latex_engine='xelatex' ? Else, there is no reason why xindy would be required.

Have you tried with latex_use_xindy = False in the conf.py, what happens then ?

The macro \spxpagem is defined also in python.ist file, used by makeindex.
If there is a python.ist file in your LaTeX build repertory which does not contain

\\let\\spxpagem \\sphinxstyleindexpagemain

then something is wrong.

I would clean up entirely the LaTeX build repertory and try again, but possibly you did that already.

[jfbu]

Ok, I checked your project and your build process seems to bypass the Sphinx provided Makefile in build repertory, as a result latexmk is not used and makeindex is certainly not called as it should be i.e. it is lacking -s python.ist flag.

I forced usage of this via hand-editing filr waflib/Tools/tex.py

....
	makeindex_fun,_=Task.compile_fun('${MAKEINDEX} -s python.ist ${SRCFILE}',shell=False)
	makeindex_fun.__doc__="""
	Execute the program **makeindex**
...

because I did not know how to set MAKEINDEXFLAGS.

....

[7/7] Compiling build/user/latex/user.tex
first pass on pdflatex
This is pdfTeX, Version 3.14159265-2.6-1.40.19 (TeX Live 2018) (preloaded format=pdflatex)
 \write18 enabled.
entering extended mode
/Users/WWW/sphinxvenv/bin/pygmentize

calling makeindex
This is makeindex, version 2.15 [TeX Live 2018] (kpathsea + Thai support).
Scanning style file ./python.ist.......done (7 attributes redefined, 0 ignored).
Scanning input file user.idx....done (231 entries accepted, 0 rejected).
Sorting entries.....done (1883 comparisons).
Generating output file user.ind....done (329 lines written, 0 warnings).
Output written in user.ind.
Transcript written in user.ilg.

calling pdflatex
This is pdfTeX, Version 3.14159265-2.6-1.40.19 (TeX Live 2018) (preloaded format=pdflatex)
 \write18 enabled.
entering extended mode
/Users/xxx/yyy/zzz/sphinxvenv/bin/pygmentize

calling pdflatex
This is pdfTeX, Version 3.14159265-2.6-1.40.19 (TeX Live 2018) (preloaded format=pdflatex)
 \write18 enabled.
entering extended mode
/Users/xxx/yyy/zzz/sphinxvenv/bin/pygmentize

.... cut lines ....

[37/44] Processing generated-posix-compliance.rst: posix-compliance/posix_rst.py posix-compliance/RTEMS-Standards-Compliance-v4a.csv -> build/posix-compliance/generated-posix-compliance.rst
[38/44] Compiling common/latex/rtemsextrafonts.sty
[39/44] Compiling common/_static/favicon.ico
[40/44] Compiling common/_static/logo.png
/Users/TTT/UUU/rtems-docs/common/rtemsext.py:53: RemovedInSphinx30Warning: app.override_domain() is deprecated. Use app.add_domain() with override option instead.
  app.override_domain(CustomStandardDomain)
/Users/XXX/YYY/sphinx/sphinx/builders/html.py:144: RemovedInSphinx30Warning: builder.script_files is deprecated. Please use app.add_js_file() instead.
  ret += other

Waf: Leaving directory `/Users/xxx/yyy/rtems-docs/build'
Build failed
Traceback (most recent call last):
  File "/Users/ZZZ/rtems-docs/.waf3-1.9.7-d27222240ebc8bcbca7fcd8f4ae914fb/waflib/Task.py", line 145, in process
    ret=self.run()
  File "/Users/ZZZ/rtems-docs/posix-compliance/wscript", line 17, in gen_posix_rst
    c.load(task.inputs[1].abspath())
  File "/Users/ZZZ/rtems-docs/posix-compliance/posix_rst.py", line 149, in load
    for row in data:
_csv.Error: iterator should return strings, not bytes (did you open the file in text mode?)

as you see above it ended however in some other unrelated error. (I am using Python 3.6.8).

But as regards your LaTeX question the crucial line

Scanning style file ./python.ist.......done (7 attributes redefined, 0 ignored).

is missing from console output in your project way of building the PDF. I obtained it by adding the -s python.ist.

By the way, your configure script checks for some LaTeX packages which are not used anymore by Sphinx (threeparttable for example, probably others).

I see there is some use of minted package, so quite a lot of customization, it would take me quite some time to understand really how your build is done. Maybe simply using the Sphinx provided Makefile via make -C <latexbuild> all-pdf could prove much easier, but attention then you have Latexmk dependency.

Waiting for feedback then I will close this as this is not issue of Sphinx.

(edit: all the above is done with current dev version of Sphinx 2.0.0+/3e0c65a245)

[kiwichris]

Ok, I checked your project

Thank you for the quick response and for taking a look at our issue and project. FYI moving our project to Sphinx has been really good. Last year we had Google Code In students sending in patches.

and your build process seems to bypass the Sphinx provided Makefile in build repertory, as a result latexmk is not used

Yes and this something I would like to fix. I would like to move to using your generated Makefile however my testing shows it fails on FreeBSD. If I use sphinx 1.8 our build fails in pdflatex and if I drop into the build directory and run gmake the output is:

WafError: 'error when calling latex' command exit status 1

(sphinx) ruru rtems-docs.git $ cd build/user/latex/
(sphinx) ruru latex $ gmake
latexmk -pdf -dvi- -ps-  'user.tex'
Latexmk: This is Latexmk, John Collins, 5 February 2015, version: 4.43a.
Latexmk: Index file 'user.idx' was written
Latexmk: Log file says output to 'user.pdf'
Latexmk: Index file 'user.idx' was written
Latexmk: Log file says output to 'user.pdf'
Latexmk: ===For rule 'latex', actual output 'user.pdf'
       ======appears not to match expected output 'user.dvi'
Latexmk: applying rule 'makeindex user.idx'...
Rule 'makeindex user.idx': The following rules & subrules became out-of-date:
      'makeindex user.idx'
------------
Run number 1 of rule 'makeindex user.idx'
------------
------------
Running 'internal xindy -L english -C utf8  -M sphinx.xdy  -o "user.ind" "user.idx"'
------------
Latexmk: calling xindy( -L english -C utf8 -M sphinx.xdy -o user.ind user.idx )
Can't exec "xindy": No such file or directory at (eval 9) line 14.
Latexmk: Errors, so I did not complete making targets
Collected error summary (may duplicate other messages):
  makeindex user.idx: Command for 'makeindex user.idx' gave return code -1
Latexmk: Use the -f option to force complete processing,
 unless error was exceeding maximum runs of latex/pdflatex.
gmake: *** [Makefile:33: user.pdf] Error 12

and makeindex is certainly not called as it should be i.e. it is lacking -s python.ist flag.

OK. This seems important.

I forced usage of this via hand-editing filr waflib/Tools/tex.py

....
	makeindex_fun,_=Task.compile_fun('${MAKEINDEX} -s python.ist ${SRCFILE}',shell=False)

Interesting.

makeindex_fun.doc="""
Execute the program makeindex
...

because I did not know how to set `MAKEINDEXFLAGS`.

....

[7/7] Compiling build/user/latex/user.tex
first pass on pdflatex
This is pdfTeX, Version 3.14159265-2.6-1.40.19 (TeX Live 2018) (preloaded format=pdflatex)
\write18 enabled.
entering extended mode
/Users/WWW/sphinxvenv/bin/pygmentize

calling makeindex
This is makeindex, version 2.15 [TeX Live 2018] (kpathsea + Thai support).
Scanning style file ./python.ist.......done (7 attributes redefined, 0 ignored).
Scanning input file user.idx....done (231 entries accepted, 0 rejected).
Sorting entries.....done (1883 comparisons).
Generating output file user.ind....done (329 lines written, 0 warnings).
Output written in user.ind.
Transcript written in user.ilg.

calling pdflatex
This is pdfTeX, Version 3.14159265-2.6-1.40.19 (TeX Live 2018) (preloaded format=pdflatex)
\write18 enabled.
entering extended mode
/Users/xxx/yyy/zzz/sphinxvenv/bin/pygmentize

calling pdflatex
This is pdfTeX, Version 3.14159265-2.6-1.40.19 (TeX Live 2018) (preloaded format=pdflatex)
\write18 enabled.
entering extended mode
/Users/xxx/yyy/zzz/sphinxvenv/bin/pygmentize

.... cut lines ....

[37/44] Processing generated-posix-compliance.rst: posix-compliance/posix_rst.py posix-compliance/RTEMS-Standards-Compliance-v4a.csv -> build/posix-compliance/generated-posix-compliance.rst
[38/44] Compiling common/latex/rtemsextrafonts.sty
[39/44] Compiling common/_static/favicon.ico
[40/44] Compiling common/_static/logo.png
/Users/TTT/UUU/rtems-docs/common/rtemsext.py:53: RemovedInSphinx30Warning: app.override_domain() is deprecated. Use app.add_domain() with override option instead.
app.override_domain(CustomStandardDomain)
/Users/XXX/YYY/sphinx/sphinx/builders/html.py:144: RemovedInSphinx30Warning: builder.script_files is deprecated. Please use app.add_js_file() instead.
ret += other

Waf: Leaving directory `/Users/xxx/yyy/rtems-docs/build'
Build failed
Traceback (most recent call last):
File "/Users/ZZZ/rtems-docs/.waf3-1.9.7-d27222240ebc8bcbca7fcd8f4ae914fb/waflib/Task.py", line 145, in process
ret=self.run()
File "/Users/ZZZ/rtems-docs/posix-compliance/wscript", line 17, in gen_posix_rst
c.load(task.inputs[1].abspath())
File "/Users/ZZZ/rtems-docs/posix-compliance/posix_rst.py", line 149, in load
for row in data:
_csv.Error: iterator should return strings, not bytes (did you open the file in text mode?)

as you see above it ended however in some other unrelated error. (I am using Python 3.6.8).

Agreed, this is our issue.

But as regards your LaTeX question the crucial line

Scanning style file ./python.ist.......done (7 attributes redefined, 0 ignored).

is missing from console output in your project way of building the PDF. I obtained it by adding the -s python.ist.

Thanks for this and as you say if the macro is defined in this file it explains what I am seeing. I will investigate.

By the way, your configure script checks for some LaTeX packages which are not used anymore by Sphinx (threeparttable for example, probably others).

Thanks. We have some users building on older version and some of this came from an old CentOS user. I should revisit this.

I see there is some use of minted package, so quite a lot of customization, it would take me quite some time to understand really how your build is done.

That is understandable. What you have given me here is really great. I inherited the minted parts and I have not looked into what it does.

Maybe simply using the Sphinx provided Makefile via make -C <latexbuild> all-pdf could prove much easier, but attention then you have Latexmk dependency.

I agree it would be nice.

Waiting for feedback then I will close this as this is not issue of Sphinx.

Agreed. I should have raised this as a question, I am sorry about that.

(edit: all the above is done with current dev version of Sphinx 2.0.0+/3e0c65a245)

OK. I am on a pip latest which seems to be 1.8.

[kiwichris]

Setting MAKEINDEXFLAGS flags works and for the record in waf you add:

ctx.env.append_value('MAKEINDEXFLAGS', ['-s', 'python.ist'])

Also I can build with xelatex as well.

Thank you for your help.

[jfbu]

For the record, with ctx.env.append_value('MAKEINDEXFLAGS', ['-s', 'python.ist']) added to waf and using a Python27 environment I build successfully all your docs with Sphinx 1.8, (and TeXLive 2018):

...
[111/111] Processing coverpage.js: wscript build/catalogue.xml common/coverpage/coverpage.js -> build/coverpage.js
Waf: Leaving directory `/Users/xxx/yyy/zzz/rtems-docs/build'
'build' finished successfully (2m41.005s)
(sphinxvenvpy27) (py27) 

I am busy today, regarding the

------------
Run number 1 of rule 'makeindex user.idx'
------------
------------
Running 'internal xindy -L english -C utf8  -M sphinx.xdy  -o "user.ind" "user.idx"'
------------

you reported on executing directly gmake, it can only happen if latexmkrc file was produced in a run of Sphinx configured for xelatex usage. Then indeed Sphinx 1.8 will configure latexmk to execute xindy, except if latex_use_xindy = False is added to conf.py, which however leads to some (silent) wrong indices with Unicode engine xelatex, if document contains Unicode characters (the files are not validly UTF-8 encoded when produced by makeindex). At my locale, I see rather this:

$ make all-pdf
latexmk -pdf -dvi- -ps-  'user.tex'
Latexmk: This is Latexmk, John Collins, 25 October 2018, version: 4.61.
=== TeX engine is 'pdfTeX'
Latexmk: Index file 'user.idx' was written
Latexmk: Log file says output to 'user.pdf'
=== TeX engine is 'pdfTeX'
Latexmk: Index file 'user.idx' was written
Latexmk: Log file says output to 'user.pdf'
Latexmk: ===For rule 'latex', the extensions differ between the
   actual output file 'user.pdf',
   and the expected output 'user.dvi'.
   This could arise from use of \pdfoutput in the source file,
   or from a configuration error
   A future version of latexmk should be able to make dynamically
   adjustments to deal with this problem
Latexmk: applying rule 'makeindex user.idx'...
Rule 'makeindex user.idx': The following rules & subrules became out-of-date:
      'makeindex user.idx'
------------
Run number 1 of rule 'makeindex user.idx'
------------
------------
Running 'makeindex -s python.ist  -o "user.ind" "user.idx"'
------------
...

You see no attempt at using xindy is done. There is some strange message of Latexmk above which I will have to check.

But then the build aborts with

! Package minted Error: You must invoke LaTeX with the -shell-escape flag.

See the minted package documentation for explanation.
Type  H <return>  for immediate help.
 ...                                              
                                                  
l.18 \fvset
           {
? X
No pages of output.
...

And indeed, to use minted package one needs to allow the pdflatex executable to run arbitrary system calls. For security reasons, Sphinx Makefile in latex build repertory does not add the shell escape flag.

Usage of minted package makes sense only if somehow some raw code ends up in Latex source files: Sphinx produces source files where code blocks have been handled by Pygmentize library. Using minted package means calling the Pygmentize library from inside the pdflatex runs (hence the need for shell escape). Someone heavily patched your use of Sphinx to bypass its standard way of handling code-blocks. Now this makes sense to use your fancy breakable boxes in PDF output because minted can be configured to use tcolorbox package ; but this definitely could be done also by redefining sphinxVerbatim environment in appropriate way. I am writing this all up a bit in the dark because today I can't examine more your set-up hence I don't exactly know... arrrghh wait, now looking at build/user/latex/user.tex I see no usage whatsoever of minted package, it all simply comes from those lines in

\usepackage{minted}
\fvset{
breaklines=true,
breakanywhere=true,
linenos,
numbersep=5pt,
gobble=0,
}

in file rtemsstyle.sty. Commenting out

% \usepackage{minted}
\fvset{
%breaklines=true,
%breakanywhere=true,
%linenos,
numbersep=5pt,
gobble=0,
}

then make clean all-pdf in build/user/latex/ succeeds. Notice though that minted package was used only to add these options to fancyvrb. The breaklines=true does something unneeded since Sphinx 1.4 or 1.5 when linebreaking was added, and linenos is natively implemented by numbers=left rather in fancyvrb (non-patched by minted). Here is screenshot from using

% \usepackage{minted} % We don't want that !
\fvset{
%breaklines=true,% NOT NEEDED
%breakanywhere=true,% not really needed
%linenos,% not native syntax of \fvset
numbers=left,% <<-- this is native syntax 
numbersep=5pt,
gobble=0,
}

In brief, I think your dependency on minted is weak. Once removed, it will be trivial to build PDFs using Sphinx native Makefile.

I need to examine more the xelatex situation. Can't today.

[kiwichris]

For the record, with ctx.env.append_value('MAKEINDEXFLAGS', ['-s', 'python.ist']) added to waf and using a Python27 environment I build successfully all your docs with Sphinx 1.8, (and TeXLive 2018):

Thanks. I have raised a bug report on this issue in the RTEMS project.

You see no attempt at using xindy is done.

I have disabled xindy in this patch ..

https://lists.rtems.org/pipermail/devel/2019-February/024753.html

There is some strange message of Latexmk above which I will have to check.

OK

And indeed, to use minted package one needs to allow the pdflatex executable to run arbitrary system calls. For security reasons, Sphinx Makefile in latex build repertory does not add the shell escape flag.

Oh OK.

Usage of minted package makes sense only if somehow some raw code ends up in Latex source files: Sphinx produces source files where code blocks have been handled by Pygmentize library. Using minted package means calling the Pygmentize library from inside the pdflatex runs (hence the need for shell escape). Someone heavily patched your use of Sphinx to bypass its standard way of handling code-blocks. Now this makes sense to use your fancy breakable boxes in PDF output because minted can be configured to use tcolorbox package ; but this definitely could be done also by redefining sphinxVerbatim environment in appropriate way. I am writing this all up a bit in the dark because today I can't examine more your set-up hence I don't exactly know... arrrghh wait, now looking at build/user/latex/user.tex I see no usage whatsoever of minted package, it all simply comes from those lines in

\usepackage{minted}
\fvset{
breaklines=true,
breakanywhere=true,
linenos,
numbersep=5pt,
gobble=0,
}

OK.

in file rtemsstyle.sty. Commenting out

Nice.

% \usepackage{minted}
\fvset{
%breaklines=true,
%breakanywhere=true,
%linenos,
numbersep=5pt,
gobble=0,
}

then make clean all-pdf in build/user/latex/ succeeds. Notice though that minted package was used only to add these options to fancyvrb. The breaklines=true does something unneeded since Sphinx 1.4 or 1.5 when linebreaking was added, and linenos is natively implemented by numbers=left rather in fancyvrb (non-patched by minted). Here is screenshot from using

Looks good.

% \usepackage{minted} % We don't want that !
\fvset{
%breaklines=true,% NOT NEEDED
%breakanywhere=true,% not really needed
%linenos,% not native syntax of \fvset
numbers=left,% <<-- this is native syntax 
numbersep=5pt,
gobble=0,
}

In brief, I think your dependency on minted is weak.

Thanks for this. I will look at removing it.

Once removed, it will be trivial to build PDFs using Sphinx native Makefile.

Excellent. Has the Makefile always been in Sphinx? I am wondering about users on older version of Sphinx.

I need to examine more the xelatex situation. Can't today.

No problem, I am on the road for a week. There is plenty here for me to sort out.

Thanks
Chris

[jfbu]

About

$ make all-pdf
latexmk -pdf -dvi- -ps-  'user.tex'
Latexmk: This is Latexmk, John Collins, 25 October 2018, version: 4.61.
......
Latexmk: ===For rule 'latex', the extensions differ between the
   actual output file 'user.pdf',
   and the expected output 'user.dvi'.
   This could arise from use of \pdfoutput in the source file,
   or from a configuration error

this is nothing special, it happens anytime one has done a manual run of pdflatex foo.tex and one then executes latemk via (g)make all-pdf which uses the Sphinx provided Makefile and latexmkrc configuration. I never saw that, because of course I never manually run pdflatex, I always trust the Sphinx LaTeX Makefile :).

About

Has the Makefile always been in Sphinx? I am wondering about users on older version of Sphinx.

It seems to have been added at fbe9ed5 in 2007, so yes, it has always been there in the latex build repertory... The move to use latexmk is more recent, and now the Makefile is a template which is configured according to latex_engine, to latex_use_xindy and also according to the document language in some circumstances. This means one can not use same build repertory with same Makefile for both pdflatex and xelatex. This is why I will have to examine when I get time how your project makes PDF builds using xelatex. But basically, you should not use xelatex if you can't run xindy: if the document has no Unicode, you don't need xelatex at all, and if it has Unicode in indexed terms, makeindex produces non-correct files for UTF-8, which xelatex handles silently (by luck) but wrongly (and lualatex breaks with an error).

[jfbu]

I will close this issue for now, as there is no bug to fix at Sphinx side.

[kiwichris]

About

$ make all-pdf
latexmk -pdf -dvi- -ps-  'user.tex'
Latexmk: This is Latexmk, John Collins, 25 October 2018, version: 4.61.
......
Latexmk: ===For rule 'latex', the extensions differ between the
   actual output file 'user.pdf',
   and the expected output 'user.dvi'.
   This could arise from use of \pdfoutput in the source file,
   or from a configuration error

this is nothing special, it happens anytime one has done a manual run of pdflatex foo.tex and one then executes latemk via (g)make all-pdf which uses the Sphinx provided Makefile and latexmkrc configuration. I never saw that, because of course I never manually run pdflatex, I always trust the Sphinx LaTeX Makefile :).

This makes sense.

About

Has the Makefile always been in Sphinx? I am wondering about users on older version of Sphinx.

It seems to have been added at fbe9ed5 in 2007, so yes, it has always been there in the latex build repertory... The move to use latexmk is more recent, and now the Makefile is a template which is configured according to latex_engine, to latex_use_xindy and also according to the document language in some circumstances.

Thanks, this is good to know.

This means one can not use same build repertory with same Makefile for both pdflatex and xelatex. This is why I will have to examine when I get time how your project makes PDF builds using xelatex.

I just pushed patches today that provides the ability to switch to use xelatex. It is driven off the conf.py for Sphinx and the latex_engine settings. It is set and you need to edit conf.py to make the change. I do not expect this to happen much if at all.

But basically, you should not use xelatex if you can't run xindy: if the document has no Unicode, you don't need xelatex at all, and if it has Unicode in indexed terms, makeindex produces non-correct files for UTF-8, which xelatex handles silently (by luck) but wrongly (and lualatex breaks with an error).

OK.

[jfbu]

@kiwichris waf has its own handling of LaTeX builds. Now, it can't replicate in a few dozen lines of Python code the expertise one finds in the Latexmk Perl script for compiling LaTeX sources in the right way. I don't know how to inject usage of Latexmk in waf build system, I guess one would have to disconnect all the waf native handling of LaTeX. Sphinx ships a Makefile which is needed to handle usage of platex+dvipdfmx which is not possible with waf (at least the version of your project) but this is not relevant to you (it is for Japanese documents), and some specific configuration of xindy, which is not relevant to you either as even with xelatex you will use latex_use_xindy = False, thus basically you only need from the Sphinx Makefile to extract the Latexmk call which will be latexmk -pdf -dvi- -ps- <filename> (where <filename> is name of tex file). This will use the file latexmkrc created by sphinx-build and which configures the makeindex call to use the oft mentioned -s python.ist flag. So you don't need the Sphinx LaTeX Makefile. But there should be some LATEXOPTS in the environment, which is expected by latexmkrc config file and defaults to empty from the Makefile (now unused). You may also want to execute for example latexmk -silent -pdf -dvi- -ps- foo for minimal console output.

Older version of Sphinx Makefile did not use Latexmk and had simple logic which was probably less refined than the waf way of handling LaTeX builds. But with Latexmk we have (since Sphinx 1.6.1) appropriate tool for the kind of LaTeX sources produced by Sphinx, even with extensions beyond Sphinx proper (I trust). This adds a Perl dependency but incorporates a more sophisticated LaTeX build set-up than the waf provided one. If one does multiple times Latexmk after changes in tex file, it knows how to minimize the number of pdflatex runs. (it also computes hash sums and has its own auxiliary files to detect what changed; in particular it analyses the Latex log file in ways which I doubt waf build system does).