Fix #656: Sphinx 5.1.0 has modified some internal LaTeX macros

[jfbu]

Fix #656

I hesitated between leaving comments inside the LATEX_PREAMBLE or rather move them out as Python comments. In the end I dif leave a latex comment about \nbsphinxincludegraphics but it should probably be removed.

This commit ensures compatibility with Sphinx 5.1.0 which has changed some internal macros related to rendering of code-boxes. The legacy code is left exactly identical when it is detected Sphinx is < 5.1.0. The new code opens up possibility to use some of the 5.1.0 added extensions regarding separate customizability of border widths and paddings.

I tested it by compiling the nbsphinx.pdf with Sphinx at least at sphinx-doc/sphinx@cdde699 which adds to Sphinx 5.1.0 a fix related to code-blocks but unrelated to current nbsphinx (which does not use verbatimwithframe=false). The pdf output looks ok.

About this:

I will provide a PR ... to refactor nbsphinxfancyoutput in a way not using internal Sphinx macros and keeping exact same behaviour otherwise.

It proved elusive for this task to not use any internal Sphinx LaTeX macro (i.e. with @ in its name), if we want to benefit from the styling and properties of Sphinx Verbatim in a custom way (in particular without actually using sphinxVerbatim environement itself for this type of cell).

---

Additional notes not related to this PR, for archival.

side notes:

There were some numerous slightly annoying LaTeX iterated warnings, urelated to this PR:

Package textcomp Warning: Symbol \textasciigrave not provided by
(textcomp)                font family zlmtt in TS1 encoding.
(textcomp)                Default family used instead on input line 7252.

They all seem to follow some earlier warning:

[10] [11] [12] [13] [14] [15]

Package textcomp Warning: Symbol \textasciigrave not provided by
(textcomp)                font family pplj in TS1 encoding.
(textcomp)                Default family used instead on input line 1267.

[16] [17] [18]
...[snip]...
LaTeX Font Warning: Font shape `T1/cmtt/l/n' undefined
(Font)              using `T1/cmtt/m/n' instead on input line 3591.

(/usr/local/texlive/2022/texmf-dist/tex/latex/base/ts1cmtt.fd)

LaTeX Font Warning: Font shape `TS1/cmtt/l/n' undefined
(Font)              using `TS1/cmtt/m/n' instead
(Font)              for symbol `textasciigrave' on input line 3591.

I am using latest TeXLive 2022.

Other latex warnings:

[25 <./python_logo.pdf> <./main-logo.pdf

pdfTeX warning: pdflatex (file ./main-logo.pdf): PDF inclusion: multiple pdfs w
ith page group included in a single page
> <./fe1ba2728605169d4270646e0702b6ef37f4aad5.png>] [26 <./6e42ca0abdf1647b0e5b
794e9605d81a09e28d28.png>] [27]

and

LaTeX Warning: Hyper reference `markdown-cells:' on page 28 undefined on input 
line 2460.


LaTeX Warning: Reference `markdown-cells:' on page 28 undefined on input line 2
460.

As I use Python 3.10.4 I received this Python warning:

.venv310/lib/python3.10/site-packages/jupytext/cell_reader.py:20: UserWarning: Sphinx Gallery in version 0.11.0 is not supported by Jupytext. Please use sphinx-gallery<=0.7.0 instead. If that is an issue, feel free to report it at https://github.com/mwouts/jupytext/issues, or even better, prepare a PR to handle the new signature of sphinx_gallery.notebook.rst2md.

Edit:

I forgot to mention that the PDF output contains some stderr cell at bottom of page 35:

.venv310/lib/python3.10/site-packages/IPython/core/
􏰀→formatters.py:343: FutureWarning: In future versions `DataFrame.to_latex` is␣ 
􏰀→expected to utilise the base implementation of `Styler.to_latex` for formatting␣ 
􏰀→and rendering. The arguments signature may therefore change. It is recommended␣ 
􏰀→instead to use `DataFrame.style.to_latex` which also contains additional␣
(page break here with continuation hint)
(at top of next page after continued from previous page hint)
􏰀→functionality. return method()

which I mention in passing.

It also illustrates the Sphinx 5.1.0 added feature that in case a long code line wraps,

it is able to accept a page break.

Formerly it would have stayed at bottom of first page as an unbreakable block.

[mgeier]

Thanks a lot for this PR, it's very much appreciated, as always!

I hesitated between leaving comments inside the LATEX_PREAMBLE or rather move them out as Python comments. In the end I dif leave a latex comment about \nbsphinxincludegraphics but it should probably be removed.

I think it's good to have that comment, it's very helpful for people like me, who don't really understand all that (La)TeX magic!

I don't really mind if it's a Python or a LaTeX comment.

This commit ensures compatibility with Sphinx 5.1.0 which has changed some internal macros related to rendering of code-boxes. The legacy code is left exactly identical when it is detected Sphinx is < 5.1.0. The new code opens up possibility to use some of the 5.1.0 added extensions regarding separate customizability of border widths and paddings.

That's great!

I tested it by compiling the nbsphinx.pdf with Sphinx at least at sphinx-doc/sphinx@cdde699 which adds to Sphinx 5.1.0 a fix related to code-blocks but unrelated to current nbsphinx (which does not use verbatimwithframe=false). The pdf output looks ok.

Yes, I also checked it, everything looks fine.

About this:

I will provide a PR ... to refactor nbsphinxfancyoutput in a way not using internal Sphinx macros and keeping exact same behaviour otherwise.

It proved elusive for this task to not use any internal Sphinx LaTeX macro (i.e. with @ in its name), if we want to benefit from the styling and properties of Sphinx Verbatim in a custom way (in particular without actually using sphinxVerbatim environement itself for this type of cell).

TBH, I don't really understand the details, but I'm glad that you are fixing it, however you see fit!

Additional notes not related to this PR, for archival.
side notes:

There were some numerous slightly annoying LaTeX iterated warnings, urelated to this PR:

Package textcomp Warning: Symbol \textasciigrave not provided by
(textcomp)                font family zlmtt in TS1 encoding.
(textcomp)                Default family used instead on input line 7252.

They all seem to follow some earlier warning:

[10] [11] [12] [13] [14] [15]

Package textcomp Warning: Symbol \textasciigrave not provided by
(textcomp)                font family pplj in TS1 encoding.
(textcomp)                Default family used instead on input line 1267.

[16] [17] [18]
...[snip]...
LaTeX Font Warning: Font shape `T1/cmtt/l/n' undefined
(Font)              using `T1/cmtt/m/n' instead on input line 3591.

(/usr/local/texlive/2022/texmf-dist/tex/latex/base/ts1cmtt.fd)

LaTeX Font Warning: Font shape `TS1/cmtt/l/n' undefined
(Font)              using `TS1/cmtt/m/n' instead
(Font)              for symbol `textasciigrave' on input line 3591.

I am using latest TeXLive 2022.

I don't really understand any of this. I'm just ignoring it. If you happen to know an easy solution, please let me know!

Other latex warnings:

[25 <./python_logo.pdf> <./main-logo.pdf

pdfTeX warning: pdflatex (file ./main-logo.pdf): PDF inclusion: multiple pdfs w
ith page group included in a single page
> <./fe1ba2728605169d4270646e0702b6ef37f4aad5.png>] [26 <./6e42ca0abdf1647b0e5b
794e9605d81a09e28d28.png>] [27]

But I guess the "multiple pdfs" are python_logo.pdf and main-logo.pdf (which are auto-generated from SVG files with rsvg-convert).

Maybe the converter settings could be changed to avoid this?

Do you know what "with page group" means?

LaTeX Warning: Hyper reference `markdown-cells:' on page 28 undefined on input 
line 2460.


LaTeX Warning: Reference `markdown-cells:' on page 28 undefined on input line 2
460.

I don't know where this comes from, but I'll search for markdown-cells. I don't know if the final colon is part of the reference or not?

As I use Python 3.10.4 I received this Python warning:

.venv310/lib/python3.10/site-packages/jupytext/cell_reader.py:20: UserWarning: Sphinx Gallery in version 0.11.0 is not supported by Jupytext. Please use sphinx-gallery<=0.7.0 instead. If that is an issue, feel free to report it at https://github.com/mwouts/jupytext/issues, or even better, prepare a PR to handle the new signature of sphinx_gallery.notebook.rst2md.

Yes I've seen that before.

It's annoying because it doesn't affect nbsphinx because we are only using the CSS from Sphinx Gallery and none of the Python stuff.

I forgot to mention that the PDF output contains some stderr cell at bottom of page 35:

.venv310/lib/python3.10/site-packages/IPython/core/
􏰀→formatters.py:343: FutureWarning: In future versions `DataFrame.to_latex` is␣ 
􏰀→expected to utilise the base implementation of `Styler.to_latex` for formatting␣ 
􏰀→and rendering. The arguments signature may therefore change. It is recommended␣ 
􏰀→instead to use `DataFrame.style.to_latex` which also contains additional␣
(page break here with continuation hint)
(at top of next page after continued from previous page hint)
􏰀→functionality. return method()

which I mention in passing.

Yeah, I've seen that, I have to look into it.

It also illustrates the Sphinx 5.1.0 added feature that in case a long code line wraps, it is able to accept a page break.
Formerly it would have stayed at bottom of first page as an unbreakable block.

That's great!

[jfbu]

Other latex warnings:

[25 <./python_logo.pdf> <./main-logo.pdf

pdfTeX warning: pdflatex (file ./main-logo.pdf): PDF inclusion: multiple pdfs w
ith page group included in a single page
> <./fe1ba2728605169d4270646e0702b6ef37f4aad5.png>] [26 <./6e42ca0abdf1647b0e5b
794e9605d81a09e28d28.png>] [27]

But I guess the "multiple pdfs" are python_logo.pdf and main-logo.pdf (which are auto-generated from SVG files with rsvg-convert).

Maybe the converter settings could be changed to avoid this?

Do you know what "with page group" means?

I did not as I don't practice graphics inclusions often enough... turns out this tex.sx question i.e. https://tex.stackexchange.com/questions/76273/multiple-pdfs-with-page-group-included-in-a-single-page-warning has by luck some quite informative answer.

So whenever you include 2 pdfs in the same page you may encounter the problem, depending how the pdf's were produced... one gets indeed the warning from using

\documentclass{article}

\usepackage{graphicx}

\begin{document}

\noindent\includegraphics{{python_logo}.pdf}

\noindent\includegraphics{{main-logo}.pdf}

\end{document}

test file (refereed to as testincludegraphics.tex below).

I tested that adding

\ifdefined\pdfsuppresswarningpagegroup
   \pdfsuppresswarningpagegroup=1 % cf https://tex.stackexchange.com/a/78020
\fi

does let the warning go away. It seems from some other answers at the tex.sx question that although PDFTeX does a bad job at emitting most of the time the warning for no real reason in some cases though the output has problems and warning is justified.

I tested with luatex and there seems to be no warnings with my test file. I then made the mistake to test above file with xetex (xelatex testincludegraphics) and

xdvipdfmx:fatal: pdf_link_obj(): passed invalid object.

No output PDF file written.
(see the transcript file for additional information)
Error 256 (driver return code) generating output;
file testincludegraphics.pdf may not be valid.
Transcript written on testincludegraphics.log.

This already happens with the above test file reduced to trying to include either one of the two files so is another issue.

Sorry for gory details, I hate this too, but only to be complete

• $ ps2pdf main-logo.pdf produces a file main-logo.pdf.pdf which I can use (alone) in my test file above with xelatex and produced pdf seems to be fine (trying main-logo.pdf leads to above build error)
• but attempting executing ps2pdf (which invokes the ghostscript toolchain and which I use sometimes to reduce pdf sizes) python-logo.pdf gives

$ ps2pdf python_logo.pdf
   **** Error: Encountered 'obj' while expecting 'endobj'.
               Treating this as a missing 'endobj', output may be incorrect.
   **** Error: Encountered 'obj' while expecting 'endobj'.
               Treating this as a missing 'endobj', output may be incorrect.
   **** Error: Encountered 'obj' while expecting 'endobj'.
               Treating this as a missing 'endobj', output may be incorrect.
   **** Error: Encountered 'obj' while expecting 'endobj'.
               Treating this as a missing 'endobj', output may be incorrect.

However, then xelatex does not complain with \includegraphics{{python_logo.pdf}.pdf}.

Anyway it looks as if something is fishy with file python_logo.pdf at least Ghostscript complains.

And a xelatex run breaks with error status 1 when one tries to use \includegraphics{{python_logo}.pdf}.

Sorry for long, I hate this too when it turns out this way (and I will not try to report this "upstream"). As has been recently aptly commented by some passer-by:

You guys are doing some crazy work there. But is this not insane? Are there no proper solutions to getting SVGs into PDFs in freakin' 2022?

[jfbu]

The "Font warnings" are another can of worms. I only looked briefly (well not so briefly sadly and worst is that I did not complete fully investigation) at

Package textcomp Warning: Symbol \textasciigrave not provided by
(textcomp) font family zlmtt in TS1 encoding.
(textcomp) Default family used instead on input line 3594.

As far as I can tell for now this is triggered by

• \usepackage[light,scaled=1.05]{zlmtt} in preamble,
• together with some backticks ` in verbatim cells, which translate into \textasciigrave,
• and the textcomp package erroneously reports that the backtick character is not available in TS1 encoding for zlmtt font family, whereas in fact it is available at slot 96.

At thsi stage my guess is that textcomp is not aware of relatively recent zlmtt font package (2014), but I pushed investigation only half-way so maybe the above is all wrong.

And some other things such as

[10] [11] [12] [13] [14] [15]

Package textcomp Warning: Symbol \textasciigrave not provided by
(textcomp) font family pplj in TS1 encoding.
(textcomp) Default family used instead on input line 1267.

[16] [17] [18]
I did not look at yet.

These things can usually be ignored if some correct glyph ends up in PDF, although sometimes it needlessly enlarge PDF size from loading a substitute font.

(all of the above with "font" meaning one of those famous TeX fonts with only room for 128 or 256 glyphs each.)

[jfbu]

Package textcomp Warning: Symbol \textasciigrave not provided by
(textcomp) font family pplj in TS1 encoding.
(textcomp) Default family used instead on input line 1267.

Apart from the LaTeX font headaches, I checked the source and it seems this originates in that part of usage.ipynb:

    "You migh have to change the line `docs/` to `docs/source` in case that you have this folder structure, which is the default when using `sphinx-quickstart. "

where there is seemingly a missing backtick avec sphinx-quickstart. In the LaTeX file the one at front becomes \textasciigrave{} mark-up. This mark-up somehow triggers the latex font warning, which currently I don't understand as I do see the glyph at slot 96 in a font table of that (TeX) font (but LaTeX font things are so complicated I may be missing something).

---

I don't really understand any of this. I'm just ignoring it. If you happen to know an easy solution, please let me know!

I was told an easy solution already more than a year ago at latex3/latex2e#478 but I had forgotten about it (and opened earlier today a new isse latex3/latex2e#905 at their project, but it is almost a duplicate).

Thus:

The LaTeX warnings related to \textasciigrave can be silenced adding this in the preamble:

\DeclareTextSymbolDefault{\textasciigrave}{TS1}

It may be worthwile to add

\tracinglostchars=3

to preamble. This causes a LaTeX build error in case a font is missing a character glyph... (since TeXLive 2021 release), and is mostly very useful in the context of Unicode engines (xetex, luatex) and OpenType fonts.

[jfbu]

LaTeX Warning: Hyper reference `markdown-cells:' on page 28 undefined on input
line 2460.

LaTeX Warning: Reference `markdown-cells:' on page 28 undefined on input line 2
460.

I don't know where this comes from, but I'll search for markdown-cells. I don't know if the final colon is part of the reference or not?

Yes the colon is part of the reference but there is no target with this name. This originates in

    "It's also possible to create a\n",
    "[link to the beginning of the current page](#),\n",
    "by simply using a `#` character:\n",
    "\n",

from markdown-cells.ipynb. I guess nbconvert creates the mark-up? I see there exists two related labels in the tex mark-up:

\section{Markdown Cells}
\label{\detokenize{markdown-cells:Markdown-Cells}}\label{\detokenize{markdown-cells::doc}}

so it should be either one of them I guess. Assuming "current page" in this context is web page for HTML output and in the LaTeX output it should go to start of included notebook? (not sure).

---

This looks more like some upstream issue (nbconvert?) but I am not sure as I don't have clear view of toolchain.

A LaTeX-hacker approach would be to modify \nbsphinxstartnotebook macro to add some \phantomsection and a \label using the name of notebook file followed by a colon... this would at least fix this one.

[mgeier]

Thanks a lot for the fix and for your detailed analysis. I've created separate issues from your comments above.