Word wrap in codeblocks does not work if a long line is enclosed in quotations marks

[derdennis]

Dear all,

we encountered a problem with one of our codeblocks where a quite long line is enclosed in simple quotations marks ": The line is flowing out of the codebox and off the page in the pdf output:

The minimal example looks like this in the rst-source code:

AAAA
====


.. code-block:: html

    <div class="nachrichten infobox {if (count($meine_wiedervorlagen//ROW) > 0) then 'borderBottom' else ()}">

Which turns into this LaTeX via make latex:

\chapter{AAAA}
\label{\detokenize{index:aaaa}}\label{\detokenize{index:welcome-to-foo-s-documentation}}
\fvset{hllines={, ,}}%
\begin{sphinxVerbatim}[commandchars=\\\{\}]
\PYG{p}{\PYGZlt{}}\PYG{n+nt}{div} \PYG{n+na}{class}\PYG{o}{=}\PYG{l+s}{\PYGZdq{}nachrichten infobox \PYGZob{}if (count(\PYGZdl{}meine\PYGZus{}wiedervorlagen//ROW) \PYGZgt{} 0) then \PYGZsq{}borderBottom\PYGZsq{} else ()\PYGZcb{}\PYGZdq{}}\PYG{p}{\PYGZgt{}}
\end{sphinxVerbatim}

Beispiel-Konfiguration für \sphinxcode{\sphinxupquote{HTML-Template}}:

and like this in the pdf output via pdflatex SIPAMBestPracticesDoku.tex:

Removing the quotations marks " fixes the issue in the PDF but leaves us without syntax highlighting and a warning (index.rst:12: WARNING: Could not lex literal_block as "html". Highlighting skipped.) at make latex:

rst-source without quotation marks:

AAAA
====


.. code-block:: html

    <div class=nachrichten infobox {if (count($meine_wiedervorlagen//ROW) > 0) then 'borderBottom' else ()}>

LaTeX without quotation marks:

\chapter{AAAA}
\label{\detokenize{index:aaaa}}\label{\detokenize{index:welcome-to-foo-s-documentation}}
\fvset{hllines={, ,}}%
\begin{sphinxVerbatim}[commandchars=\\\{\}]
\PYGZlt{}div class=nachrichten infobox \PYGZob{}if (count(\PYGZdl{}meine\PYGZus{}wiedervorlagen//ROW) \PYGZgt{} 0) then \PYGZsq{}borderBottom\PYGZsq{} else ()\PYGZcb{}\PYGZgt{}
\end{sphinxVerbatim}

Beispiel-Konfiguration für \sphinxcode{\sphinxupquote{HTML-Template}}:

PDF without quotation marks:

As suggested by @jfbu over in #3110 who could not reproduce my problem, I am opening this new issue.

Because @jfbu suspected a problem with Pygmentize I upgraded it from 2.2.0 to the latest 2.3.1 but this did not do the trick.

The used versions of all the software are:

• Sphinx Version: 1.7.1
• Python Version: Python 2.7.14
• Pygments Version: 2.2.0 / 2.3.1
• PDFLaTeX Version: MiKTeX-pdfTeX 2.9.6607 (1.40.18) (MiKTeX 2.9.6600 64-bit)
• OS: Windows 10

I would be very happy to provide additional information and try out proposed solutions.

Thank you for your great work, cheers

Dennis

[tk0miya]

Could you try latest Sphinx release? On my local, it seems working fine.

[derdennis]

Thank you for your suggestion. I upgraded to Sphinx 1.8.3 tried the build and got the same result.

[jfbu]

Thanks for detailed report. At my locale if I create a file snippet.html with these contents

<div class="nachrichten infobox {if (count($meine_wiedervorlagen//ROW) > 0) then 'borderBottom' else ()}">

and then execute pygmentize -l snippet.html I get this:

This is with

testpygmentize$ pygmentize -V
Pygments version 2.2.0, (c) 2006-2017 by Georg Brandl.

I thus realized my pygmentize was not updated in the virtualenv I use for Sphinx. I have now updated to 2.3.1. Console output looks now like this:

but make latexpdf produces same PDF with no problem.

To debug this, it would be nice if you could make the small project containing only this code-block and zip the _build/latex repertory from make latex or sphinx-build -b latex and attach it here.

The LaTeX code you post is exactly as expected.

[jfbu]

Inside the LaTeX build repertory where the tex file is produced there should be a sphinxhighlight.sty file. Can you please post it ?

[derdennis]

Thank you very much for your help @jfbu, it is really appreciated, merci beaucoup.

I tried my pygmentize and it looks like this:

The zipped up _build/latex is here and contains the mentioned sphinxhighlight.sty:

latex.zip

Please let me know if you need any other information. Thank you again.

[jfbu]

Yes that's it. If I dump my sphinxhighlight.sty into your build repertory it works fine.

After sorting lines I can see the two files differ but I don't have time right now to investigate more. I am posting here the sphinxhighlight.sty I get at my locale. Notice that it is normal order of lines differ (I think) but not more differences.

I am doing this with

Python version: 2.7.15 (CPython)
Docutils version: 0.14 
Pygments version: 2.3.1
Jinja2 version: 2.10

You can fix temporarily your issue perhaps (I am not sure if your full document would need a bigger sphinxhighlight.sty and short on time right now) by using

latex_additional_files = ['sphinxhighlight.sty']

with mine added to source of your project.

[jfbu]

I am SOOOO stupid!

Of course you are using a custom Pygments style.

According to here when the style specifies a background color for some elements the LaTeX formatter uses \colorbox. This makes line breaking completely impossible.

I tested all styles, and only three cause the problem on your MWE: pastie, murphy and colorful.

I thus guess you have

pygments_style = 'colorful'

in your conf.py. Remove that line or use another style.

[jfbu]

Notice that can override that pygments_style for LaTeX builds only, e.g.

sphinx-build -b latex . <latexbuilddir here> -D 'pygments_style=sphinx'

(you will then get same syntax coloring as if no pygments_style config in conf.py), or using make syntax (Unix system)

make latex O="-D pygments_style=sphinx"

The latter allows make latexpdf, but for the former you should move to latex build repertory and build the PDF there. As you are on Windows which I can't test, I am not sure how that goes exactly. And one pdflatex is not enough.

Personally on Unixen I do either make -C <latexbuilddir> all-pdf or make latexpdf to start with and I pass extra config options as shown above.

To get available styles (apart from "sphinx" one)

$ pygmentize -L "styles"

[derdennis]

Dear @jfbu, you are obviously not stupid. On the contrary, your solution is on point:

We indeed used the pygments_style = 'colorful'. After switching the option to friendly the word wrap in the PDF codebox works as expected:

Encore une fois, merci beaucoup, thank you once again.

[jfbu]

At 3.5.0 line wrapping in code-blocks even for tokens on a colored background will be possible (#8854). However this will need to be added to conf.py:

latex_elements = {
    'sphinxsetup': "verbatimforcewraps",
}

With the example of this ticket:

.. code-block:: html

   <div class="nachrichten infobox {if (count($meine_wiedervorlagen//ROW) > 0) then 'borderBottom' else ()}">

and also

pygments_style = 'colorful'

(to reproduce the original problem), the following output is obtained:

This is with Pygments version: 2.7.4 and 3.x branch of Sphinx (3.5.0 upcoming release).

(edit actually the screen shot is fom a build with master branch which has not same font as 3.x branch for code-blocks, sorry for imprecision)

In LaTeX, colored boxes are unbreakable, so the effect is achieved by treating each character individually. Hnce the hard-wrap as seen above. (also some PDF viewers might have problems with many successive small colored boxes and show artefact lines at joining)

Generally speaking the hard-wrap (if activated via the conf.py setting as above) is triggered only in case of extreme necessity, i.e. for something unbreakable longer than a line (this can be configured).

Here it is first realized by the latex code that something very long has no break point (and this is aftereffect of being in a \colorbox), then in second stage, the Pygments mark-up is re-parsed so that each character is separated (and its own \colorbox) and breaks can happen anywhere.

One may wish a less radical second stage regarding colored tokens, and that it could still do the linebreaks at spaces only. I will open a feature request :)