literalinclude xml: TeX capacity exceeded, sorry [main memory size=3000000].

[petsuter]

A document with just the following fails:

.. literalinclude:: large.xml
    :language: xml
..

where large.xml is about 1000 lines of this:

<a>
                <e f="g" h="i" />
                <e f="g" h="i" />
                <e f="g" h="i" />
                <e f="g" h="i" />
                ... same line repeated ~1000 times ...
</a>

(for some reason the 16 spaces indentation are important to trigger this)

The error is:

...
("C:\Program Files\MiKTeX 2.9\tex\latex\psnfss\t1pcr.fd")
Underfull \vbox (badness 10000) detected at line 1156

Underfull \vbox (badness 10000) detected at line 1156
! TeX capacity exceeded, sorry [main memory size=3000000].
\spx@VerbatimFBox ...ip \fboxsep \copy \@tempboxa
                                                  \vskip \fboxsep }\kern -\w...
l.1156 \end{Verbatim}

!  ==> Fatal error occurred, no output PDF file produced!
Transcript written on SNSNetworkProtocol.log.

Build finished; the PDF files are in _build/latex.

[jfbu]

I have tested it on 1.4.8 (and on master branch), and I can go up to 1487 lines, there is an error starting with 1488 lines, but it is not one on main memory and it is slightly different (and more depressing):

./temp.tex:1551: Dimension too large.
\fb@put@frame ...p \ifdim \dimen@ >\ht \@tempboxa 
                                                  \fb@putboxa #1\fb@afterfra...l.1551 \end{sphinxVerbatim}

This comes from use of package framed. This package works by first piling up all contents in a big virtual TeX box, then it splits it into smaller pieces. The maximal dimension which TeX can handle is (an epsilon less) than 16384pt (where pt is the TeX point, 72.27 to an inch), if we divide this by 1488 we get about 11.01pt. In a default 10pt document the baseline skip is 12pt, but code-blocks actually issue \small which gives a baseline skip of exactly 11pt. This explains it.

edit: 1488+the two <a> and </a> lines is 1490 and

> (2**30-1)/(1489.0*65536);
                                  11.00335795

> (2**30-1)/(1490.0*65536);
                                  10.99597314

However I don't understand your memory error. If it really is the first error, it would be interesting to know which version of Sphinx you use and how up-to-date is your MikTeX install.

The dimension too big error is dramatic and there is nothing we can do about it apart from finding a completely different way to handle breakable frames.

memo: The package tcolorbox is not an option for Sphinx currently because it is too recent, and it is impossible to ship it as this would mean ship also TikZ which is very big with many files, the only way is that user has an up-to-date TeX installation. But its documentation has some promising piece:

The maximal total height of the upper and of the lower part of normal breakable
tcolorboxes is about 65536pt (ca. 2300cm) apiece. If such a part gets longer, the output
will get buggy without warning. For very oversized boxes which are longer than 65536pt,
use the unlimited value for /tcb/breakable P. 353 . With the unlimited setting, the
applied algorithm has (virtually) no height limit for boxes, but very likely the compiler
memory will have to be increased for boxes longer than 300 pages (depending on compiler
settings and box content). But it is recommended to use unlimited for critical large boxes
only.

This looks promising because 23m is already 4 times the limitation of "framed" which is the TeX maximal dimension (5.75m).

However you error is otherwise, on TL2016, the TeX main memory size is at 5000000, yours is set at 3000000, I recall that was old setting for TL too, but don't remember which year.

I am not familiar with MikTeX but this explains how to increase the main memory size. This may be your better option, if that's really the problem.

But I am afraid you will hit the dimension too large error next. Maybe you are lucky and the literal include is small enough to not trigger it.

But if you have too many lines your only option then is to split the literal-include into smaller pieces (note that in my test the output occupies about 26 usletter pages).

Notice that Sphinx 1.0 already used "framed.sty" package to draw the frame around literal includes or code blocks. Thus this problem has been there for a long time.

With Sphinx 1.5 you will be able modify the font size used for code blocks. Insert this

.. raw:: latex

   \fvset{fontset=\footnotesize}

before the location of the literal include. You can reset it to \small after. Or make it the document default by inserting the command in the 'preamble' key. I tested it and it worked as expected, as long as the produce things does not occupy more than about 26 usletter-sized sheets with the default Sphinx layout of 1in margins, it works.

[jfbu]

For info, I tested tcolorbox and could typeset, replacing in the tex file thesphinxVerbatim environment by (this example had no special characters in need of being escaped from TeX)

\small\ttfamily\begin{tcolorbox}[breakable]\obeylines\obeyspaces...\end{tcolorbox}

up to about 5950 lines, i.e. about 105 usletter pages. I added

\usepackage{tcolorbox}\tcbuselibrary{breakable}

in the preamble.

The problem is that this LaTeX package needs TL2012 at least, has had quite active development since and needs TikZ which itself became really stable about two years ago (it was probably stable already but at least in some directions there were breaking change, like the atan function, but that may be irrelevant to use by Sphinx).

dev notes: adopting tcolorbox would mean for Sphinx to require its users to use TL2013, and possibly at least TL2014 or even TL2015, to have a stable base. It could be possible to provide an option which would tell sphinx.sty to use "tcolorbox" in replacement of "framed", but the code would become even more difficult to maintain and anyhow Travis-CI must work currently on TL2009.

Ultimately if at some point in the future tcolorbox replaces framed, then the Sphinx LaTeX code could become much easier to maintain, as currently framed is used by sphinx.sty in a way requiring expert level at LaTeX.

[petsuter]

Thanks for investigating.

If it really is the first error, it would be interesting to know which version of Sphinx you use and how up-to-date is your MikTeX install.

I installed both Sphinx 1.4.8 (pip install Sphinx) and MikTeX 2.9 today, so should be up-to-date.

Is TL referring to TeX Live? Is that recommended for Sphinx over MikTeX? I don't know much about LaTeX, I just want a way to create a PDF.

[jfbu]

Yes, TL refers to TeX Live. No, it is not especially recommended over MikTeX, you should be fine with MikTeX. It happens that TeX Live has a higher default memory setting for TeX it seems, but you can increase it in MikTeX too, as explained in the link I referred too (I don't have any experience with Windows, hence don't know about MikTeX).

Sorry for having put much information in my two posts above.

A feature of TeX Live is that it gets updated from May to May, then it gets frozen, and you must re-install it entirely. They rarely update binaries in-between, but only the macro packages. If you add to this that Ubuntu used to distribute already old versions, you end up with situation that a Linux user of Sphinx may well have TeXLive 2009, in its initial release of June 2009, that it never was updated after that, and anyhow even fully updated contains macro packages of April-May 2010 at the best.

I did not reproduce the memory error you reported, possibly because of the higher memory setting in TeXLive.

[jfbu]

I have reproduced your issue with Sphinx 1.4.8, after having re-configured TeXLive 2016 to use a reduced TeX memory setting at 3000000. Maybe I should have started with that !

The good news is that if you find the way to tell MikTeX to regenerate the format files with a memory setting increased at 5000000 words of memory, your problem will be solved !

(because you have a literalinclude of only about 1000 lines)

[jfbu]

The second good news is that there is no problem with the master branch of Sphinx (but there is on the stable branch). Hence even if you don't manage to increase the TeX main memory, you have another escape which would be use the current master branch.

[jfbu]

ah sorry no, it worked on master branch because I had conf.py configured to use xetex (whose memory setting had not been reduced), due to some remnant of other test. With pdflatex there is still the problem also on master branch (I was wondering what had changed).

Well, to sum up you must increase the memory of TeX, and as long as your literal includes do not exceed about 1500 lines (about 26 pages) you should be ok.

[tk0miya]

@jfbu Are there anything to do on sphinx side?
If none, I would like to close this.

[jfbu]

@tk0miya there is nothing to do on Sphinx side currently. The original issue at 1000 lines was explained from a too low memory setting in MikTeX.

Investigation revealed an intrinsic limitation currently in Sphinx that code-blocks or literal includes must each have less than about 1500 lines. In future we could allow up to about 6000 lines, by refactoring sphinx.sty to use package tcolorbox rather than package framed. But package tcolorbox requires recent up-to-date and complete TeX installation.

It seems with tcolorbox one can go even beyond that extended range via its "unlimited" style option, but this probably also requires from user to extend TeX's memory as well.

[tk0miya]

Thanks,
About tcolorbox, let's discuss after upgrading TeXLive :-)

[stephenfin]

I'd like to reopen this issue because we've been seeing this issue quite often in OpenStack. I also have a very simple reproducer, described below. We don't control the memory limit everywhere docs are built (primarily in CI environments, though I suspect readthedocs will have the same issue) so this isn't an option. I wonder if we could do one of the following:

1. Warn the user if they've got a literalinclude that generates more than 1500 lines
2. Do (1) but make the max size a configuration option (so people can override it in environments where they can set the memory limit)
3. Break these large documents into multiple boxes and use a negative margin to overlap them (is that a thing in LaTeX?)
4. Use tcolorbox instead of framed, since it's been 3 years and that sounds like it might resolve the issue for us?

(@jfbu, I suspect this is mostly aimed at you 😄)

[stephenfin]

Forgot the reproducer (attachment). For reference, I'm just creating a new project using sphinx-quickstart, then adding this large source file to _static and including it via .. rawinclude

[tjchhajed]

I encountered the same issue with large files in our Sphinx documentation when converting to PDF. Specifically, including a requirements.txt file with more than 1365 lines using the literalinclude directive causes the PDF build to fail.

As a workaround, we had to split the inclusion into multiple literalinclude directives, each covering fewer than 1365 lines. This solution works but is not ideal for maintaining large or frequently changing files.

Version we use:
sphinx=7.3.7

[jfbu]

@tjchhajed Thanks for reviving this issue. We must solve it, I agree.

Thinking about it I see now a strategy which we could implement and solve this. The idea is simply to let the LaTeX file contain code-blocks contents split in successive chunks with say at most 900 lines each, which is very easy to do on the Python side, but must be inside suitable LaTeX mark-up with associated TeX code which is definitely not a two-minutes job. (simpler aspects will be to take care of codeline numbers, possible highlights, much more difficult wll be to join the chunks in the output in a smooth way with no separation). I hope to handle it during next two months or so, if not contributed by someone else, and perhaps much earlier, but right now I can not due to other tasks.

edit: a completely different strategy would be to try to produce same output as currently but without using at all the LaTeX framed package. This would be very significant refactoring but the nature of the contents (succession of lines with no vertical stretch) appears to allow it (contrarily to say contents of an admonition which may end up being arbitrary LaTeX structures). Having one color background command per line may perhaps cause unexpected PDF issues (in the past I have had some with colouring very many small pieces of text), but it may be worth a try. Much more adventurous than the stratgy of above paragraph.