Automatically install missing latex packages in Rnw vignettes

[jeroen]

I am using a docker image with tinytex to build R (source) packages on r-universe. I am looking for a way to automatically install missing latex packages used by vignettes.

Here is an example tbm_supplement.Rnw of a failing build: https://github.com/r-universe/r-forge/runs/2240700906?check_suite_focus=true

Error: processing vignette 'tbm_supplement.Rnw' failed with diagnostics:
Running 'texi2dvi' on 'tbm_supplement.tex' failed.
LaTeX errors:
! LaTeX Error: File `accents.sty' not found.

Type X to quit or <RETURN> to proceed,
or enter new name. (Default extension: sty)

According to the tinytex manual I could use tlmgr search --global --file "/accents.sty however I would like an automated solution that works for all packages at once.

I do not control the source code for these packages, but I can easily add hacks to my docker image that does the building, for example to override the vignette engine for Rnw files, or run some code before R CMD build.

---

By filing an issue to this repo, I promise that

• I have fully read the issue guide at https://yihui.org/issue/.
• I have provided the necessary information about my issue.
  • If I'm asking a question, I have already asked it on Stack Overflow or RStudio Community, waited for at least 24 hours, and included a link to my question there.
  • If I'm filing a bug report, I have included a minimal, self-contained, and reproducible example, and have also included xfun::session_info('tinytex'). I have upgraded all my packages to their latest versions (e.g., R, RStudio, and R packages), and also tried the development version: remotes::install_github('yihui/tinytex').
  • If I have posted the same issue elsewhere, I have also mentioned it in this issue.
• I have learned the Github Markdown syntax, and formatted my issue correctly.

I understand that my issue may be closed if I don't fulfill my promises.

[cderv]

I don't know the exact answer for it but my understanding is that R will run tools::tex2pdf() for a vignette that should produce .pdf when the vignette engine produces a .tex file.

%\VignetteEngine{knitr::knitr} does produce a .tex file I believe, which lead to R running tools::tex2dvi() to produce PDF.

If the engine was to produce .pdf from .Rnw directly using tinytex I think this would prevent R trying to render to .pdf. I am not aware of such engine yet. Maybe this is something that could be added to tinytex or knitr ?

I know you can use a Makefile in the vignettes/ folder to prevent R to build pdf vignette using texi2pdf() itself. It is in the doc from CRAN but I don't know a lot of example of such Makefile. Only one in R.rsp that I found once trying to understand vignette engines. It would require creating a generic Makefile to use for .Rnw vignettes if possible and adding this Makefile in the folder of the vignettes - not sure you can do that as it is close to touching the source code. knitr::knit2pdf() does .Rnw to .pdf using tinytex.

EDIT: in fact it is easy to find those Makefile with a github search

Anyway, just sharing thoughts. @yihui may have more targeted insights for this.

BTW, about this in your dockerfile

# Install TinyTex + common packages and put it on the PATH
RUN R -e 'install.packages("tinytex");tinytex::install_tinytex();tinytex:::install_yihui_pkgs()'

You should be able to directly install a binary of TinyTeX with the all the relevant packages by using

tinytex:::install_prebuilt('TinyTeX')

This is the prebuild binary with the most packages included. See https://github.com/yihui/tinytex-releases for those prebuilt binary and other installation method. This may save some installation time from tinytex:::install_yihui_pkgs().

[yihui]

@jeroen There are a couple of ways to solve this problem. I'm not sure which one below is dirtier :)

1. When R CMD build fails, you could check if it was due to missing LaTeX packages: if tinytex::parse_packages('00check.log') returns a non-empty vector, use tinytex::tlmgr_install() to install the package(s). Here 00check.log is just an example (sometimes I use it myself). In your case, you can capture the stderr and pass the text to the text argument of parse_packages().
2. Before R CMD build, if you detect .Rnw vignettes in a package, run knitr::knit2pdf() on them, so missing packages can be pre-installed. This is because knitr::knit2pdf() calls tinytex::latexmk().
3. Write an executable shell script that calls a command like Rscript -e "tinytex::latexmk('$1')", and point the environment variable PDFLATEX to this script, so that tools::texi2pdf() uses this shell script to compile .tex to PDF, instead of pdflatex. See ?tools::texi2pdf.

I've always wanted to do 3 (tinytex.sh or tinytex.exe) but didn't have time. It should be simple for *nix. I'm less familiar with Windows.

The 4th possibility, which is kind of similar to 3, is to rewrite tinytex::latexmk() in another language, so we can have an executable that works on any platform (essentially feature request #251). Then you just need to point the env var PDFLATEX to this executable. This is also what I wanted to do for long but it would require substantially more effort. The good news is that someone has mostly done it with Deno when working on a larger project, and we need to discuss how to extract this part out of the project. (Update: this has been done but not announced yet. Stay tuned.)

This suggestion by @cderv above is a good one:

tinytex:::install_prebuilt('TinyTeX')

This version of TinyTeX was (pre)built exactly for the reason of building PDF vignettes from other R packages. The file tools/pkgs-yihui.txt contains names of LaTeX packages that I needed to install locally to check the reverse dependencies of packages I maintain (in particular, knitr and rmarkdown). You can probably maintain a list somewhere, too, to avoid searching and installing missing packages that you have previously discovered.

[yihui]

Actually there is the 5th possibility. That is, have an R core sponsor tinytex::latexmk() in tools::texi2pdf() (not sure if @zeileis is interested), e.g., let tools::texi2pdf(tinytex = TRUE) call tinytex::latexmk(). By default, this can be FALSE, but configurable via a global option (e.g., texi2pdf.tinytex) or environment variable (e.g., R_TEXI2PDF_TINYTEX).

This will introduce a Suggests dependency on the tinytex package, but this is not the first time that base R has soft dependencies on other CRAN packages, e.g., utils::news() uses commonmark and xml2 to process NEWS.md.

[jeroen]

I have started with trying #3: set PDFLATEX to a script like your example. But no luck so far:

Error: processing vignette 'tbm_supplement.Rnw' failed with diagnostics:
Running 'texi2dvi' on 'tbm_supplement.tex' failed.
Messages:
/usr/bin/texi2dvi: TeX neither supports -recorder nor outputs \openout lines in its log file
--- failed re-building ‘tbm_supplement.Rnw’

SUMMARY: processing the following file failed:
  ‘tbm_supplement.Rnw’

Error: Vignette re-building failed.
Execution halted

I did confirm that tinytex is able to compile the tex file when I manually run it, so I am guessing R invokes PDFLATEX in a more complex way? Perhaps the script should support additional arguments from pdflatex, rather than only the file parameter?

[yihui]

Besides the env var PDFLATEX, you also need to set R_TEXI2DVICMD=emulation to bypass the program texi2dvi.

I was wrong above: $1 should be $2, i.e., the .tex file is passed to the PDFLATEX command in the second argument: https://github.com/wch/r-source/blob/2adaba8bb75b8fac8df2e7f0135349b8663a38da/src/library/tools/R/utils.R#L466-L467

writeLines('Rscript -e "tinytex::latexmk(\'$2\', clean = FALSE)"', 'tinytex.sh')
system('chmod +x tinytex.sh')
Sys.setenv(PDFLATEX = normalizePath('tinytex.sh'))

writeLines('\\documentclass{article}\\begin{document}abc\\end{document}', 'test.tex')

tools::texi2pdf('test.tex', texi2dvi = 'emulation')

[cderv]

@yihui what about passing your special program directly in texi2dvi=

file would be the third argument - it seems to work.

writeLines('Rscript -e "tinytex::latexmk(\'$3\')"', 'tinytex.sh')
system('chmod +x tinytex.sh')
writeLines('\\documentclass{article}\\begin{document}abc\\end{document}', 'test.tex')
tools::texi2pdf('test.tex', texi2dvi = normalizePath('tinytex.sh'))

[yihui]

@cderv That might work, too. I didn't test it, but I guess it would be easier to bypass the texi2dvi command altogether, because the function tools::texi2dvi() seems to run certain commands to check the functionality of the texi2dvi command, e.g., https://github.com/wch/r-source/blob/2adaba8bb75b8fac8df2e7f0135349b8663a38da/src/library/tools/R/utils.R#L303

Anyway, I just figured out why my previous attempt didn't work. I need to call tinytex::latexmk(clean = FALSE) in the shell script, otherwise tools::texi2dvi() will error because the LaTeX log file has been deleted. See my edited reply above for a working example.

[cderv]

Oh I see. It did not seem to cause issue as the document get correctly converted when I tested. I guess it would easy to write a script that understand this --help command to handle this correctly.

But setting texi2dvi option to "emulation" and the PDFLATEX env var is cleaner and not hacky. A lot simpler than the Makefile too.

Thanks for explaining.

👍

[jeroen]

OK that seems to work, even without setting R_TEXI2DVICMD. Is there any benefit/drawback from bypassing texi2dvi? Or does it just waste time on something that we don't need?

[zeileis]

I don't have the power to make any changes in tools::texi2dvi() even though I had written the original code for it a long time ago - but essentially everything was rewritten by R Core (which I'm not part of). I'll try to find out whether a soft dependency would be something R Core would consider and what would need to be done for that.

[yihui]

@jeroen If it works fine without setting R_TEXI2DVICMD but only setting PDFLATEX, I guess it should be fairly safe and I can't think of possible drawbacks. My impression is that texi2dvi is only used for building R manuals (e.g., R-intro.texi). The possible benefit of bypassing it is that you can skip the unnecessary checks in tools::texi2dvi() like the one I mentioned above.

@zeileis Since we have a workaround that works okay, we don't have to ask for R core to consider the support. In fact, I'm a little worried that some of them may seriously frown upon tinytex's feature of automatically installing LaTeX packages, since this violates a CRAN policy (packages shouldn't install anything without users' permission). This is one crime that I have never confessed to CRAN, since turning off this feature by default would make the package much less useful and also waste users' time on reading the documentation (on how to turn it on). Once we rewrite the R package in another language and have a real tinytex executable, I'll be able to walk out of the shadow...

[zeileis]

OK, I understand. I have already written to Kurt and asked him (a) how CRAN generally deals with the problem of installing the required LaTeX packages and (b) whether automatically installing LaTeX packages would be a useful feature for tools::texi2dvi(), - either via tintytex::latexmk() or otherwise. I'll report what he says here before taking any further steps.

[jeroen]

To wrap this up, this solution in r-universe-org/build-source@97183f4 seems to work. A shell script pdftinytex with:

#!/bin/sh
Rscript -e "tinytex::pdflatex('$2',clean = FALSE)"

Put that on the PATH and then we run:

R_TEXI2DVICMD=emulation PDFLATEX=pdftinytex R CMD build $PACKAGE

The only drawback I found is that tinytex::pdflatex does not raise an error when it fails to install a ctan package. This made it difficult to debug the cause of error for e.g. #293 because the only thing that I noticed was that the vignette failed to render due to missing latex packages, but not why tinytex had failed to install it.

[zeileis]

Just as a last quick follow-up after briefly talking to Kurt how CRAN deals with LaTeX packages: The Linux machines just seem to have texlive-full installed. But Kurt found the suggestion of a R_TEXI2PDF_TINYTEX variable interesting and will consider it the next time he works on the base R code related to this (given that this wasn't a pressing issue anymore). I offered to help.