Running rmarkdown from the command line without the need for X11 capability #1100

ndrubins · 2017-07-15T07:08:38Z

Hi there,

I'm trying to run an Rmarkdown script from the command line on my linux cluster, that will generate an html report, using this command:
R -e "rmarkdown::render('script.Rmd',output_file='output.html')"

I don't have X11 installed on my cluster nodes which caused it to crash so the solution was to install xvfb, and I'm now running it with this command:
xvfb-run --server-args="-screen 0 1024x768x24 -noreset" R -e "rmarkdown::render('script.Rmd',output_file='output.html')"

While it doesn't crash immediately anymore, xvfb has RAM usage issues, which I thought would be solved by adding the -noreset argument, but unfortunately that's not the case, and it still occasionally seq faults and crashes when it's out of RAM.

So my question is if there's an alternative way to solve this? Perhaps it is possible to build the Rmarkdown package without the dependence on X11?

The text was updated successfully, but these errors were encountered:

yihui · 2017-07-16T02:21:51Z

It might be due to the fact that the HTML output format uses the PNG device by default. I'm not quite sure, but you can change the default graphics device of your output format (usually the dev argument) to a device independent of X11, such as svg.

jjallaire · 2017-07-16T10:05:22Z

In most cases the png() device can use the cairo back-end rather than X11. My guess is that your version of R doesn't have cario support compiled in -- you can check this with capabilities("cairo")

…

On Sat, Jul 15, 2017 at 10:21 PM, Yihui Xie ***@***.***> wrote: It might be due to the fact that the HTML output format uses the PNG device by default. I'm not quite sure, but you can change the default graphics device of your output format (usually the dev argument) to a device independent of X11, such as svg. — You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub <#1100 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AAGXx1y2hrVzHVHS1IujajFTcBI-7yfPks5sOXO_gaJpZM4OY8LN> .

ndrubins · 2017-07-16T18:04:34Z

Thanks a lot for the quick responses.

I actually did try the cairo solution but it still crashes with the error that it cannot start the graphic device.

Here I'm invoking R from the cluster node I'm trying to run my code on - testing for capabilities("cairo")

[user@n01 ~]$ R

R version 3.3.1 (2016-06-21) -- "Bug in Your Hair"
Copyright (C) 2016 The R Foundation for Statistical Computing
Platform: x86_64-pc-linux-gnu (64-bit)

R is free software and comes with ABSOLUTELY NO WARRANTY.
You are welcome to redistribute it under certain conditions.
Type 'license()' or 'licence()' for distribution details.

R is a collaborative project with many contributors.
Type 'contributors()' for more information and
'citation()' on how to cite R or R packages in publications.

Type 'demo()' for some demos, 'help()' for on-line help, or
'help.start()' for an HTML browser interface to help.
Type 'q()' to quit R.

> capabilities("cairo")
cairo
 TRUE

Ok, so as far as I understand this means I have capabilities("cairo")

Then, trying this example Rmarkdown code (test.Rmd), which sets the device to CairoPNG :

---
title: test
output: html_document
---

```{r setup, include=FALSE,echo=FALSE}
library(knitr)
library(plotly)
library(Cairo)
knitr::opts_chunk$set(dev="CairoPNG")

set.seed(100)
plot_ly(y = ~rnorm(50), type = "box") %>% add_trace(y = ~rnorm(50, 1))

Calling it from the command line produces the X11 error:


[user@n01 ~]$ R -e rmarkdown::render"('test.Rmd',output_file='test.html')"

R version 3.3.1 (2016-06-21) -- "Bug in Your Hair"
Copyright (C) 2016 The R Foundation for Statistical Computing
Platform: x86_64-pc-linux-gnu (64-bit)

R is free software and comes with ABSOLUTELY NO WARRANTY.
You are welcome to redistribute it under certain conditions.
Type 'license()' or 'licence()' for distribution details.

R is a collaborative project with many contributors.
Type 'contributors()' for more information and
'citation()' on how to cite R or R packages in publications.

Type 'demo()' for some demos, 'help()' for on-line help, or
'help.start()' for an HTML browser interface to help.
Type 'q()' to quit R.
> rmarkdown::render('test.Rmd',output_file='test.html')


processing file: test.Rmd
  |................                                                 |  25%
  ordinary text without R code

  |................................                                 |  50%
label: setup (with options)
List of 2
 $ include: logi FALSE
 $ echo   : logi FALSE

Quitting from lines 7-11 (test.Rmd)
Error in .External2(C_X11, paste("png::", filename, sep = ""), g$width,  :
  unable to start device PNG
Calls: <Anonymous> ... block_exec -> chunk_device -> dev_new -> do.call -> <Anonymous>
In addition: Warning messages:
1: In grDevices::png(f) : unable to open connection to X11 display ''
2: In (function (filename = "Rplot%03d.png", width = 480, height = 480,  :
  unable to open connection to X11 display ''

Execution halted

Am I not setting the device to Cairo properly?

Thanks a lot

ndrubins · 2017-07-21T19:01:52Z

Just checking in to see if there's any idea as to why I'm getting that X11 error despite setting the device to CairoPNG

yihui · 2017-07-21T19:29:19Z

Although your capabilities("cairo") is true, I still feel there is something wrong with cairo, and png() silently fell back to X11 anyway. You can check getOption('bitmapType'). If cairo is properly supported, it should return cairo, otherwise it will be Xlib: https://github.com/wch/r-source/blob/af7f52f70101960861e5d995d3a4bec010bc89e6/src/library/grDevices/R/zzz.R#L34

Typically this can happen if you build R from source, but your R startup message says "Platform: x86_64-pc-linux-gnu (64-bit)", which indicates that you probably installed a prebuilt binary (what exactly is your OS?). There might be something wrong with the binary package.

ndrubins · 2017-07-21T19:34:59Z

You're right.

I do get the Xlib message to getOption('bitmapType').

My OS is Linux

yihui · 2017-07-21T19:38:55Z

Yeah I know it is Linux. I mean the specific distribution. I haven't had problems like this with Debian/Ubuntu (apt-get install r-base), and I don't have much experience with other distributions.

ndrubins · 2017-07-21T20:04:15Z

Sorry about that,
It's:
Linux version 3.10.0-327.4.4.el7.x86_64 (builder@kbuilder.dev.centos.org) (gcc version 4.8.3 20140911 (Red Hat 4.8.3-9) (GCC) ) #1 SMP Tue Jan 5 16:07:00 UTC 2016

yihui · 2017-07-21T20:06:10Z

I see. Then I cannot offer much help. A blind shot is to upgrade R and see if it makes any difference (the latest version of R is 3.4.1 now).

lillamy2012 · 2017-09-07T11:54:48Z

I have the same problem. I am trying to run the render function on a cluster without x11 (R -e "rmarkdown::render('test.Rmd')"). Using the opts_chunk$set doesn't do anything. However when I added the following line:
options(bitmapType='cairo')
to my .Rprofile, the command above runs without problems. I tried to add options(bitmapType='cairo') in the Rmd file but it does not work. I am not really sure why this is, can it really be R it self that is the problem? I mean I am able to set the bitmapType via the Rprofile. It also works if I run render in a R session were I first set bitmapType. We have a range of versions on the cluster and at least all recent ( I didn't try the older ones as I am not going to use them anyway) had the same problem.

My solution works but I am not 100% happy as I would prefer to have all code in the document. Do you have any suggestions?

yihui · 2017-09-07T16:34:36Z

@lillamy2012 Are you also using Red Hat? My guess is that it is an issue with base R (I have pointed out the relevant line in R source above). I think you can set options(bitmapType = 'cairo') in your R Markdown document if it works in .Rprofile, but you have to set it before any R code chunks that generate plots. If it does not work, please reduce your document to a minimal example and send it here. Thanks!

lillamy2012 · 2017-09-08T08:59:30Z

Hi Yihul! Thank you very much for the reply.
I did what you suggested but it does not work I am afraid. My example Rmd looks like this:

---
title: "Title"
author: "Author"
date: "03/09/2017"
output:
  html_document
---

```{r setup, include=FALSE}
options(bitmapType = 'cairo')

```{r simulate_data}
x <- rnorm(100)
y <- 2*x + rnorm(100)

That is, in this test.Rmd I do not even generate a plot. As before, with the options(bitmapType = 'cairo') in a .Rprofile it runs without problem.

The output I get is:

processing file: test.Rmd
  |.............                                                    |  20%
  ordinary text without R code

  |..........................                                       |  40%
label: setup (with options) 
List of 1
 $ include: logi FALSE

Quitting from lines 10-11 (test.Rmd) 
Error in .External2(C_X11, paste0("png::", filename), g$width, g$height,  : 
  unable to start device PNG
Calls: <Anonymous> ... block_exec -> chunk_device -> dev_new -> do.call -> <Anonymous>
In addition: Warning messages:
1: In grDevices::png(f) : unable to open connection to X11 display ''
2: In (function (filename = "Rplot%03d.png", width = 480, height = 480,  :
  unable to open connection to X11 display ''

Execution halted

So the error occurs in the setup chunk.

I will try to talk with the guys handling the cluster here to see how R was installed and if they can help out by e.g reinstalling R.

Thanks again!

yihui · 2017-09-08T13:57:11Z

@lillamy2012 Excellent. That is super helpful. I know how to fix it now.

Fix #1100: respect knitr's chunk option 'dev' if it has been set

hariiyer16 · 2018-02-07T19:58:32Z

To add to this discussion, I have a weird X11 issue when I run rmarkdown. The error I get is:
In grDevices::png(f) : unable to open connection to X11 display ''

I run R on a linux cluster:
Linux 2.6.32-696.18.7.el6.x86_64 #1 SMP Thu Dec 28 20:15:47 EST 2017 x86_64 x86_64 x86_64 GNU/Linux

I connect to the cluster using a Windows machine (in which case I use MobaXterm). I also connect to the cluster from MacOS, in which case I use the default terminal. Following is the weird issue I have:
On the MobaXterm: My *.rmd file was running fine until yesterday, and was generating an html output. However today, all of a sudden, it started giving the above X11 error. In all instances of running the *.rmd, I had included options(bitmapType="cairo") in the *.rmd script based on a suggestion in StackOverflow (for a prior issue I had). Howeverver, I have no clue why it stopped suddenly.
The weird part is, the script works when I run it from my MacOS using the terminal (ssh).

Any idea why this inconsistency based on the mode I use to connect to the Linux cluster?

yihui · 2018-02-07T20:29:44Z

@hariiyer16 Is everything up-to-date on the cluster? e.g. Have you tried update.packages(ask = FALSE, checkBuilt = TRUE)? devtools::session_info() and capabilities() will be helpful for diagnostics.

ndrubins · 2018-02-07T22:25:15Z

It ended up being a combination of how R is inatslled and having to run it with: xvfb-run R -e rmarkdown::render

…

On Wed, Feb 7, 2018 at 12:29 PM, Yihui Xie ***@***.***> wrote: @hariiyer16 <https://github.com/hariiyer16> Is everything up-to-date <https://yihui.name/en/2017/05/when-in-doubt-upgrade/> on the cluster? e.g. Have you tried update.packages(ask = FALSE, checkBuilt = TRUE)? devtools::session_info() and capabilities() will be helpful for diagnostics. — You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub <#1100 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/ABGsHuCwXmxscnVHlOE2nVz1xfLc8_zDks5tSge7gaJpZM4OY8LN> .

yihui · 2018-02-07T22:41:14Z

@ndrubins If you use the latest version of rmarkdown (>= 1.8), it should not matter how R was installed or run -- knitr will automatically check if png() works, and switch to svg() if png() doesn't work.

ndrubins · 2018-02-07T22:48:26Z

Thanks

…

On Wed, Feb 7, 2018 at 2:41 PM, Yihui Xie ***@***.***> wrote: @ndrubins <https://github.com/ndrubins> If you use the latest version of rmarkdown (>= 1.8), it should not matter how R was installed or run -- *knitr* will automatically check if png() works, and switch to svg() if png() doesn't work. — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#1100 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/ABGsHoCrFf7naHLENnlU72p8Hx75iOjZks5tSiaNgaJpZM4OY8LN> .

gaborcsardi · 2019-07-25T14:45:55Z

I think the solution to this mystery is that it is not enough to have cairo support for the "cairo" device to be selected, but R must be also compiled with pango support. See here:
https://github.com/wch/r-source/blob/5a156a0865362bb8381dcd69ac335f5174a4f60c/src/library/grDevices/src/init.c#L34

tcgriffith · 2019-12-24T08:41:10Z

Here to share my solution(actually yihui's) to compile the Rmd file in a server without X11 capability.
In short, I used the Cairo package, the dev parameter of the setup chunk should also be set, otherwise, the setup chunk won't run and the rmd will not be compiled.

My setup code:

# CairoPNG from Cairo package
```{r setup, include=FALSE, dev="CairoPNG"}
knitr::opts_chunk$set(dev="CairoPNG")
```

My school cluster's headless R Capabilities:

> capabilities()
       jpeg         png        tiff       tcltk         X11        aqua 
      FALSE       FALSE       FALSE       FALSE       FALSE       FALSE 
   http/ftp     sockets      libxml        fifo      cledit       iconv 
       TRUE        TRUE        TRUE        TRUE        TRUE        TRUE 
        NLS     profmem       cairo         ICU long.double     libcurl 
       TRUE        TRUE       FALSE        TRUE        TRUE        TRUE

Original discussion

github-actions · 2020-11-03T10:31:39Z

This old thread has been automatically locked. If you think you have found something related to this, please open a new issue by following the issue guide (https://yihui.org/issue/), and link to this old issue if necessary.

yihui added this to the v1.7 milestone Aug 29, 2017

yihui added the bug an unexpected problem or unintended behavior label Sep 8, 2017

jjallaire closed this as completed in 7d93e87 Sep 9, 2017

jjallaire added a commit that referenced this issue Sep 9, 2017

Merge pull request #1140 from yihui/bugfix/1100

d31ec87

Fix #1100: respect knitr's chunk option 'dev' if it has been set

gaborcsardi mentioned this issue Jul 25, 2019

Error: callr subprocess failed: X11 is not available tidyverse/reprex#268

Closed

github-actions bot locked as resolved and limited conversation to collaborators Nov 3, 2020

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Running rmarkdown from the command line without the need for X11 capability #1100

Running rmarkdown from the command line without the need for X11 capability #1100

ndrubins commented Jul 15, 2017 •

edited

Loading

yihui commented Jul 16, 2017

jjallaire commented Jul 16, 2017 via email

ndrubins commented Jul 16, 2017 •

edited

Loading

ndrubins commented Jul 21, 2017

yihui commented Jul 21, 2017

ndrubins commented Jul 21, 2017

yihui commented Jul 21, 2017

ndrubins commented Jul 21, 2017

yihui commented Jul 21, 2017

lillamy2012 commented Sep 7, 2017

yihui commented Sep 7, 2017 •

edited

Loading

lillamy2012 commented Sep 8, 2017

yihui commented Sep 8, 2017

hariiyer16 commented Feb 7, 2018

yihui commented Feb 7, 2018

ndrubins commented Feb 7, 2018 via email

yihui commented Feb 7, 2018

ndrubins commented Feb 7, 2018 via email

gaborcsardi commented Jul 25, 2019

tcgriffith commented Dec 24, 2019

github-actions bot commented Nov 3, 2020

Running rmarkdown from the command line without the need for X11 capability #1100

Running rmarkdown from the command line without the need for X11 capability #1100

Comments

ndrubins commented Jul 15, 2017 • edited Loading

yihui commented Jul 16, 2017

jjallaire commented Jul 16, 2017 via email

ndrubins commented Jul 16, 2017 • edited Loading

ndrubins commented Jul 21, 2017

yihui commented Jul 21, 2017

ndrubins commented Jul 21, 2017

yihui commented Jul 21, 2017

ndrubins commented Jul 21, 2017

yihui commented Jul 21, 2017

lillamy2012 commented Sep 7, 2017

yihui commented Sep 7, 2017 • edited Loading

lillamy2012 commented Sep 8, 2017

yihui commented Sep 8, 2017

hariiyer16 commented Feb 7, 2018

yihui commented Feb 7, 2018

ndrubins commented Feb 7, 2018 via email

yihui commented Feb 7, 2018

ndrubins commented Feb 7, 2018 via email

gaborcsardi commented Jul 25, 2019

tcgriffith commented Dec 24, 2019

github-actions bot commented Nov 3, 2020

ndrubins commented Jul 15, 2017 •

edited

Loading

ndrubins commented Jul 16, 2017 •

edited

Loading

yihui commented Sep 7, 2017 •

edited

Loading