Dead kernel when running any R code, PNG and X11 related

[okorn-src]

Hi,

I'm not sure this is a bug but maybe just some clarification or a workaround required.

Seems related to Issue #386

When attempting to execute any R code (not even graphics related, just something like "a <- 1" will do it), the kernel dies.

Error output:

Error in .External2(C_X11, paste("png::", filename, sep = ""), g$width,  :
  unable to start device PNG
Calls: <Anonymous> ... evaluate -> dev.new -> do.call -> <Anonymous> -> ok_device
In addition: Warning message:
In ok_device(filename, ...) : unable to open connection to X11 display ''
Execution halted

My R capabilities:

       jpeg         png        tiff       tcltk         X11        aqua
       TRUE        TRUE       FALSE       FALSE       FALSE       FALSE
   http/ftp     sockets      libxml        fifo      cledit       iconv
       TRUE        TRUE        TRUE        TRUE        TRUE        TRUE
        NLS     profmem       cairo         ICU long.double     libcurl
       TRUE       FALSE        TRUE       FALSE        TRUE       FALSE

I'm running on a headless Centos 6.7 server. R was configured with X11 support but capabilities() says it's FALSE because a graphics device is not found.

However I have both PNG and Cairo support enabled, I'm wondering if "Cairo" is supposed to be preferenced over PNG?

I installed IRkernel from Github yesterday July 31st.

> sessionInfo()
R version 3.2.2 (2015-08-14)
Platform: x86_64-pc-linux-gnu (64-bit)
Running under: CentOS release 6.7 (Final)

locale:
 [1] LC_CTYPE=en_AU.UTF-8       LC_NUMERIC=C
 [3] LC_TIME=en_AU.UTF-8        LC_COLLATE=en_AU.UTF-8
 [5] LC_MONETARY=en_AU.UTF-8    LC_MESSAGES=en_AU.UTF-8
 [7] LC_PAPER=en_AU.UTF-8       LC_NAME=C
 [9] LC_ADDRESS=C               LC_TELEPHONE=C
[11] LC_MEASUREMENT=en_AU.UTF-8 LC_IDENTIFICATION=C

attached base packages:
[1] stats     graphics  grDevices utils     datasets  methods   base

other attached packages:
[1] IRkernel_0.6

loaded via a namespace (and not attached):
 [1] R6_2.1.2        magrittr_1.5    IRdisplay_0.4.3 pbdZMQ_0.2-3
 [5] tools_3.2.2     uuid_0.1-2      stringi_1.0-1   jsonlite_1.0

If this isn't a bug, my intention is to run an "xvfb" server to enable an X11 graphics device and see if that resolves the issue but I'd rather try to make it work without resorting to that.

Any help/info appreciated.

[flying-sheep]

if you have the PNG capability and your PNG device doesn’t work, that’s an R bug or a problem with your setup.

R was configured with X11 support but capabilities() says it's FALSE because a graphics device is not found.

this also sounds strongly like there’s something wrong with your installation.

you can circumvent it by installing the Cairo package, which will take precedence over your broken device.

[okorn-src]

Thanks for the reply.

The Cairo PNG device is working in my installation:

library(datasets)
png("test.png", width=1200, height=1200, units="px", type="cairo")
boxplot(mpg~cyl,data=mtcars, main="Car Milage Data", xlab="Number of Cylinders", ylab="Miles Per Gallon")
dev.off()

Creates the expected output.

I have no X11 device because I'm running on a headless server - this is reflected in R capabilities() showing X11 as FALSE.

Is there a reason why IRkernel attempts to use png() with X11 when my capabilities() say X11 is not available? Is this expected behaviour?

Is there a way for me to tell IRkernel to use Cairo and not X11?

Secondly, I'm not sure why a graphics device is required to execute code:

a <- 1

I have not used Jupyter before so maybe the answer is blindingly obvious, and if so I do apologise - I'm just trying to set up multi-user Jupyter on a compute server that is used by my team, with R support being vital to this setup.

EDIT: I guess these questions might pertain to Jupyter and not IRkernel? Sorry I'm not sure which responsibilities belong where at the moment.

[flying-sheep]

hmm, so there’s two places where devices are used:

1. the dummy device is png. the dummy device is started before every code cell is run to capture possible plots.
2. when plots are displayed we use multiple formats. when generating them, we prefer Cairo if the Cairo package (not capability) is available.

since you have a broken installation that doesn’t allow you to use png with its default settings, you need to either fix the installation or make sure the png device is never used with default settings:

1. you can put options(device = something_that_isnt_pdf) into your .Rprofile which overrides the dummy device
2. make sure that the Cairo package is used via install.packages('Cairo') (yes, i really meant package, not capability)

this is a bit complicated and nonobvious, but look:

we have changed the dummy device so often and discussed and debugged stuff so many times that i really don’t want to touch it. see: many weird setups and broken/unusual installations wanted to work with us and every change broke something else until we settled on our current dummy device. i’m sorry that it doesn’t work for you, but it’s complex enough as is. i think expecting png('some-filename') to work isn’t that far fetched

[okorn-src]

Thank you very much for taking the time to explain the things I was missing, and for linking to the relevant code!

I understand now the expectation for the png() device, thanks. It's fine to have these requirements, it just really was non-obvious to me - and maybe others, I hope this ticket helps them too.

I will definitely try the Cairo package (not capability =) as that will be simpler than maintaining something like an "xvfb" virtual X11 device on the machine, and less messy than overriding the dummy device in R - but nice to have that option as a back up too.

Many thanks!

[flying-sheep]

no problem! 😄

but you seem to have missed a detail:

if png('filename') doesn’t work, you’ll need to both install Cairo and set the device option to override the dummy device, as otherwise, png is used in both pieces of code.

alternatively, you can make png work somehow, be it the xvfb device or a new installation with another configuration.

[okorn-src]

You're right about still needing png() to work - I resolved my issue by setting ~/.Rprofile for "root" (Jupyterhub process user) as follows:

## Set default 'type' for png() calls - useful when X11 device is not available!
## NOTE: Needs 'cairo' capability
options(bitmapType='cairo')

Cheers!

[flying-sheep]

wonderful, didn’t know that can be done. and as a side effect, your R environment got saner 😄

[ahsanshah]

I was stuck on this same issue for hours. I tried multiple installs/reinstalls of conda r-essentials and updated zeromq, pcre, etc. Kernel still kept dying every time I enter library(dplyr). The notebook console showed the following:

Error in .External2(C_X11, paste("png::", filename, sep = ""), g$width, :
unable to start device PNG

I was running jupyter notebook server headless remotely via a nohup comand. When I ran the same command via VNC on the VM itself, it seems to have resolved the issue at least for me. This issue has indeed been a pain.

[flying-sheep]

yeah exactly: you have R compiled to use png in a way that requires X11 to run.

it won’t work until you either

1. have X11 running, or
2. correctly compile R so that it works without X11, or
3. do the thing three comments up with the 👍 and 🎉 reactions

[steverweber]

another workaround...

/opt/r-3.4.1/bin/R -q -e "IRkernel::installspec(user=FALSE, name='opt-r-3.4.1', displayname='R 3.4.1')"

# force options(bitmapType='cairo')
cat > /usr/local/share/jupyter/kernels/opt-r-3.4.1/kernel.json <<EOF
{
  "argv": ["/opt/r-3.4.1/lib/R/bin/R", "--slave", "-e", "options(bitmapType='cairo') ; IRkernel::main()", "--args", "{connection_file}"],
  "display_name": "R 3.4.1",
  "language": "R"
}
EOF

[MaxGhenis]

I'm also hitting this--running from Crouton and a conda MRO installation--and the solutions suggested here haven't worked (or I'm not doing them right):

• Per Dead kernel when running any R code, PNG and X11 related #388 (comment) I set .Rprofile.
• Per Dead kernel when running any R code, PNG and X11 related #388 (comment) I ran R drawing stuff (e.g. plot(iris)) from the Linux UI. This works fine but Jupyter still breaks.
• Per Dead kernel when running any R code, PNG and X11 related #388 (comment) I tried that snippet but it failed since I have nothing in /opt. I suspect I'd need to do this elsewhere given I'm using conda.

Some relevant pieces of my conda list:

# Name                    Version                   Build  Channel
r                         3.4.3                  mro343_0  
r-irkernel                0.8.11                 mro343_0    r
r-cairo                   1.5_9                  r3.4.1_1    r
cairo                     1.14.12              h77bcde2_0  
jupyter                   1.0.0                      py_1    conda-forge

[okorn-src]

Hi @MaxGhenis,

In that snippet, simply substitute the R binary path with the one that lives in your Conda environment and it should work. If you don't know what that path is, activate your Conda env on the shell and do: which R

The ~/.Rprofile thing for root user may only be applicable to Jupyterhub - if you're using single-user Jupyter, I'm guessing that you'd have to set your own user's ~/.Rprofile? Try this first perhaps before modifying your kernel.json?

[sdwfrost]

I tried both ~/.Rprofile and modifying kernel.json, but without success. My repository is here. Any ideas welcome - I really need IRkernel!

[flying-sheep]

well, i don’t know what epirecipes is and I can’t help without any kind of information. if you have the regular R with X11 support and so on, it’ll work. If you have something weird as mentioned above, one of the workarounds above helps. If you have something else, I need

• an error message that says what went wrong

• where you got R from:

  • source?
  • linux distribution repository? which one?
  • official binaries?
  • microsoft implementation?
  • sth. else?

• which compile options if you compiled it yourself

• packages relevant to the error message you give me

[dhirschfeld]

@MaxGhenis - it looks like you're mixing mro and non-mro packages so you might be also/instead running into ContinuumIO/anaconda-issues#9679

You probably want to install the mro version:

conda install --force defaults::r-cairo=1.5.9=mro343h889e2dd_0

[MaxGhenis]

@dhirschfeld what channel would this be on?

PackagesNotFoundError: The following packages are not available from current channels:

  - r-cairo==1.5.9=mro343h889e2dd_0

[dhirschfeld]

defaults. Your problem is it should only be a single =

[MaxGhenis]

Not sure I follow. That was the error message resulting from the command you suggested:

conda install --force defaults::r-cairo=1.5.9=mro343h889e2dd_0

[dhirschfeld]

The error message has == in it which suggests you typed == yourself.

[MaxGhenis]

I didn't:

(xenial)maxghenis@localhost:~$ conda install --force defaults::r-cairo=1.5.9=mro343h889e2dd_0
Solving environment: failed

PackagesNotFoundError: The following packages are not available from current channels:

  - defaults::r-cairo==1.5.9=mro343h889e2dd_0

[dhirschfeld]

What conda are you running? Does it work without the defaults::?

[MaxGhenis]

I'm on 4.5.8 (up-to-date). Same error without defaults:: (but without it in the error).

[dhirschfeld]

Sorry @MaxGhenis for the run around - it's a typo on my part:

λ conda search r-cairo
Loading channels: done
# Name                  Version           Build  Channel
<snip>
r-cairo                   1.5_9 mro343h889e2dd_0  pkgs/r

...the version should be 1.5_9 not 1.5.9

[MaxGhenis]

Still getting the error replacing 1.5.9 with 1.5_9, with and without defaults:::

(xenial)maxghenis@localhost:~$ conda install --force defaults::r-cairo=1.5_9=mro343h889e2dd_0
Solving environment: failed

PackagesNotFoundError: The following packages are not available from current channels:

  - defaults::r-cairo==1.5_9=mro343h889e2dd_0

[dhirschfeld]

Might be because I'm on win64 you should check the exact build string for your OS using conda search r-cairo.

I thought the build strings would be the same, but maybe that's not the case.

The version you want is the one with mro in the build string

[MaxGhenis]

That worked to install r-cairo, but the kernel is still dying (terminal output).

[dhirschfeld]

It looks like you're still running into the png issue.

I'm out of my depth on the png issue but I'd be inclined to try patching your kernel.json as described in #388 (comment).

[MaxGhenis]

Replacing that comment's /opt/r-3.4.1/bin/R with /home/maxghenis/miniconda3/bin/R got it to complete, but didn't stop the kernel failures.

I wonder if the crouton error message when starting jupyter notebook might be at fault, so opened dnschneid/crouton#3859.

[raivivek]

@MaxGhenis Wonder if you you were able to fix it at all

[MaxGhenis]

No I wasn't.

[flying-sheep]

Sorry for being of no help here, but In ok_device(filename, ...) : no png support in this version of R means that I can’t do anything!

you have to point the kernel JSON to a version that has png support.

[sdwfrost]

I finally got too frustrated with conda due to lots of clashes, and my need to have a single environment where everything works. I ended up just installing R from a PPA:

RUN add-apt-repository ppa:marutter/rrutter && \
    apt-get update && \
    apt-get install -yq \
    r-base r-base-dev && \
    apt-get clean && \
    rm -rf /var/lib/apt/lists/*


RUN mkdir -p /usr/local/share/jupyter/kernels
RUN R -e "setRepositories(ind=1:2);install.packages(c(\
    'devtools'), dependencies=TRUE, clean=TRUE, repos='https://cran.microsoft.com/snapshot/2018-09-01')"
RUN R -e "devtools::install_github('IRkernel/IRkernel')" && \
    R -e "IRkernel::installspec()" && \
    mv $HOME/.local/share/jupyter/kernels/ir* /usr/local/share/jupyter/kernels/ && \
    chmod -R go+rx /usr/local/share/jupyter && \
    rm -rf $HOME/.local && \
    fix-permissions /usr/local/share/jupyter /usr/local/lib/R
RUN pip install rpy2
RUN R -e "devtools::install_github('mrc-ide/odin',upgrade=FALSE)"

[diogomrks]

/opt/r-3.4.1/bin/R -q -e "IRkernel::installspec(user=FALSE, name='opt-r-3.4.1', displayname='R 3.4.1')"

force options(bitmapType='cairo')

cat > /usr/local/share/jupyter/kernels/opt-r-3.4.1/kernel.json <<EOF
{
"argv": ["/opt/r-3.4.1/lib/R/bin/R", "--slave", "-e", "options(bitmapType='cairo') ; IRkernel::main()", "--args", "{connection_file}"],
"display_name": "R 3.4.1",
"language": "R"
}
EOF

Brilliant out of all of the solutions here and elsewhere this is the only one that worked, my only issue is now when I install a package the console outputs come up on the log instead of the kernel console

This is in the editor/kernell/console

This is the log output

Not sure how this affects other day to day tasks yet

[flying-sheep]

See the topmost pinned issue: https://github.com/IRkernel/IRkernel/issues