R kernel slow/freeze then dies with large R object - using R alone works fine

[kklot]

Description

Using R with large object (model output RDS file size ~600Mb). Loading in to R everything is fine, no lag, no freezing.

Running the same code in ipynb, even running a simple str(x, 1) cause it to freeze. Happened in both mac OS M1 (16GB RAM) as well as AlmaLinux server with 128GB RAM.

Reproduce

I don't know if I should share my large object for this report.

Expected behavior

Same performance as in R's terminal.

[krassowski]

This is likely a problem with the specific R kernel that you are using, not JupyterLab. Do you use irkernel, xeus-r, juniper, or yet another R kernel?

[kklot]

I have only irkernel currently, thanks for pointing to others. Do you have a recommendation for one over the others regarding performance?

[krassowski]

I would suggest trying out https://github.com/jupyter-xeus/xeus-r

[JasonWeill]

@kklot Any update on the performance when using xeus-r?

[kklot]

Summarise: changing the kernel did not help, freezing after running a few cells similar to IRKernel.

Other problems came, bunches of installation issues as xeus-r is not in conda channels, then vscode xeus-r can not render figure (just showed plot without title) while jupyterlab can --> something not right between vscode and xeus-r.

I ended up download the files locally (which I was avoiding due to their large size), and worked on it with R in terminal, which run smoothly.

[JasonWeill]

@kklot Thanks for the update! To help people reproduce and fix this, could you provide an example very large dataset, or code that would generate a synthetic example similar to the one with which you observed this bug?

[kklot]

Hi, I can't share the file, it is just a huge list of embeded list of matrices (R-INLA model output with full posterior, marginal distribution,...) of a ~2 million observation).

It seems that uncheck kernel completion helps so far, but now there is no autocomplete at all.