C stack error within RStudio console for code without error in terminal

[technocrat]

System details

RStudio Edition : Desktop
RStudio Version : 1.3.1091
OS Version      :  Ubuntu 20.04 derivative
R Version       :  4.0.2

Steps to reproduce the problem

Run the code block in RStudio with CTRL-ENTER or in Console

    library(Runuran)
    pdf <- function (x) { exp(-0.5*x^2) }
    gen <- pinv.new(pdf = pdf, lb = -Inf, ub = Inf)
    Cstack_info()`

Describe the problem in detail

Execution differs in RStudio, which throws the error:

# Error: C stack usage  7969228 is too close to the limit

and fails to create the gen object from the same code in the Terminal pane or shell. No difference whether executed from a saved script with .R extension or in an untitled/unsaved window.

Describe the behaviour you expected

Code would run identically in RStudio Console as Terminal

MWE

        library(Runuran)
        pdf <- function (x) { exp(-0.5*x^2) }
        gen <- pinv.new(pdf = pdf, lb = -Inf, ub = Inf)
        Cstack_info()
        # size    current  direction eval_depth
        # 7969177      22992          1          2
        # Error: C stack usage  7969228 is too close to the limit

(Curiously, not reproducible using `reprex`)
    
No problem in terminal, whether root or not. No problem with the code block in an RMarkdown chunk. Fail in executing the block from CTRL-ENTER in an R script, with the Console pane, but not in the Terminal pane.
    
Under RStudio 1.3.1093 (latest as of 2020-10-14)

```sessionInfo()
#> R version 4.0.2 (2020-06-22)
#> Platform: x86_64-pc-linux-gnu (64-bit)
#> Running under: Pop!_OS 20.04 LTS ## Ubuntu with hot pants
#> 
#> Matrix products: default
#> BLAS:   /usr/lib/x86_64-linux-gnu/blas/libblas.so.3.9.0
#> LAPACK: /usr/lib/x86_64-linux-gnu/lapack/liblapack.so.3.9.0
#> 
#> locale:
#>  [1] LC_CTYPE=en_US.UTF-8       LC_NUMERIC=C              
#>  [3] LC_TIME=en_US.UTF-8        LC_COLLATE=en_US.UTF-8    
#>  [5] LC_MONETARY=en_US.UTF-8    LC_MESSAGES=en_US.UTF-8   
#>  [7] LC_PAPER=en_US.UTF-8       LC_NAME=C                 
#>  [9] LC_ADDRESS=C               LC_TELEPHONE=C            
#> [11] LC_MEASUREMENT=en_US.UTF-8 LC_IDENTIFICATION=C       
#> 
#> attached base packages:
#> [1] stats     graphics  grDevices utils     datasets  methods   base     
#> 
#> loaded via a namespace (and not attached):
#>  [1] crayon_1.3.4      digest_0.6.25     backports_1.1.10  lifecycle_0.2.0  
#>  [5] reprex_0.3.0.9001 magrittr_1.5      evaluate_0.14     pillar_1.4.6     
#>  [9] highr_0.8         stringi_1.5.3     rlang_0.4.8       rstudioapi_0.11  
#> [13] fs_1.5.0          vctrs_0.3.4       ellipsis_0.3.1    rmarkdown_2.4    
#> [17] styler_1.3.2      tools_4.0.2       stringr_1.4.0     purrr_0.3.4      
#> [21] xfun_0.18         yaml_2.2.1        compiler_4.0.2    pkgconfig_2.0.3  
#> [25] htmltools_0.5.0   knitr_1.30        tibble_3.0.4

### Assurances

- [x ] I have read the guide for [submitting good bug reports](https://github.com/rstudio/rstudio/wiki/Writing-Good-Bug-Reports).
- [x ] I have installed the latest version of RStudio, and confirmed that the issue still persists.
- [x ] If I am reporting a RStudio crash, I have included a [diagnostics report](https://support.rstudio.com/hc/en-us/articles/200321257-Running-a-Diagnostics-Report).
- [x ] I have done my best to include a minimal, self-contained set of instructions for consistently reproducing the issue.

[kevinushey]

Looks like RStudio is getting stuck in a loop from object.size():

(lldb) bt 10
* thread #1, queue = 'com.apple.main-thread', stop reason = breakpoint 2.1
  * frame #0: 0x000000011041fb86 libR.dylib`Rf_errorcall(call=0x00007ff6a300f6e0, format="C stack usage  %ld is too close to the limit") at errors.c:857:17 [opt]
    frame #1: 0x000000011041fb1c libR.dylib`R_SignalCStackOverflow(usage=<unavailable>) at errors.c:99:5 [opt]
    frame #2: 0x0000000112a71493 utils.so`objectsize + 2723
    frame #3: 0x0000000112a70a9c utils.so`objectsize + 172
    frame #4: 0x0000000112a7146a utils.so`objectsize + 2682
    frame #5: 0x0000000112a70bf7 utils.so`objectsize + 519
    frame #6: 0x0000000112a70a9c utils.so`objectsize + 172
    frame #7: 0x0000000112a7146a utils.so`objectsize + 2682
    frame #8: 0x0000000112a70bf7 utils.so`objectsize + 519
    frame #9: 0x0000000112a70a9c utils.so`objectsize + 172

You can reproduce outside of RStudio with, for example:

> object.size(gen)
Error: C stack usage  7969180 is too close to the limit

I suspect this is caused by recursive references within the R object of some form.

[kevinushey]

FWIW pryr::object_size() does the right thing here.

[technocrat]

Mystery deepens: The stack limit does not prevent the creation of the gen object in the Terminal, or throw the error message unless expected with object.size() while in Console, the message is thrown and the object is not created.

Since there are many cases in which ulimit gets pushed, RStudio vs terminal choice determines whether the code runs or not. Hard to squeeze a feature out of the difference, rather than a bug.

[mikebessuille]

Unfortunately it's too late to fix this in 1.4. Moving to 1.5. Workaround could be to turn off auto refresh on the environment pane.